paper-fetch

Fetch the legal open-access PDF for a paper given a DOI (or title). Tries multiple OA sources in priority order and stops at the first hit.

Resolution order

1. Unpaywall —

   https://api.unpaywall.org/v2/{doi}?email=$UNPAYWALL_EMAIL

   , read

   best_oa_location.url_for_pdf

   (skipped if

   UNPAYWALL_EMAIL

   not set)
2. Semantic Scholar —

   https://api.semanticscholar.org/graph/v1/paper/DOI:{doi}?fields=openAccessPdf,externalIds

3. arXiv — if

   externalIds.ArXiv

   present,

   https://arxiv.org/pdf/{arxiv_id}.pdf

4. PubMed Central OA — if PMCID present,

   https://www.ncbi.nlm.nih.gov/pmc/articles/{pmcid}/pdf/

5. bioRxiv / medRxiv — if DOI prefix is

   10.1101

   , query

   https://api.biorxiv.org/details/{server}/{doi}

   for the latest version PDF URL
6. Otherwise → report failure with title/authors so the user can request via ILL

If only a title is given, resolve to a DOI first via Semantic Scholar

search_paper_by_title

Usage

python scripts/fetch.py <DOI> [--out DIR] [--dry-run] [--format json|text]

Flags

doi

10.1038/s41586-020-2649-2

--batch FILE

--out DIR

pdfs

--dry-run

--format

json

json

text

Output contract

stdout emits a single JSON object (when

--format json

Success (all DOIs resolved):

{
  "ok": true,
  "data": {
    "results": [
      {
        "doi": "10.1038/s41586-020-2649-2",
        "success": true,
        "source": "unpaywall",
        "pdf_url": "https://...",
        "file": "pdfs/Author_2020_Title.pdf",
        "meta": {"title": "...", "year": 2020, "author": "Smith"}
      }
    ],
    "summary": {"total": 1, "succeeded": 1, "failed": 0}
  }
}

Partial failure (batch mode — some DOIs failed, exit code 1):

{
  "ok": true,
  "data": {
    "results": [
      {
        "doi": "10.1038/s41586-020-2649-2",
        "success": true,
        "source": "semantic_scholar",
        "pdf_url": "https://...",
        "file": "pdfs/Harris_2020_Array_programming_with_NumPy.pdf",
        "meta": {"title": "Array programming with NumPy", "year": 2020, "author": "Charles R. Harris"}
      },
      {
        "doi": "10.1234/nonexistent",
        "success": false,
        "source": null,
        "pdf_url": null,
        "file": null,
        "meta": {},
        "error": {"code": "not_found", "message": "No open-access PDF found", "retryable": false}
      }
    ],
    "summary": {"total": 2, "succeeded": 1, "failed": 1}
  }
}

Top-level failure (bad arguments, exit code 3):

{
  "ok": false,
  "error": {
    "code": "validation_error",
    "message": "Provide a DOI or --batch file",
    "retryable": false
  }
}

stderr carries human-readable progress diagnostics (source attempts, download status).

Exit codes

0

1

3

Error codes in JSON

validation_error

not_found

download_network_error

download_not_a_pdf

download_host_not_allowed

download_size_exceeded

download_io_error

internal_error

Examples

# Single DOI (JSON output for agents)
python scripts/fetch.py 10.1038/s41586-020-2649-2

# Dry-run preview (resolve without downloading)
python scripts/fetch.py 10.1038/s41586-020-2649-2 --dry-run

# Human-readable output
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format text

# Batch download
python scripts/fetch.py --batch dois.txt --out ./papers

# Works without UNPAYWALL_EMAIL (skips Unpaywall, uses remaining 4 sources)
python scripts/fetch.py 10.1038/s41586-020-2649-2

Notes

• UNPAYWALL_EMAIL

  is optional but recommended. Set it once:

  export UNPAYWALL_EMAIL=you@example.com

  (e.g. in

  ~/.zshrc

  ). Without it, Unpaywall is skipped and the remaining 4 sources are still tried.
• Downloads are restricted to a host allowlist of known OA providers, with a 50 MB size limit per PDF.
• Never attempts to bypass paywalls. If no OA copy exists, the skill reports failure — do not suggest Sci-Hub or similar.
• Default output directory:

  ./pdfs/

  . Filenames:

  {first_author}_{year}_{short_title}.pdf

  .

Auto-update

When installed via

git clone

fetch.py

git pull --ff-only

• Non-blocking — the current invocation is not delayed; the pull runs in a new session and is fully detached
• Silent — all output goes to

  /dev/null

  , JSON contract on stdout is never polluted
• Throttled — at most once every 24 hours (stamped via

  .git/.paper-fetch-last-update

  )
• Safe —

  --ff-only

  refuses to merge if you have local edits; conflicts never happen
• Convergence — updates apply on the next invocation, not the current one (because the pull is backgrounded)

Environment variables

PAPER_FETCH_NO_AUTO_UPDATE

PAPER_FETCH_UPDATE_INTERVAL

86400

Auto-update is a no-op when the skill is not a git checkout (e.g. tarball install), when

git

rm <skill_dir>/.git/.paper-fetch-last-update