paper-fetch

Fetch the legal open-access PDF for a paper given a DOI (or title). Tries multiple OA sources in priority order and stops at the first hit.

Agent-native. Structured JSON envelope on stdout, NDJSON progress on stderr, stable exit codes, machine-readable schema, TTY-aware format default, idempotent retries.

Resolution order

1. Unpaywall —

   https://api.unpaywall.org/v2/{doi}?email=$UNPAYWALL_EMAIL

   , read

   best_oa_location.url_for_pdf

   (skipped if

   UNPAYWALL_EMAIL

   not set)
2. Semantic Scholar —

   https://api.semanticscholar.org/graph/v1/paper/DOI:{doi}?fields=openAccessPdf,externalIds

3. arXiv — if

   externalIds.ArXiv

   present,

   https://arxiv.org/pdf/{arxiv_id}.pdf

4. PubMed Central OA — if PMCID present,

   https://www.ncbi.nlm.nih.gov/pmc/articles/{pmcid}/pdf/

5. bioRxiv / medRxiv — if DOI prefix is

   10.1101

   , query

   https://api.biorxiv.org/details/{server}/{doi}

   for the latest version PDF URL
6. Otherwise → report failure with title/authors so the user can request via ILL

If only a title is given, resolve to a DOI first via Semantic Scholar

search_paper_by_title

Usage

python scripts/fetch.py <DOI> [options]
python scripts/fetch.py --batch <FILE|-> [options]
python scripts/fetch.py schema           # machine-readable self-description

Flags

doi

-

--batch FILE

-

--out DIR

pdfs

--dry-run

--format

json

text

json

text

--pretty

--stream

--overwrite

--idempotency-key KEY

<out>/.paper-fetch-idem/

--timeout SECONDS

30

--version

Agent discovery:

schema

subcommand

python scripts/fetch.py schema

Emits a complete machine-readable description of the CLI on stdout (no network). Includes

cli_version

schema_version

schema_version

Output contract

stdout emits a single JSON envelope. Every envelope carries a

meta

Success (all DOIs resolved):

{
  "ok": true,
  "data": {
    "results": [
      {
        "doi": "10.1038/s41586-021-03819-2",
        "success": true,
        "source": "unpaywall",
        "pdf_url": "https://www.nature.com/articles/s41586-021-03819-2.pdf",
        "file": "pdfs/Jumper_2021_Highly_accurate_protein_structure_predic.pdf",
        "meta": {"title": "Highly accurate protein structure prediction with AlphaFold", "year": 2021, "author": "Jumper"},
        "sources_tried": ["unpaywall"]
      }
    ],
    "summary": {"total": 1, "succeeded": 1, "failed": 0},
    "next": []
  },
  "meta": {
    "request_id": "req_a908f5156fc1",
    "latency_ms": 2036,
    "schema_version": "1.1.0",
    "cli_version": "0.3.0",
    "sources_tried": ["unpaywall"]
  }
}

Partial (batch mode — some DOIs failed, exit code reflects the failure class):

{
  "ok": "partial",
  "data": {
    "results": [
      { "doi": "10.1038/s41586-021-03819-2", "success": true, "source": "unpaywall", ... },
      {
        "doi": "10.1234/nonexistent",
        "success": false,
        "source": null,
        "pdf_url": null,
        "file": null,
        "meta": {},
        "sources_tried": ["unpaywall", "semantic_scholar"],
        "error": {
          "code": "not_found",
          "message": "No open-access PDF found",
          "retryable": true,
          "retry_after_hours": 168,
          "reason": "OA availability changes over time; retry after embargo lifts or preprint appears"
        }
      }
    ],
    "summary": {"total": 2, "succeeded": 1, "failed": 1},
    "next": ["paper-fetch 10.1234/nonexistent --out pdfs"]
  },
  "meta": { ... }
}

The

next

--idempotency-key

Failure (bad arguments, exit code 3):

{
  "ok": false,
  "error": {
    "code": "validation_error",
    "message": "Provide a DOI or --batch file",
    "retryable": false
  },
  "meta": { ... }
}

Per-item skipped (destination already exists, no

--overwrite

{
  "doi": "10.1038/s41586-021-03819-2",
  "success": true,
  "source": "unpaywall",
  "pdf_url": "https://...",
  "file": "pdfs/Jumper_2021_...pdf",
  "skipped": true,
  "skip_reason": "file_exists",
  "sources_tried": ["unpaywall"]
}

Idempotency replay (re-run with the same

--idempotency-key

The cached envelope is returned verbatim, but

meta.request_id

meta.latency_ms

meta.replayed_from_idempotency_key

Stderr progress (NDJSON)

When

--format json

{"event": "start",       "request_id": "req_...", "elapsed_ms": 2,    "doi": "10.1038/..."}
{"event": "source_try",  "request_id": "req_...", "elapsed_ms": 2,    "doi": "...", "source": "unpaywall"}
{"event": "source_hit",  "request_id": "req_...", "elapsed_ms": 2036, "doi": "...", "source": "unpaywall", "pdf_url": "..."}
{"event": "download_ok", "request_id": "req_...", "elapsed_ms": 4120, "doi": "...", "file": "..."}

Event types:

start

source_try

source_hit

source_miss

source_skip

source_enrich

source_enrich_failed

download_ok

download_error

download_skip

dry_run

not_found

update_check_spawned

request_id

elapsed_ms

source_enrich

author

title

fields

source_enrich_failed

unknown_<year>_…

When

--format text

Exit codes

0

1

retry_after_hours

2

3

4

The taxonomy lets an orchestrator route failures deterministically: exit 4 is worth retrying immediately, exit 1 is not, exit 3 is a bug in the caller.

Error codes in JSON

validation_error

not_found

retry_after_hours: 168

download_network_error

download_not_a_pdf

download_host_not_allowed

download_size_exceeded

download_io_error

internal_error

Examples

# Single DOI (JSON output when piped; text when in a terminal)
python scripts/fetch.py 10.1038/s41586-020-2649-2

# Dry-run preview (resolve without downloading)
python scripts/fetch.py 10.1038/s41586-020-2649-2 --dry-run

# Force JSON (for agents even inside a terminal)
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format json

# Human-readable with pretty colors in a pipeline
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format text

# Batch download, safely retriable
python scripts/fetch.py --batch dois.txt --out ./papers \
    --idempotency-key monday-review-batch

# Pipe DOIs from another tool
zot -F ids.json query ... | jq -r '.[].doi' | python scripts/fetch.py --batch -

# Agent discovery
python scripts/fetch.py schema --pretty

# Streaming mode — one result per line as each DOI resolves
python scripts/fetch.py --batch dois.txt --stream

# Works without UNPAYWALL_EMAIL (skips Unpaywall, uses remaining 4 sources)
python scripts/fetch.py 10.1038/s41586-020-2649-2

Environment

UNPAYWALL_EMAIL

PAPER_FETCH_ALLOWED_HOSTS

PAPER_FETCH_NO_AUTO_UPDATE

PAPER_FETCH_UPDATE_INTERVAL

86400

Notes

• Auth is delegated. The agent never runs a login subcommand. The human or the orchestrator sets

  UNPAYWALL_EMAIL

  in the environment; the agent inherits it. Missing email degrades gracefully to the remaining 4 sources.
• Trust is directional. CLI arguments are validated once at the entry point. The host allowlist and 50 MB size cap are enforced in the environment layer, not at the agent's request. An agent cannot loosen safety by passing a flag — only by the operator setting

  PAPER_FETCH_ALLOWED_HOSTS

  .
• Downloads are naturally idempotent. Re-running against the same

  --out

  skips files that already exist (deterministic filename:

  {first_author}_{year}_{short_title}.pdf

  ). Pair with

  --idempotency-key

  to also replay the exact envelope without any network I/O.
• Never attempts to bypass paywalls. If no OA copy exists, the skill reports failure honestly — do not suggest Sci-Hub or similar.
• Default output directory:

  ./pdfs/

  .

Auto-update

When installed via

git clone

fetch.py

git pull --ff-only

• Non-blocking — the current invocation is not delayed; the pull runs in a new session and is fully detached
• Silent — all git output goes to

  /dev/null

  , the stdout envelope is never polluted
• Throttled — at most once every 24 hours (stamped via

  .git/.paper-fetch-last-update

  )
• Safe —

  --ff-only

  refuses to merge if you have local edits; conflicts never happen
• Observable — when a pull is spawned, stderr emits

  {"event": "update_check_spawned", ...}

  (JSON mode) or a prose notice (text mode)
• Convergence — updates apply on the next invocation, not the current one (because the pull is backgrounded)

Force an immediate check with

rm <skill_dir>/.git/.paper-fetch-last-update