Paper Review Skill

Multi-stage academic paper review using the OpenJudge

PaperReviewPipeline

1. Safety check — jailbreak detection + format validation
2. Correctness — objective errors (math, logic, data inconsistencies)
3. Review — quality, novelty, significance (score 1–6)
4. Criticality — severity of correctness issues
5. BibTeX verification — cross-checks references against CrossRef/arXiv/DBLP

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for paper_review
pip install litellm
pip install pypdfium2  # only if using vision mode (use_vision_for_pdf=True)

Gather from user before running

OPENAI_API_KEY

ANTHROPIC_API_KEY

gpt-5.2

anthropic/claude-opus-4-6

dashscope/qwen-vl-plus

"NeurIPS 2025"

"The Lancet"

"Focus on experimental design"

"en"

"zh"

Quick start

File type is auto-detected:

.pdf

.tar.gz

.zip

.bib

# Basic PDF review
python -m cookbooks.paper_review paper.pdf

# With discipline and venue
python -m cookbooks.paper_review paper.pdf \
  --discipline cs --venue "NeurIPS 2025"

# Chinese output
python -m cookbooks.paper_review paper.pdf --language zh

# Custom reviewer instructions
python -m cookbooks.paper_review paper.pdf \
  --instructions "Focus on experimental design and reproducibility"

# PDF + BibTeX verification
python -m cookbooks.paper_review paper.pdf \
  --bib references.bib --email your@email.com

# Vision mode (for models that prefer images over text extraction)
python -m cookbooks.paper_review paper.pdf \
  --vision --vision_max_pages 30 --format_vision_max_pages 10

# TeX source package
python -m cookbooks.paper_review paper_source.tar.gz \
  --discipline biology --email your@email.com

# TeX source package with Chinese output and custom instructions
python -m cookbooks.paper_review paper_source.tar.gz \
  --language zh --instructions "This is a short paper, be concise"

# Verify a standalone BibTeX file
python -m cookbooks.paper_review --bib_only references.bib --email your@email.com

All options

input

--bib_only

--model

gpt-4o

--api_key

--base_url

/v1

/v1/chat/completions

--discipline

--venue

--instructions

--language

en

en

zh

--bib

--email

--paper_name

--output

--no_safety

--no_correctness

--no_criticality

--no_bib

--vision

--vision_max_pages

30

--format_vision_max_pages

10

--vision_max_pages

--timeout

7500

Interpreting results

Review score (1–6):

• 1–2: Reject (major flaws or well-known results)
• 3: Borderline reject
• 4: Borderline accept
• 5–6: Accept / Strong accept

Correctness score (1–3):

• 1: No objective errors
• 2: Minor errors (notation, arithmetic in non-critical parts)
• 3: Major errors (wrong proofs, core algorithm flaws)

BibTeX verification:

• verified

  : found in CrossRef/arXiv/DBLP

• suspect

  : title/author mismatch or not found — manual check recommended

Model selection

This pipeline uses litellm for model calls. Provider prefixes are handled automatically by the pipeline — see the table below.

IMPORTANT: The model MUST support multimodal (vision) input. PDF review uses vision mode (

--vision

The

--model

provider/model-name

--model

gpt-5.2

gpt-5-mini

OPENAI_API_KEY

gpt-5.2

anthropic/claude-opus-4-6

anthropic/claude-sonnet-4-6

ANTHROPIC_API_KEY

anthropic/

claude-opus-4-6

dashscope/qwen-vl-plus

dashscope/qwen-vl-max

DASHSCOPE_API_KEY

dashscope/

--api_key

--base_url

--base_url

Note on prefixes: The

dashscope/

and

anthropic/

prefixes are interpreted by the pipeline itself — do not add them to the actual API key or base URL. For OpenAI models the bare model name (e.g.

gpt-5.2

) is sufficient.

If the user does not specify a model, choose one based on available API keys:

1. DASHSCOPE_API_KEY

   set → use

   dashscope/qwen-vl-plus

   (vision-capable)

2. OPENAI_API_KEY

   set → search web for the latest vision-capable OpenAI model and use it (currently

   gpt-5.2

   )

3. ANTHROPIC_API_KEY

   set → search web for the latest vision-capable Anthropic model and use it with

   anthropic/

   prefix (currently

   anthropic/claude-opus-4-6

   )

Vision mode is enabled by default for PDF review. Pages are rendered as images, which preserves formatting, figures, and tables. To disable, pass

--no_vision

Additional resources

• Full

  PipelineConfig

  options: reference.md
• Discipline details and venues: reference.md

Troubleshooting API errors

CRITICAL: When the pipeline fails with an API error, you MUST diagnose and fix the root cause. Do NOT fall back to reading the PDF as plain text yourself and calling the API manually — this bypasses the entire review pipeline and produces incorrect, incomplete results.

Diagnose by reading the full error message, then follow the checklist below:

AuthenticationError / 401

• The API key is wrong or not set.
• Check the correct env var for the provider (see Model selection table).
• For DashScope:

  echo $DASHSCOPE_API_KEY

  — must be non-empty.
• Fix: export the correct key and re-run.

NotFoundError / 404 — model not found

• The model name string is wrong.
• Search the web for the provider's current model list and use the exact API ID.
• Common mistakes: using a ChatGPT UI name instead of the API ID, outdated snapshot suffix.
• Fix: correct

  --model

  and re-run.

BadRequestError / 400

• Often caused by

  --base_url

  ending with

  /v1/chat/completions

  instead of

  /v1

  . litellm appends the path automatically — strip everything after

  /v1

  .
• May also indicate the model does not support vision/image input. Use a vision-capable model (see Model selection) or omit

  --vision

  .
• Fix: correct

  --base_url

  or switch to a vision-capable model and re-run.

Connection error / endpoint not reachable

• --base_url

  points to the wrong host or port.
• Test the endpoint first:

  curl <base_url>/models -H "Authorization: Bearer <key>"

• Fix: correct

  --base_url

  to the reachable endpoint and re-run.

Timeout

• The model is taking too long (common for long PDFs with vision mode).
• Fix: increase

  --timeout

  (default 7500 s) or reduce

  --vision_max_pages

  .

After fixing, always re-run the full pipeline command.

Never summarise or interpret the paper yourself as a substitute for a failed pipeline run.