Reference Hallucination Arena Skill

Evaluate how accurately LLMs recommend real academic references using the OpenJudge

RefArenaPipeline

1. Load queries — from JSON/JSONL dataset
2. Collect responses — BibTeX-formatted references from target models
3. Extract references — parse BibTeX entries from model output
4. Verify references — cross-check against Crossref / PubMed / arXiv / DBLP
5. Score & rank — compute verification rate, per-field accuracy, discipline breakdown
6. Generate report — Markdown report + visualization charts

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib

Gather from user before running

OPENAI_API_KEY

DASHSCOPE_API_KEY

./evaluation_results/ref_hallucination_arena

"en"

"zh"

Quick start

CLI

# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
  --output_dir ./my_results --save

Python API

import asyncio
from cookbooks.ref_hallucination_arena.pipeline import RefArenaPipeline

async def main():
    pipeline = RefArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    for rank, (model, score) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {score:.1%}")

asyncio.run(main())

CLI options

--config

--output_dir

--save

False

--fresh

False

Minimal config file

task:
  description: "Evaluate LLM reference recommendation capabilities"

dataset:
  path: "./data/queries.json"

target_endpoints:
  model_a:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

  model_b:
    base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
    api_key: "${DASHSCOPE_API_KEY}"
    model: "qwen3-max"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

Full config reference

task

description

scenario

dataset

path

shuffle

false

max_queries

null

null

target_endpoints.<name>

base_url

api_key

${ENV_VAR}

model

system_prompt

{num_refs}

max_concurrency

5

extra_params

temperature

tool_config.enabled

false

tool_config.tavily_api_key

tool_config.max_iterations

10

tool_config.search_depth

"advanced"

"basic"

"advanced"

verification

crossref_mailto

pubmed_api_key

max_workers

10

timeout

30

verified_threshold

0.7

evaluation

timeout

120

retry_times

3

output

output_dir

./evaluation_results/ref_hallucination_arena

save_queries

true

save_responses

true

save_details

true

report

enabled

true

language

"zh"

"zh"

"en"

include_examples

3

chart.enabled

true

chart.orientation

"vertical"

"horizontal"

"vertical"

chart.show_values

true

chart.highlight_best

true

Dataset format

Each query in the JSON/JSONL dataset:

{
  "query": "Please recommend papers on Transformer architectures for NLP.",
  "discipline": "computer_science",
  "num_refs": 5,
  "language": "en",
  "year_constraint": {"min_year": 2020}
}

query

discipline

computer_science

biomedical

physics

chemistry

social_science

interdisciplinary

other

num_refs

language

"zh"

"en"

"zh"

year_constraint

{"exact": 2023}

{"min_year": 2020}

{"max_year": 2015}

{"min_year": 2020, "max_year": 2024}

Official dataset: OpenJudge/ref-hallucination-arena

Interpreting results

Overall accuracy (verification rate):

• > 75% — Excellent: model rarely hallucinates references
• 60–75% — Good: most references are real, some fabrication
• 40–60% — Fair: significant hallucination, use with caution
• < 40% — Poor: model frequently fabricates references

Per-field accuracy:

• title_accuracy

  — % of titles matching real papers

• author_accuracy

  — % of correct author lists

• year_accuracy

  — % of correct publication years

• doi_accuracy

  — % of valid DOIs

Verification status:

• VERIFIED

  — title + author + year all exactly match a real paper

• SUSPECT

  — partial match (e.g. title matches but authors differ)

• NOT_FOUND

  — no match in any database

• ERROR

  — API timeout or network failure

Ranking order: overall accuracy → year compliance rate → avg confidence → completeness

Output files

evaluation_results/ref_hallucination_arena/
├── evaluation_report.md          # Detailed Markdown report
├── evaluation_results.json       # Rankings, per-field accuracy, scores
├── verification_chart.png        # Per-field accuracy bar chart
├── discipline_chart.png          # Per-discipline accuracy chart
├── queries.json                  # Loaded evaluation queries
├── responses.json                # Raw model responses
├── extracted_refs.json           # Extracted BibTeX references
├── verification_results.json     # Per-reference verification details
└── checkpoint.json               # Pipeline checkpoint for resume

API key by model

gpt-*

o1-*

o3-*

OPENAI_API_KEY

claude-*

ANTHROPIC_API_KEY

qwen-*

dashscope/*

DASHSCOPE_API_KEY

deepseek-*

DEEPSEEK_API_KEY

api_key

base_url

Additional resources

• Full config examples: cookbooks/ref_hallucination_arena/examples/
• Documentation: docs/validating_graders/ref_hallucination_arena.md
• Official dataset: HuggingFace
• Leaderboard: openjudge.me/leaderboard