AI Model Benchmarking Guide

Overview

Rigorous evaluation is the backbone of machine learning research. A model is only as credible as its evaluation protocol: which benchmarks were used, how metrics were computed, whether results are reproducible, and how they compare to baselines. The proliferation of LLMs has made this both more important and more complex, with over 60 established benchmarks and a rapidly evolving landscape.

This guide covers the practical side of model benchmarking: how to use the EleutherAI Language Model Evaluation Harness (lm-evaluation-harness), how to select benchmarks for different research claims, how to avoid common evaluation pitfalls, and how to present results for publication. The focus is on academic rigor rather than leaderboard chasing.

Whether you are evaluating a fine-tuned model for a paper, comparing architectures for an ablation study, or reviewing a submitted manuscript's evaluation section, these patterns will help ensure the evaluation is sound.

The lm-evaluation-harness

The EleutherAI lm-evaluation-harness is the de facto standard for LLM evaluation in academic research, supporting 60+ tasks and used by most major LLM papers.

Installation and Basic Usage

# Install
pip install lm-eval

# Run a single benchmark
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-2-7b-hf \
    --tasks mmlu \
    --batch_size auto \
    --output_path results/llama2-7b/

# Run multiple benchmarks
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-2-7b-hf \
    --tasks mmlu,hellaswag,arc_challenge,winogrande,truthfulqa_mc2 \
    --batch_size auto \
    --num_fewshot 5 \
    --output_path results/llama2-7b/

Programmatic API

import lm_eval

results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=meta-llama/Llama-2-7b-hf",
    tasks=["mmlu", "hellaswag", "arc_challenge"],
    num_fewshot=5,
    batch_size="auto",
    device="cuda",
)

# Access results
for task, metrics in results["results"].items():
    print(f"{task}: {metrics}")

Benchmark Selection by Research Claim

Research Claim	Required Benchmarks	Why
General knowledge	MMLU, ARC, TriviaQA	Broad factual coverage
Reasoning	GSM8K, BBH, ARC-Challenge	Multi-step logical reasoning
Coding	HumanEval, MBPP, DS-1000	Code generation and understanding
Instruction following	MT-Bench, AlpacaEval, IFEval	Open-ended instruction quality
Safety	TruthfulQA, ToxiGen, BBQ	Truthfulness, toxicity, bias
Multilingual	MGSM, XWinograd, FLORES	Cross-lingual transfer
Long context	SCROLLS, LongBench, RULER	Long document understanding
Domain-specific	MedQA, LegalBench, SciQ	Professional domain knowledge

Core Benchmarks Deep Dive

MMLU (Massive Multitask Language Understanding)

- 57 subjects: STEM, humanities, social sciences, professional
- 14,042 questions, multiple choice (4 options)
- Standard: 5-shot evaluation
- Metric: Accuracy (macro-averaged across subjects)
- Citation: Hendrycks et al., 2021

Score interpretation:
  < 30%: Below random (model is miscalibrated)
  30-40%: Near random (4 choices = 25% baseline)
  40-60%: Basic knowledge
  60-70%: Strong general knowledge
  70-80%: Expert-level for most subjects
  > 80%: State-of-the-art (as of 2024)

GSM8K (Grade School Math)

- 8,792 grade school math word problems
- Requires multi-step arithmetic reasoning
- Standard: 8-shot chain-of-thought
- Metric: Exact match on final numerical answer
- Citation: Cobbe et al., 2021

Common pitfalls:
  - Regex matching for final answer extraction
  - Calculator use vs. pure model computation
  - Reporting with vs. without chain-of-thought

HumanEval (Code Generation)

- 164 Python programming problems
- Function signature + docstring -> implementation
- Metric: pass@k (k=1 standard, k=10 and k=100 also reported)
- Citation: Chen et al., 2021

pass@k computation (unbiased estimator):
  pass@k = 1 - C(n-c, k) / C(n, k)
  where n = total samples, c = correct samples

Evaluation Pitfalls

Pitfall	Problem	Solution
Data contamination	Benchmark data in training set	Use canary strings, report contamination analysis
Prompt sensitivity	Results vary with prompt format	Report results across 3+ prompt variants
Few-shot selection	Cherry-picked examples boost scores	Use fixed random seed for example selection
Metric gaming	Optimizing for specific metrics	Report multiple metrics, include calibration
Incomplete reporting	Only showing best results	Report mean and std across seeds
Version mismatch	Different benchmark versions	Pin exact dataset version and commit hash

Contamination Detection

def check_contamination(training_data: list, benchmark_data: list, n: int = 13) -> dict:
    """
    Check for n-gram overlap between training data and benchmark.
    13-gram overlap is the standard threshold (GPT-4 technical report).
    """
    from collections import defaultdict

    def extract_ngrams(text, n):
        words = text.lower().split()
        return set(tuple(words[i:i+n]) for i in range(len(words) - n + 1))

    # Build training n-gram index
    train_ngrams = set()
    for text in training_data:
        train_ngrams.update(extract_ngrams(text, n))

    # Check benchmark items
    contaminated = []
    for i, item in enumerate(benchmark_data):
        item_ngrams = extract_ngrams(item, n)
        overlap = item_ngrams & train_ngrams
        if overlap:
            contaminated.append({
                "index": i,
                "overlap_count": len(overlap),
                "overlap_ratio": len(overlap) / max(len(item_ngrams), 1),
            })

    return {
        "total_items": len(benchmark_data),
        "contaminated_items": len(contaminated),
        "contamination_rate": len(contaminated) / len(benchmark_data),
        "details": contaminated,
    }

Reporting Results for Publication

Standard Results Table Format

| Model | Params | MMLU | GSM8K | HumanEval | ARC-C | HellaSwag | Avg |
|-------|--------|------|-------|-----------|-------|-----------|-----|
| Baseline | 7B | 45.2 | 12.3 | 15.8 | 42.1 | 72.3 | 37.5 |
| Ours | 7B | 52.1 (+6.9) | 28.7 (+16.4) | 22.0 (+6.2) | 48.9 (+6.8) | 76.1 (+3.8) | 45.6 |
| Ours (ablation A) | 7B | 49.8 | 24.1 | 19.5 | 46.2 | 74.8 | 42.9 |

All results: 5-shot for MMLU, 8-shot CoT for GSM8K, 0-shot for HumanEval,
25-shot for ARC-C, 10-shot for HellaSwag. Mean of 3 seeds reported.

Best Practices

• Always report the exact evaluation framework version (e.g.,

  lm-eval v0.4.2

  ).
• Use the same number of few-shot examples as the original benchmark paper.
• Report standard deviations across at least 3 random seeds.
• Include a contamination analysis for any new model trained on web data.
• Compare against published numbers using the same evaluation code -- do not mix results from different frameworks.
• Report inference details: precision (fp16/bf16/int8), context length, decoding strategy.

References

• lm-evaluation-harness -- EleutherAI's evaluation framework (12,000+ stars)
• Open LLM Leaderboard -- Hugging Face community leaderboard
• MMLU paper -- Hendrycks et al., 2021
• Holistic Evaluation of Language Models (HELM) -- Stanford CRFM
• Chatbot Arena -- Human preference-based evaluation (LMSYS)