Auto Arena Skill

End-to-end automated model comparison using the OpenJudge

AutoArenaPipeline

1. Generate queries — LLM creates diverse test queries from task description
2. Collect responses — query all target endpoints concurrently
3. Generate rubrics — LLM produces evaluation criteria from task + sample queries
4. Pairwise evaluation — judge model compares every model pair (with position-bias swap)
5. Analyze & rank — compute win rates, win matrix, and rankings
6. Report & charts — Markdown report + win-rate bar chart + optional matrix heatmap

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for auto_arena (chart generation)
pip install matplotlib

Gather from user before running

gpt-4

qwen-max

OPENAI_API_KEY

DASHSCOPE_API_KEY

20

./evaluation_results

"zh"

"en"

Quick start

CLI

# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save

# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
  --queries_file queries.json --save

# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save

# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save

Python API

import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline

async def main():
    pipeline = AutoArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    print(f"Best model: {result.best_pipeline}")
    for rank, (model, win_rate) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {win_rate:.1%}")

asyncio.run(main())

Minimal Python API (no config file)

import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
from cookbooks.auto_arena.schema import OpenAIEndpoint

async def main():
    pipeline = AutoArenaPipeline(
        task_description="Customer service chatbot for e-commerce",
        target_endpoints={
            "gpt4": OpenAIEndpoint(
                base_url="https://api.openai.com/v1",
                api_key="sk-...",
                model="gpt-4",
            ),
            "qwen": OpenAIEndpoint(
                base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
                api_key="sk-...",
                model="qwen-max",
            ),
        },
        judge_endpoint=OpenAIEndpoint(
            base_url="https://api.openai.com/v1",
            api_key="sk-...",
            model="gpt-4",
        ),
        num_queries=20,
    )
    result = await pipeline.evaluate()
    print(f"Best: {result.best_pipeline}")

asyncio.run(main())

CLI options

--config

--output_dir

--queries_file

--save

False

--fresh

False

--rerun-judge

False

Minimal config file

task:
  description: "Academic GPT assistant for research and writing tasks"

target_endpoints:
  model_v1:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4"
  model_v2:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-3.5-turbo"

judge_endpoint:
  base_url: "https://api.openai.com/v1"
  api_key: "${OPENAI_API_KEY}"
  model: "gpt-4"

Full config reference

task

description

scenario

target_endpoints.<name>

base_url

api_key

${ENV_VAR}

model

system_prompt

extra_params

temperature

max_tokens

judge_endpoint

Same fields as

target_endpoints.<name>

gpt-4

qwen-max

query_generation

num_queries

20

seed_queries

categories

endpoint

queries_per_call

10

num_parallel_batches

3

temperature

0.9

top_p

0.95

max_similarity

0.85

enable_evolution

false

evolution_rounds

1

complexity_levels

["constraints", "reasoning", "edge_cases"]

evaluation

max_concurrency

10

timeout

60

retry_times

3

output

output_dir

./evaluation_results

save_queries

true

save_responses

true

save_details

true

report

enabled

false

language

"zh"

"zh"

"en"

include_examples

3

chart.enabled

true

chart.orientation

"horizontal"

"horizontal"

"vertical"

chart.show_values

true

chart.highlight_best

true

chart.matrix_enabled

false

chart.format

"png"

"png"

"svg"

"pdf"

Interpreting results

Win rate: percentage of pairwise comparisons a model wins. Each pair is evaluated in both orders (original + swapped) to eliminate position bias.

Rankings example:

  1. gpt4_baseline       [################----] 80.0%
  2. qwen_candidate      [############--------] 60.0%
  3. llama_finetuned      [##########----------] 50.0%

Win matrix:

win_matrix[A][B]

Checkpoint & resume

The pipeline saves progress after each step. Interrupted runs resume automatically:

• --fresh

  — ignore checkpoint, start from scratch

• --rerun-judge

  — re-run only the pairwise evaluation step (useful when switching judge models); keeps queries, responses, and rubrics intact
• Adding new endpoints to config triggers incremental response collection; existing responses are preserved

Output files

evaluation_results/
├── evaluation_results.json     # Rankings, win rates, win matrix
├── evaluation_report.md        # Detailed Markdown report (if enabled)
├── win_rate_chart.png          # Win-rate bar chart (if enabled)
├── win_rate_matrix.png         # Matrix heatmap (if matrix_enabled)
├── queries.json                # Generated test queries
├── responses.json              # All model responses
├── rubrics.json                # Generated evaluation rubrics
├── comparison_details.json     # Pairwise comparison details
└── checkpoint.json             # Pipeline checkpoint

API key by model

gpt-*

o1-*

o3-*

OPENAI_API_KEY

claude-*

ANTHROPIC_API_KEY

qwen-*

dashscope/*

DASHSCOPE_API_KEY

deepseek-*

DEEPSEEK_API_KEY

api_key

base_url

Additional resources

• Full config examples: cookbooks/auto_arena/examples/
• Documentation: Auto Arena Guide