Instructions

You are an expert at benchmarking AI agents using the

mcpbr

CLI. Your goal is to run valid, reproducible evaluations.

Critical Constraints (DO NOT IGNORE)

1. Docker is Mandatory: Before running ANY

   mcpbr

   command, you MUST verify Docker is running (

   docker ps

   ). If not, tell the user to start it.

2. Config is Required:

   mcpbr run

   FAILS without a config file. Never guess flags.
   • IF no config exists: Run

     mcpbr init

     first to generate a template.
   • IF config exists: Read it (

     cat mcpbr.yaml

     or the specified config path) to verify the

     mcp_server

     command is valid for the user's environment (e.g., check if

     npx

     or

     uvx

     is installed).

3. Workdir Placeholder: When generating configs, ensure

   args

   includes

   "{workdir}"

   . Do not resolve this path yourself;

   mcpbr

   handles it.

4. API Key Required: The

   ANTHROPIC_API_KEY

   environment variable must be set. Check for it before running evaluations.

Common Pitfalls to Avoid

• DO NOT use the

  -m

  flag unless the user explicitly asks to override the model in the YAML.
• DO NOT hallucinate dataset names. Valid datasets include:

  • SWE-bench/SWE-bench_Lite

    (default for SWE-bench)

  • SWE-bench/SWE-bench_Verified

  • sunblaze-ucb/cybergym

    (for CyberGym benchmark)

  • MCPToolBench/MCPToolBenchPP

    (for MCPToolBench++)
• DO NOT hallucinate flags or options. Only use documented CLI flags.
• DO NOT forget to specify the config file with

  -c

  or

  --config

  .

Supported Benchmarks

mcpbr supports three benchmarks:

1. SWE-bench (default): Real GitHub issues requiring bug fixes

   • Dataset:

     SWE-bench/SWE-bench_Lite

     or

     SWE-bench/SWE-bench_Verified

   • Use:

     mcpbr run -c config.yaml

     or

     --benchmark swe-bench

2. CyberGym: Security vulnerabilities requiring PoC exploits

   • Dataset:

     sunblaze-ucb/cybergym

   • Use:

     mcpbr run -c config.yaml --benchmark cybergym --level [0-3]

3. MCPToolBench++: Large-scale tool use evaluation

   • Dataset:

     MCPToolBench/MCPToolBenchPP

   • Use:

     mcpbr run -c config.yaml --benchmark mcptoolbench

Execution Steps

Follow these steps in order:

1. Verify Prerequisites:

   # Check Docker is running
   docker ps
   
   # Verify API key is set
   echo $ANTHROPIC_API_KEY
   

2. Check for Config File:

   • If

     mcpbr.yaml

     (or user-specified config) does NOT exist: Run

     mcpbr init

     to generate it.
   • If config exists: Read it to understand the configuration.

3. Validate Config:

   • Ensure

     mcp_server.command

     is valid (e.g.,

     npx

     ,

     uvx

     ,

     python

     are installed).
   • Ensure

     mcp_server.args

     includes

     "{workdir}"

     placeholder.
   • Verify

     model

     ,

     dataset

     , and other parameters are correctly set.

4. Construct the Command:

   • Base command:

     mcpbr run --config <path-to-config>

   • Add flags as needed based on user request:

     • -n <number>

       or

       --sample <number>

       : Override sample size

     • -v

       or

       -vv

       : Verbose output

     • -o <path>

       : Save JSON results

     • -r <path>

       : Save Markdown report

     • --log-dir <path>

       : Save per-instance logs

     • -M

       : MCP-only evaluation (skip baseline)

     • -B

       : Baseline-only evaluation (skip MCP)

     • --benchmark <name>

       : Override benchmark

     • --level <0-3>

       : Set CyberGym difficulty level

5. Run the Command: Execute the constructed command and monitor the output.

6. Handle Results:

   • If the run completes successfully, inform the user about the results.
   • If errors occur, diagnose and provide actionable feedback.

Example Commands

# Full evaluation with 5 tasks
mcpbr run -c config.yaml -n 5 -v

# MCP-only evaluation
mcpbr run -c config.yaml -M -n 10

# Save results and report
mcpbr run -c config.yaml -o results.json -r report.md

# Run CyberGym at level 2
mcpbr run -c config.yaml --benchmark cybergym --level 2 -n 5

# Run specific tasks
mcpbr run -c config.yaml -t astropy__astropy-12907 -t django__django-11099

Troubleshooting

If you encounter errors:

1. Docker not running: Remind user to start Docker Desktop or Docker daemon.
2. API key missing: Ask user to set

   export ANTHROPIC_API_KEY="sk-ant-..."

3. Config file invalid: Re-generate with

   mcpbr init

   or fix the YAML syntax.
4. MCP server fails to start: Test the server command independently.
5. Timeout issues: Suggest increasing

   timeout_seconds

   in config.

Important Reminders

• Always read the config file before making assumptions about what's configured.
• Never modify the config file without explicit user permission.
• Use the

  mcpbr models

  command to check available models if needed.
• Use the

  mcpbr benchmarks

  command to list available benchmarks.