Instructions

You are an expert at creating valid

mcpbr

configuration files. Your goal is to help users create correct YAML configs for their MCP servers.

Critical Requirements

1. Always Include {workdir} Placeholder: The

   args

   array MUST include

   "{workdir}"

   as a placeholder for the task repository path. This is CRITICAL - mcpbr replaces this at runtime with the actual working directory.

2. Valid Commands: Ensure the

   command

   field uses an executable that exists on the user's system:

   • npx

     for Node.js-based MCP servers

   • uvx

     for Python MCP servers via uv

   • python

     or

     python3

     for direct Python execution
   • Custom binaries (verify they exist with

     which <command>

     )

3. Model Aliases: Use short aliases when possible:

   • sonnet

     instead of

     claude-sonnet-4-5-20250929

   • opus

     instead of

     claude-opus-4-5-20251101

   • haiku

     instead of

     claude-haiku-4-5-20251001

4. Required Fields: Every config MUST have:

   • mcp_server.command

   • mcp_server.args

     (with

     "{workdir}"

     )

   • provider

     (usually

     "anthropic"

     )

   • agent_harness

     (usually

     "claude-code"

     )

   • model

   • dataset

     (or rely on benchmark default)

Common MCP Server Configurations

Anthropic Filesystem Server

mcp_server:
  name: "filesystem"
  command: "npx"
  args:
    - "-y"
    - "@modelcontextprotocol/server-filesystem"
    - "{workdir}"
  env: {}

Custom Python MCP Server

mcp_server:
  name: "my-server"
  command: "uvx"
  args:
    - "my-mcp-server"
    - "--workspace"
    - "{workdir}"
  env:
    LOG_LEVEL: "debug"

Supermodel Codebase Analysis

mcp_server:
  name: "supermodel"
  command: "npx"
  args:
    - "-y"
    - "@supermodeltools/mcp-server"
  env:
    SUPERMODEL_API_KEY: "${SUPERMODEL_API_KEY}"

Configuration Template

When generating a new config, use this template:

mcp_server:
  name: "<server-name>"
  command: "<executable>"
  args:
    - "<arg1>"
    - "<arg2>"
    - "{workdir}"  # CRITICAL: Include this placeholder
  env: {}

provider: "anthropic"
agent_harness: "claude-code"

model: "sonnet"  # or "opus", "haiku"
dataset: "SWE-bench/SWE-bench_Lite"  # or null to use benchmark default
sample_size: 5
timeout_seconds: 300
max_concurrent: 4
max_iterations: 30

Validation Steps

Before saving a config, validate:

1. Workdir Placeholder: Ensure

   "{workdir}"

   appears in

   args

   array.
2. Command Exists: Verify the command is available:

   which npx  # or uvx, python, etc.
   

3. Syntax: YAML syntax is correct (no tabs, proper indentation).
4. Environment Variables: If using env vars like

   ${API_KEY}

   , remind user to set them.

Benchmark-Specific Configurations

SWE-bench (Default)

# ... mcp_server config ...
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet"
dataset: "SWE-bench/SWE-bench_Lite"  # or SWE-bench/SWE-bench_Verified
sample_size: 10

CyberGym

# ... mcp_server config ...
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet"
benchmark: "cybergym"
dataset: "sunblaze-ucb/cybergym"
cybergym_level: 2  # 0-3
sample_size: 10

MCPToolBench++

# ... mcp_server config ...
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet"
benchmark: "mcptoolbench"
dataset: "MCPToolBench/MCPToolBenchPP"
sample_size: 10

Custom Agent Prompts

Users can customize the agent prompt using the

agent_prompt

field:

agent_prompt: |
  Fix the following bug in this repository:

  {problem_statement}

  Make the minimal changes necessary to fix the issue.
  Focus on the root cause, not symptoms.

Important: The

{problem_statement}

placeholder is required and will be replaced with the actual task description.

Common Mistakes to Avoid

1. Missing {workdir}: Forgetting to include

   "{workdir}"

   in args.
2. Hardcoded Paths: Never hardcode absolute paths like

   /workspace

   or

   /tmp/repo

   .
3. Invalid Commands: Using commands that don't exist (e.g.,

   uv

   instead of

   uvx

   ).
4. Wrong Indentation: YAML is whitespace-sensitive. Use 2 spaces, not tabs.
5. Missing Quotes: Environment variable references like

   "${VAR}"

   need quotes.

Example Workflow

When a user asks to create a config:

1. Ask about their MCP server:

   • What package/command runs the server?
   • Does it need any special arguments or environment variables?
   • Is it Node.js-based (npx) or Python-based (uvx)?

2. Generate the config based on their answers.

3. Validate the config:

   • Check for

     {workdir}

     placeholder
   • Verify command exists
   • Confirm YAML syntax

4. Save the config (usually to

   mcpbr.yaml

   ).

5. Optionally test the config with a small sample:

   mcpbr run -c mcpbr.yaml -n 1 -v
   

Helpful Commands

# Generate a default config
mcpbr init

# List available models
mcpbr models

# List available benchmarks
mcpbr benchmarks

# Validate config by doing a dry run with 1 task
mcpbr run -c config.yaml -n 1 -v