Eval Runner

Benchmark the agent's performance against defined scenarios. Adapted from n-trax eval system.

Commands

run <category/name>

1. Read YAML from

   .claude/evals/scenarios/<category>/<name>.yml

2. Parse fields (name, category, task_prompt, success_criteria, budget)
3. Execute setup steps if defined
4. Record start time
5. Execute task via reflexion workflow (read corrections first)
6. Record end time and iteration count
7. Validate ALL success criteria
8. Write result JSON to

   .claude/evals/results/<timestamp>-<name>.json

9. Report summary

run-all [category]

1. Glob

   .claude/evals/scenarios/**/*.yml

2. Skip scenarios with

   status: retired

3. For each: run in isolation (git stash), record result, restore
4. Update

   .claude/evals/pass-history.json

   with each result
5. Aggregate and report

run-split <optimization|holdout>

1. Glob

   .claude/evals/scenarios/**/*.yml

2. Read each YAML, filter by

   split

   field matching the requested set
3. Skip scenarios with

   status: retired

4. For each matching scenario: run in isolation, record result, restore
5. Update

   .claude/evals/pass-history.json

   with each result
6. Aggregate and report (label output clearly as "Optimization Set" or "Holdout Set")

report

1. Read all results from

   .claude/evals/results/

2. Generate summary table:

| Category    | Pass Rate | Avg Iterations | Avg Time | Notes |
|-------------|-----------|----------------|----------|-------|
| discovery   | ...       | ...            | ...      |       |
| delivery    | ...       | ...            | ...      |       |
| integration | ...       | ...            | ...      |       |
| **Overall** | ...       | ...            | ...      |       |

3. List failure patterns and recommendations

prune

1. Read

   .claude/evals/pass-history.json

2. Flag evals where

   last_5

   is all-pass (

   saturated

   ) or all-fail (

   broken

   )
3. Flag evals with no runs in 30+ days (

   stale

   )
4. For saturated evals, suggest: retire or increase difficulty
5. For broken evals, suggest: fix criteria or retire
6. Present recommendations — do NOT auto-retire
7. On user confirmation: set

   status: retired

   in scenario YAML, update pass-history.json, log in decision-log.md

mine

Analyze audit logs to propose new eval scenarios from observed failure patterns.

1. Read

   .claude/state/change-log.jsonl

   (last 100 entries)
2. Read

   .claude/state/diamond-state-audit.jsonl

   (all entries)
3. Group change-log entries by

   session_id

   , identify: a. Correction clusters: 3+ edits to same file in one session (agent struggled) b. Skill friction: edits to

   .claude/skills/*/SKILL.md

   during a session (instructions unclear) c. Missing test coverage: 5+ files changed with no test file edits
4. Count diamond bypass entries from audit log
5. Cross-reference with existing scenarios in pass-history.json to avoid duplicates
6. For each pattern, output a proposed eval scenario as YAML template
7. Tag proposals with

   source: trace-mining

   and originating

   session_id

8. Do NOT auto-create files — present for human review

See

.claude/evals/schema.md

§Trace Mining Heuristics for pattern-to-eval mappings.

Result Format

See

.claude/evals/schema.md

for YAML scenario and JSON result formats.

Pass History

After writing each result JSON (step 8 of

run

), also update

.claude/evals/pass-history.json

:

• Increment

  runs

  and

  passes

  (if passed) for the eval
• Append

  true

  /

  false

  to

  last_5

  (trim to keep only last 5)
• Update

  last_run

  timestamp

Creating Scenarios

Place YAML files in

.claude/evals/scenarios/<category>/

. Define task_prompt, success_criteria, and budget. Set

split

,

status

, and

source

fields per schema.