Customization

Before executing, check for user customizations at:

~/.claude/skills/PAI/USER/SKILLCUSTOMIZATIONS/Evals/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:8888/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \
     > /dev/null 2>&1 &
   

2. Output text notification:

   Running the **WorkflowName** workflow in the **Evals** skill to ACTION...
   

This is not optional. Execute this curl command immediately upon skill invocation.

Evals - AI Agent Evaluation Framework

Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).

Key differentiator: Evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.

When to Activate

• "run evals", "test this agent", "evaluate", "check quality", "benchmark"
• "regression test", "capability test"
• Compare agent behaviors across changes
• Validate agent workflows before deployment
• Verify ALGORITHM ISC rows
• Create new evaluation tasks from failures

Core Concepts

Three Grader Types

Evaluation Types

Key Metrics

• pass@k: Probability of at least 1 success in k trials (measures capability)
• pass^k: Probability all k trials succeed (measures consistency/reliability)

Workflow Routing

Tools/AlgorithmBridge.ts

Tools/FailureToTask.ts log

Tools/FailureToTask.ts convert-all

Tools/SuiteManager.ts create

Tools/SuiteManager.ts check-saturation

Quick Reference

CLI Commands

# Run an eval suite
bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts convert-all

# Manage suites
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts list
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts check-saturation <name>
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts graduate <name>

ALGORITHM Integration

Evals is a verification method for THE ALGORITHM ISC rows:

# Run eval and update ISC row
bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u

ISC rows can specify eval verification:

| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |

Available Graders

Code-Based (Fast, Deterministic)

string_match

regex_match

binary_tests

static_analysis

state_check

tool_calls

Model-Based (Nuanced)

llm_rubric

natural_language_assert

pairwise_comparison

Domain Patterns

Pre-configured grader stacks for common agent types:

coding

conversational

research

computer_use

See

Data/DomainPatterns.yaml

Task Schema (YAML)

task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/security_review.md

  trials: 3
  pass_threshold: 0.75

Resource Index

Types/index.ts

Graders/CodeBased/

Graders/ModelBased/

Tools/TranscriptCapture.ts

Tools/TrialRunner.ts

Tools/SuiteManager.ts

Tools/FailureToTask.ts

Tools/AlgorithmBridge.ts

Data/DomainPatterns.yaml

Key Principles (from Anthropic)

1. Start with 20-50 real failures - Don't overthink, capture what actually broke
2. Unambiguous tasks - Two experts should reach identical verdicts
3. Balanced problem sets - Test both "should do" AND "should NOT do"
4. Grade outputs, not paths - Don't penalize valid creative solutions
5. Calibrate LLM judges - Against human expert judgment
6. Check transcripts regularly - Verify graders work correctly
7. Monitor saturation - Graduate to regression when hitting 95%+
8. Build infrastructure early - Evals shape how quickly you can adopt new models

Related

• ALGORITHM: Evals is a verification method
• Science: Evals implements scientific method
• Browser: For visual verification graders