Stata Skill Contributor Guide

This skill helps you contribute to the stata-skill project — the Stata reference skill for Claude Code. It covers how to run the evaluation pipeline, analyze results, and make data-driven improvements to the skill documentation.

Cost Warning

Running the eval pipeline uses Claude Agent SDK

query()

Start small: run 1-3 tasks relevant to your change before running the full suite. Use

--runs 1

Quick Reference

tests/eval.py

tests/rubric.md

tests/tasks/task_*.md

tests/runner-only-plugin/

tests/results/

tests/EVAL_RESULTS_2026-03-22.md

ROADMAP.md

CONTRIBUTING.md

Running the Eval Pipeline

Basic commands

# Single task, single run — fast sanity check (~$1)
python tests/eval.py tests/tasks/task_01_data_cleaning.md

# Single task, 5 runs — measure variance (~$5)
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5

# Save results as a named baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/my_baseline.json

# Compare against a baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --compare tests/results/my_baseline.json

A/B testing a doc change

# 1. Save baseline BEFORE your edit
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --save tests/results/before_mi_fix.json

# 2. Make your edit to the reference doc

# 3. Re-run and compare
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --compare tests/results/before_mi_fix.json

Look for: mean score going up, SD staying flat or going down, no new failure modes.

Testing modes

# Full skill (default) — loads the complete plugin
python tests/eval.py tests/tasks/task_*.md

# Runner-only — minimal plugin with just execution instructions, no reference docs
python tests/eval.py tests/tasks/task_*.md --runner-only

# No skill — no plugin at all
python tests/eval.py tests/tasks/task_*.md --no-skill

# Alternative skill version — point at a different plugin directory
python tests/eval.py tests/tasks/task_*.md --skill-path /path/to/modified/plugin

The runner-only baseline is the meaningful comparison. It isolates what the reference material adds beyond "here's how to run Stata." The no-skill baseline is weaker (agent can't even execute code).

Parallel execution

Run multiple tasks in parallel by launching separate processes:

python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/v3_task01.json &
python tests/eval.py tests/tasks/task_07_did.md --runs 5 \
    --save tests/results/v3_task07.json &
wait

Batch 3-4 tasks at a time to avoid resource exhaustion. Run directories are created atomically, so parallel execution is safe.

Output structure

Each run creates

tests/results/run_NNN/

transcript.json

judge_findings.md

metadata.json

code_files/

Aggregate results are saved to the

--save

Multi-Agent Post-Test Analysis

After running evals, dispatch Sonnet subagents to analyze the results in parallel. This is the workflow that surfaces actionable improvements.

Step 1: Generate the comparison table

# Quick script to build the comparison table from saved results
import json, statistics
for i in range(1, 25):
    tn = f"{i:02d}"
    with open(f"tests/results/v3_task{tn}.json") as f:
        full = [r["score"] for r in json.load(f) if r["score"] is not None]
    with open(f"tests/results/v3_runner_task{tn}.json") as f:
        runner = [r["score"] for r in json.load(f) if r["score"] is not None]
    fm, rm = statistics.mean(full), statistics.mean(runner)
    print(f"Task {tn}: full={fm:.1f} runner={rm:.1f} delta={fm-rm:+.1f}")

Step 2: Dispatch analysis subagents

Launch 4-5 parallel subagents, each investigating a specific pattern in the results. The prompts below are templates — adapt them to your actual results.

Agent 1 — Regression analysis (skill hurts):

Investigate tasks where the full skill scores LOWER than runner-only. Read the result JSONs, the relevant reference docs, and the judge reasoning for low-scoring runs. Identify specific examples in the reference docs that teach wrong patterns. Compare what the with-skill agent does vs the runner-only agent.

Agent 2 — Redundancy analysis (skill adds nothing):

Investigate tasks where full skill and runner-only score identically. Read the reference docs that would be triggered. Determine whether the content is redundant — the model already knows this material from training. Identify which reference files could be deprioritized or trimmed.

Agent 3 — High-variance analysis (unreliable tasks):

Investigate tasks with SD > 10. Compare the LOW-scoring run vs HIGH-scoring run for each. Identify what went wrong — wrong syntax in the doc? Agent misinterpretation? Context collapse (agent wrote code for wrong task)? Quote specific code that failed.

Agent 4 — Biggest wins analysis (what works):

Investigate the 5 tasks with the largest positive deltas. Read the runner-only judge reasoning to identify specific knowledge gaps. Categorize: community package syntax, Stata-specific idioms, workflow patterns, gotcha avoidance.

Agent 5 — Graphics/weak area analysis:

Investigate the lowest-scoring full-skill tasks. Compare successful vs failed runs. Identify missing content in the reference docs.

Step 3: Compile findings and make changes

Each agent returns specific, evidence-backed proposals. Typical findings:

• Bugs in docs: Wrong syntax that the agent copies faithfully (highest priority to fix)
• Missing gotchas: Common errors the docs don't warn about
• Missing workflows: Multi-step patterns the agent needs but can't find
• Redundant content: Topics the model already knows (lowest priority)

Fix bugs first, add missing gotchas second, then re-run to verify improvement.

Example: prior results

See

tests/EVAL_RESULTS_2026-03-22.md

• Found 4 bugs in reference docs (wrong

  mi estimate

  pattern, wrong

  ssc install

  command, wrong variable names)
• Added 3 missing gotchas (

  graph box

  /

  twoway

  incompatibility, DiD event-study syntax,

  nlcom

  clobbers

  e()

  )
• Added 1 missing workflow (standardized coefficients in coefplot)
• Identified the "anchoring effect" — skills keep the agent on-task even when the content is "redundant"

Deeper Engagement: Roadmap

See

ROADMAP.md

• Gotchas-only skill variant: Test whether a stripped-down skill with just gotchas (no reference files) scores close to the full skill. If so, the references are mostly noise.
• Better test prompts: Current prompts are structured step-by-step. Real researcher interactions are messier — goal-oriented, ambiguous, multi-step. Design tests that match real use.
• Harder tests: Several tasks hit ceiling (both arms score 55/55). Need tasks that discriminate — exact numerical verification, edge cases, less-common options.
• The anchoring effect: The skill's biggest contribution isn't always syntax knowledge — it's keeping the agent focused on the correct task. This has implications for how all skills should be designed and evaluated.
• Real-world empirical test suite: Tasks based on actual published papers with replication data.
• Dynamic help file access: Let the agent query Stata's built-in help system at runtime.

Contribution Workflow Summary

1. Identify a gap: Run the eval, read judge findings from low-scoring runs, or pick a roadmap item
2. Save a baseline:

   --runs 5 --save

   on the relevant tasks
3. Make your change: Edit the reference/package doc
4. Re-run and compare:

   --compare

   against the baseline
5. Dispatch analysis agents if the results are surprising or if you want deeper insight
6. Open a PR with your results (scores, variance, before/after). See

   CONTRIBUTING.md

   for details.