Plugin Development Workflow

This repo is the Elixir/Phoenix Claude Code plugin. When editing plugin files, follow this workflow to ensure quality.

Before You Start

Run

make help

to see all available commands:

make eval          # Quick: lint + score changed skills/agents
make eval-all      # Full: all 40 skills + 20 agents
make eval-fix      # Auto-fix + show failures
make test          # 52 pytest tests for eval framework
make ci            # Full CI pipeline

Scoring Individual Files (CLI)

IMPORTANT: Always use

-m

module syntax, never run scorer.py directly.

# Score ONE skill (use -m, NOT direct file path)
python3 -m lab.eval.scorer plugins/elixir-phoenix/skills/verify/SKILL.md

# Score ONE skill with pretty output
python3 -m lab.eval.scorer plugins/elixir-phoenix/skills/verify/SKILL.md --pretty

# Score all skills
python3 -m lab.eval.scorer --all

# Score ONE agent
python3 -m lab.eval.agent_scorer plugins/elixir-phoenix/agents/verification-runner.md

# Score all agents
python3 -m lab.eval.agent_scorer --all
make ci            # Full CI pipeline

When Editing Skills (plugins/elixir-phoenix/skills/*/SKILL.md)

1. Read CLAUDE.md conventions (size limits, frontmatter requirements)
2. Make your changes
3. Run

   make eval

   — it auto-detects changed skills and scores them
4. If FAIL: check the dimension that failed, fix it
5. Run

   make lint

   to verify markdown formatting
6. Commit

Skill requirements (eval checks all of these):

• Frontmatter: name, description, effort. Description must start with action verb + include "Use when..."
• Iron Laws section with 1+ numbered items
• Under 185 lines (command skills) or 150 lines (reference skills)
• No section exceeds 45 lines
• All

  /phx:

  references point to existing skills
• All

  references/*.md

  paths exist
• No dangerous code patterns outside Iron Laws sections
• Code examples present (1+ fenced code blocks)
• "Use when..." in description (for trigger accuracy)

When Editing Agents (plugins/elixir-phoenix/agents/*.md)

1. Make your changes
2. Run

   make eval-agents

   to score all agents
3. Agent requirements:

   • permissionMode: bypassPermissions

     (always — background agents need it)

   • disallowedTools: Write, Edit, NotebookEdit

     for review/analysis agents
   • model matches effort: haiku=low, sonnet=medium, opus=high
   • Under 300 lines (specialist) or 535 lines (orchestrator)

When Editing Eval Framework (lab/eval/*.py)

1. Make your changes
2. Run

   make test

   — 52 pytest tests must pass
3. Run

   make eval-all

   — verify no skills/agents regressed
4. If adding new matchers: add tests in

   lab/eval/tests/test_matchers.py

When Editing Hooks (plugins/elixir-phoenix/hooks/scripts/*.sh)

1. Make your changes
2. Run

   make lint

   (markdown in hook comments)
3. Test the hook manually (hooks run on Edit/Write/Bash events)
4. Check CLAUDE.md hook documentation is still accurate

Autoresearch (Self-Improvement Loop)

If

make eval-fix

shows failures, it suggests an autoresearch command:

# Copy-paste the suggested command from eval-fix output
claude -p 'Run autoresearch. Score all skills...' --allowedTools 'Edit,Read,Write,Bash,Glob,Grep'

This runs the autoresearch loop: find weakest skill → fix ONE issue → re-score → keep/revert.

Pre-Commit Checklist

Before committing any plugin changes:

• make lint

  passes

• make eval

  passes (changed files)

• make test

  passes (if eval framework changed)
• CHANGELOG.md updated (if user-visible change)
• Version bumped in plugin.json (if releasing)

References

• CLAUDE.md — full conventions, size limits, checklist

• lab/eval/

  — scoring framework (24 matchers, 8 dimensions)

• lab/autoresearch/

  — self-improvement loop

• lab/findings/interesting.jsonl

  — log interesting discoveries here