Skill Tester

The agent validates skill packages for structure compliance, tests Python scripts for syntax and stdlib-only imports, and scores quality across four dimensions (documentation, code quality, completeness, usability) with letter grades and improvement recommendations. It supports BASIC, STANDARD, and POWERFUL tier classification.

Quick Start

# Validate skill structure and documentation
python skill_validator.py engineering/my-skill --tier POWERFUL --json

# Test all Python scripts in a skill
python script_tester.py engineering/my-skill --timeout 30

# Score quality with improvement roadmap
python quality_scorer.py engineering/my-skill --detailed --minimum-score 75

Core Workflows

Workflow 1: Validate a New Skill

1. Run

   skill_validator.py

   with target tier to check structure, frontmatter, required sections, and scripts
2. Review errors (blocking) and warnings (non-blocking) in the report
3. Fix all errors -- missing SKILL.md, invalid frontmatter, external imports
4. Validation checkpoint: Score >= 60; zero errors; all scripts pass

   ast.parse()

python skill_validator.py engineering/my-skill --tier STANDARD --json

Workflow 2: Test Skill Scripts

1. Run

   script_tester.py

   to execute syntax validation, import analysis, and runtime tests
2. Review per-script results: argparse detection,

   --help

   output, sample data execution
3. Fix failures: add

   if __name__ == "__main__"

   guards, replace external imports with stdlib
4. Validation checkpoint: All scripts pass syntax; zero external imports;

   --help

   exits cleanly

python script_tester.py engineering/my-skill --timeout 60 --json

Workflow 3: Score and Improve Quality

1. Run

   quality_scorer.py

   with

   --detailed

   for component-level breakdowns
2. Review the prioritized improvement roadmap (up to 5 items)
3. Address HIGH-priority items first (documentation gaps, missing error handling)
4. Re-run to verify score improvement
5. Validation checkpoint: Overall score >= 75; no dimension below 50%

python quality_scorer.py engineering/my-skill --detailed --minimum-score 75 --json

Tier Requirements

Quality Scoring Dimensions

Grades: A+ (97+) through F (<40). Exit code 0 for A+ through C-, exit code 2 for D, exit code 1 for F.

CI/CD Integration

# GitHub Actions example
- name: Validate Changed Skills
  run: |
    for skill in $(git diff --name-only | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u); do
      python engineering/skill-tester/scripts/skill_validator.py $skill --json
      python engineering/skill-tester/scripts/script_tester.py $skill
      python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75
    done

Anti-Patterns

• Padding SKILL.md with filler -- line count thresholds measure substantive content; blank lines and boilerplate do not count
• External imports disguised as stdlib -- the import allowlist is manually maintained; if a legit stdlib module is flagged, add it to

  stdlib_modules

• Missing argparse help strings -- usability scoring requires

  help=

  parameters on every argument; empty help strings score zero
• No

  __main__

  guard -- scripts without

  if __name__ == "__main__"

  fail runtime tests when imported
• Relying on SKILL.md for usability -- usability is scored from scripts and README independently; a detailed SKILL.md does not compensate for missing

  --help

  output

Troubleshooting

SKILL.md too short

---

yaml.safe_load()

---

skill_validator.py

script_tester.py

stdlib_modules

--timeout

_validate_tier_compliance

skill_md_exists

min_scripts_count

skill_md_length

TIER_REQUIREMENTS

README.md

argparse

help=

Usage

assets/

--json

--verbose

Success Criteria

• Structure pass rate above 95%: Validated skills pass all required-file and directory-structure checks on first run in at least 95% of cases.
• Script syntax zero-defect: Every Python script in a validated skill compiles without

  SyntaxError

  via

  ast.parse()

  .
• Standard library compliance 100%: No external (non-stdlib) imports detected across all validated scripts.
• Quality score consistency within 5 points: Re-running

  quality_scorer.py

  on an unchanged skill produces scores that vary by no more than 5 points across runs.
• Execution time under 10 seconds per skill: Full validation, testing, and scoring pipeline completes in under 10 seconds for a single skill with up to 3 scripts.
• Actionable recommendation density: Every skill scoring below 75/100 receives at least 3 prioritized improvement suggestions in the roadmap.
• CI/CD gate reliability: When integrated as a GitHub Actions step, the tool exits with non-zero status for every skill that fails critical checks, blocking the merge.

Scope & Limitations

Covers:

• Structural validation of skill directories against tier-specific requirements (BASIC, STANDARD, POWERFUL)
• Static analysis of Python scripts including syntax checking, import validation, argparse detection, and main guard verification
• Multi-dimensional quality scoring across documentation, code quality, completeness, and usability
• Dual output formatting (JSON for CI/CD pipelines, human-readable for developer consumption)

Does NOT cover:

• Functional correctness of script logic or algorithm accuracy — the tester verifies structure and conventions, not business logic
• Performance benchmarking or memory profiling of scripts — see

  engineering/performance-profiler

  for runtime analysis
• Security vulnerability scanning of script code — see

  engineering/skill-security-auditor

  for dependency and code security audits
• Cross-skill dependency resolution or integration testing — skills are validated in isolation without verifying inter-skill compatibility

Integration Points

engineering/skill-security-auditor

skill_validator.py

skill-security-auditor

engineering/ci-cd-pipeline-builder

skill_validator.py

script_tester.py

quality_scorer.py

engineering/changelog-generator

quality_scorer.py

engineering/pr-review-expert

skill_validator.py --json

engineering/performance-profiler

script_tester.py

performance-profiler

engineering/tech-debt-tracker

quality_scorer.py --json

Tool Reference

skill_validator.py

Purpose: Validates a skill directory's structure, documentation, and Python scripts against the claude-skills ecosystem standards. Checks required files, YAML frontmatter, required SKILL.md sections, directory layout, script syntax, import compliance, and tier-specific requirements.

Usage:

python skill_validator.py <skill_path> [--tier TIER] [--json] [--verbose]

Parameters:

skill_path

--tier

BASIC

STANDARD

POWERFUL

--json

--verbose

Example:

python skill_validator.py engineering/my-skill --tier POWERFUL --json

Output Formats:

• Human-readable (default): Grouped report with STRUCTURE VALIDATION, SCRIPT VALIDATION, ERRORS, WARNINGS, and SUGGESTIONS sections. Displays overall score out of 100 with compliance level (EXCELLENT, GOOD, ACCEPTABLE, NEEDS_IMPROVEMENT, POOR).
• JSON (

  --json

  ): Object with keys

  skill_path

  ,

  timestamp

  ,

  overall_score

  ,

  compliance_level

  ,

  checks

  (dict of check name to pass/message/score),

  warnings

  ,

  errors

  ,

  suggestions

  .

Exit codes:

0

1

script_tester.py

Purpose: Tests all Python scripts within a skill's

scripts/

--help

assets/

Usage:

python script_tester.py <skill_path> [--timeout SECONDS] [--json] [--verbose]

Parameters:

skill_path

--timeout

30

--json

--verbose

Example:

python script_tester.py engineering/my-skill --timeout 60 --json

Output Formats:

• Human-readable (default): Report with SUMMARY (total/passed/partial/failed counts), GLOBAL ERRORS, and per-script sections showing status, execution time, individual test results, errors, and warnings.
• JSON (

  --json

  ): Object with keys

  skill_path

  ,

  timestamp

  ,

  summary

  (counts and overall status),

  global_errors

  ,

  script_results

  (dict per script with

  overall_status

  ,

  execution_time

  ,

  tests

  ,

  errors

  ,

  warnings

  ).

Exit codes:

0

1

2

quality_scorer.py

Purpose: Provides a comprehensive multi-dimensional quality assessment for a skill. Evaluates four equally weighted dimensions — Documentation (25%), Code Quality (25%), Completeness (25%), and Usability (25%) — and produces an overall score, letter grade (A+ through F), tier recommendation, and a prioritized improvement roadmap.

Usage:

python quality_scorer.py <skill_path> [--detailed] [--minimum-score SCORE] [--json] [--verbose]

Parameters:

skill_path

--detailed

--minimum-score

0

1

--json

--verbose

Example:

python quality_scorer.py engineering/my-skill --detailed --minimum-score 75 --json

Output Formats:

• Human-readable (default): Report with overall score and letter grade, per-dimension scores with weights, summary statistics (highest/lowest dimension, dimensions above 70%, dimensions below 50%), and a prioritized improvement roadmap (up to 5 items with HIGH/MEDIUM/LOW priority). When

  --detailed

  is used, component-level breakdowns appear under each dimension.
• JSON (

  --json

  ): Object with keys

  skill_path

  ,

  timestamp

  ,

  overall_score

  ,

  letter_grade

  ,

  tier_recommendation

  ,

  summary_stats

  ,

  dimensions

  (per-dimension name/weight/score/details/suggestions),

  improvement_roadmap

  (list of priority/dimension/suggestion/current_score objects).

Exit codes:

0

1

--minimum-score

2