Open-agreements recipe-quality-audit

Name: recipe-quality-audit
Author: open-agreements

install

source · Clone the upstream repo

git clone https://github.com/open-agreements/open-agreements

Claude Code · Install into ~/.claude/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/open-agreements/open-agreements "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/recipe-quality-audit" ~/.claude/skills/open-agreements-open-agreements-recipe-quality-audit && rm -rf "$T"

manifest: skills/recipe-quality-audit/SKILL.md

source content

recipe-quality-audit

Audit a single NVCA recipe's quality and produce a structured scorecard.

Security model

This skill operates on local repository files only — no network access required.
Source document downloads (Tier 2 checks) use
```
ensureSourceDocx()
```
which fetches from known template source URLs only.
No credentials or API keys are needed.

Usage

Run the audit for a specific recipe:

Audit the recipe: nvca-certificate-of-incorporation

Or audit all recipes:

Audit all NVCA recipes and update the quality tracker

Checks

Tier 1: Structural (no source download needed)

#	Check	How
S1	File inventory	Does recipe have metadata.yaml, replacements.json, clean.json? Optional: computed.json, normalize.json, selections.json
S2	Metadata schema valid	Run existing `validateRecipeMetadata()` from `src/core/metadata.ts`
S3	Field-to-replacement coverage	For each field in metadata, is there a replacement key referencing `{field_name}` ?
S4	Ambiguous keys	Flag replacement keys < 8 chars without context qualifier (e.g., `[name]` , `[its]` )
S5	Smart quote coverage	Keys with apostrophes should have smart-quote variants (or patcher normalizes — check patcher has normalizeQuotes)
S6	Source SHA present	`source_sha256` in metadata.yaml
S7	Test fixture exists	`integration-tests/fixtures/{recipe-id}-*.json` exists

Tier 2: Behavioral (requires source download)

#	Check	How
B1	Source download + scan	Count all `\[[_A-Z]` prefixed bracket patterns in source (underscore-fill or capitalized placeholders)
B2	Replacement coverage ratio	(keys with match in source) / (total bracket patterns). Target: >80%
B3	Unmatched underscore patterns	`[___+]` patterns in source not in replacements.json
B4	Clean effectiveness	After clean, no footnotes, no "Note to Drafter", no preamble

Tier 3: Fill quality (requires fill run)

#	Check	How
F1	Default-only fill	Fill with defaults, run verifyOutput, count blank placeholders
F2	Full-values fill	Fill with all fields from test fixture, assert all verify checks pass
F3	Formatting anomaly count	Check 8 from verifier (single-char underlined runs)
F4	Zero-match replacement keys	Keys that existed in replacements.json but matched nothing in the source

Output: Quality Scorecard

{
  "recipe_id": "nvca-voting-agreement",
  "maturity": "beta",
  "scores": { "structural": "6/7", "behavioral": "3/4", "fill": "0/4", "total": "9/15" },
  "checks": [
    { "id": "S1", "name": "File inventory", "passed": true, "details": "metadata.yaml, replacements.json, clean.json present" },
    { "id": "S7", "name": "Test fixture exists", "passed": false, "details": "No fixture matching integration-tests/fixtures/nvca-voting-agreement-*.json" }
  ],
  "field_coverage": { "metadata_fields": 14, "replacement_refs": 10, "uncovered": 4 },
  "recommendations": [
    "Add test fixture for fill testing (S7)",
    "Add replacement keys for 4 uncovered fields (S3)"
  ]
}

Maturity Tiers

scaffold: metadata-only, can't fill
beta: has replacements + clean, score < 11/15 OR no test fixture
production: score >= 11/15 AND has test fixture AND has computed.json (if conditional sections exist in source)

Workflow

When running the audit:

Read
```
content/recipes/QUALITY_TRACKER.md
```
for current state
Run Tier 1 checks (always possible)
Run Tier 2 checks if source document is available (use
```
ensureSourceDocx()
```
)
Run Tier 3 checks if a test fixture exists
Compute scorecard and maturity tier
Output the scorecard as JSON
Update the quality tracker if requested

Implementation Notes

Use

validateRecipeMetadata()

from

src/core/metadata.ts

for S2

Use

ensureSourceDocx()

from

src/core/recipe/downloader.ts

for B1-B4

Use
```
runRecipe()
```
from
```
src/core/recipe/index.ts
```
for F1-F4
Bracket pattern detection uses
```
\[[_A-Z]
```
prefix to avoid counting citations and legal references
Zero-match keys come from
```
PatchResult.zeroMatchKeys
```
returned by the patcher
Cross-reference zero-match keys with
```
cleanConfig.removeRanges
```
and
```
cleanConfig.removeParagraphPatterns
```
to suppress expected zero-matches