Distill

Overview

All subagent dispatches use disk-mediated dispatch. See

shared/dispatch-convention.md

Convert heavy document formats to token-efficient representations (Markdown, CSV) for LLM consumption. The core deliverable is the

.digest.md

Skill type: Rigid — follow exactly, no shortcuts.

Models:

• PDF structuring agent: Sonnet
• Digest agent: Sonnet
• Orchestrator: runs on whatever model the session uses

Announce at start: "I'm using the distill skill to convert documents to token-efficient formats."

Invocation API

/distill <path> [path2 ...]
/distill <directory>

Examples:

• /distill docs/report.pdf

  — convert one file

• /distill docs/report.pdf data/sheet.xlsx slides/deck.pptx

  — convert multiple files

• /distill docs/

  — convert all supported files in directory (single-level, not recursive)

Mixed mode is supported:

/distill docs/ extra/report.pdf

The Process

Execute phases in this order. Each phase completes for all files before the next begins.

Phase 0: Tool Availability Check

At skill start, before processing any files, check for required tools:

which pandoc

apt install pandoc

brew install pandoc

which pdftotext

apt install poppler-utils

brew install poppler

which python3

which unzip

which pdfdetach

Build a set of available tiers. Route files only to available tiers. Files targeting unavailable tiers get routed to unsupported-with-guidance (Phase 1b).

Phase 1: Input Resolution

1a: Build File List

Individual file paths: Use directly. Verify each file exists.

Directory paths: Single-level glob for files with supported extensions (not recursive). Build file list sorted alphabetically. Report: "Found {N} convertible files in {directory}: {list}."

Supported extensions for glob:

.pdf

.docx

.rtf

.html

.htm

.odt

.epub

.rst

.org

.tex

.ipynb

.pptx

.xlsx

Mixed mode: Process both directory globs and individual paths. Deduplicate by absolute path.

1b: Route Files to Tiers

For each file, determine the conversion tier by extension:

.docx

docx

.rtf

rtf

.html

html

.htm

html

.odt

odt

.epub

epub

.rst

rst

.org

org

.tex

latex

.ipynb

ipynb

.pdf

.pptx

.xlsx

Unsupported formats: Output actionable guidance per this table, then continue with remaining files:

.xls

.ods

.odp

.key

.numbers

.pages

Unknown extensions: "Unsupported format: {ext}. Supported formats: docx, rtf, html, odt, epub, rst, org, tex, ipynb, pdf, pptx, xlsx."

Unavailable tier: If a file's tier is unavailable (tool missing from Phase 0), report: "{file}: requires {tool} (not installed). Skipping."

Phase 2: Pre-Flight Checks

Run per-file safety checks before conversion. Failures are per-file — do not halt the batch.

Zip Bomb Detection (docx, pptx, xlsx)

Office formats are ZIP archives. If

unzip

UNCOMPRESSED=$(unzip -l "$INPUT_PATH" 2>/dev/null | tail -1 | awk '{print $1}')

If uncompressed size exceeds 500MB (524288000 bytes), abort this file: "File uncompressed size ({size}) exceeds 500MB safety limit. Skipping."

If

unzip

PDF Attachment Detection

For PDF files, if

pdfdetach

ATTACHMENTS=$(pdfdetach -list "$INPUT_PATH" 2>/dev/null | grep -c "^[0-9]")

If attachments found, warn: "PDF contains {N} embedded attachments. These are not extracted — only text content is converted." Continue with conversion.

Encoding Validation

After conversion (not before), verify output is valid UTF-8:

file --mime-encoding "$OUTPUT_PATH"

If not UTF-8, attempt re-encoding:

iconv -f <detected-charset> -t UTF-8 "$OUTPUT_PATH" -o "$OUTPUT_PATH.tmp" && mv "$OUTPUT_PATH.tmp" "$OUTPUT_PATH"

Phase 3: Conversion

Process files sequentially. For each file:

Tier 1: Pandoc-Native

INPUT_PATH="$1"
OUTPUT_PATH="${INPUT_PATH%.*}.md"
FORMAT="$2"  # from routing table

pandoc -f "$FORMAT" -t markdown --wrap=none "$INPUT_PATH" -o "$OUTPUT_PATH"

Shell safety: All file paths via quoted shell variables. Never inline interpolation. Never use unquoted

$()

Error handling:

• Non-zero exit code: report "pandoc conversion failed for {file}: {error}" and continue
• Empty output: report "pandoc produced empty output for {file}" and continue

Idempotency: Overwrites existing output files without warning.

Tier 2: PDF (pdftotext + Claude structuring)

Step 1 — Extract:

INPUT_PATH="$1"
TEXT_PATH="${INPUT_PATH%.*}.txt"
OUTPUT_PATH="${INPUT_PATH%.*}.md"

pdftotext -layout "$INPUT_PATH" "$TEXT_PATH"

Scanned PDF detection: Count total characters and pages:

CHARS=$(wc -c < "$TEXT_PATH")
PAGES=$(pdfinfo "$INPUT_PATH" 2>/dev/null | grep "^Pages:" | awk '{print $2}')

If

pdfinfo

pdftotext

.txt

Step 2 — Structure: Dispatch a Sonnet agent using

skills/distill/pdf-structurer-prompt.md

OUTPUT_PATH

.txt

Tier 3: Python Venv

Venv setup (once per invocation, only if Tier 3 files exist):

VENV="/tmp/crucible-distill-venv"

# Health check
if [ -d "$VENV" ]; then
    "$VENV/bin/python3" -c "import sys" 2>/dev/null || rm -rf "$VENV"
fi

# Create if missing
if [ ! -d "$VENV" ]; then
    echo "Installing Python dependencies (one-time setup, ~15 seconds)..."
    python3 -m venv "$VENV"
    "$VENV/bin/pip" install --quiet python-pptx==1.0.2 openpyxl==3.1.5
    if [ $? -ne 0 ]; then
        echo "Failed to install Python dependencies."
        echo "Manual install: pip install python-pptx==1.0.2 openpyxl==3.1.5"
        echo "PPTX and XLSX conversion will be skipped."
        # Route remaining Tier 3 files to unsupported
        return
    fi
fi

PPTX conversion:

"$VENV/bin/python3" skills/distill/convert_pptx.py --input "$INPUT_PATH" --output "$OUTPUT_PATH"

XLSX conversion:

"$VENV/bin/python3" skills/distill/convert_xlsx.py --input "$INPUT_PATH" --output-dir "$(dirname "$INPUT_PATH")"

Output: one CSV per sheet at

{basename}-{sheetname}.csv

Phase 4: Digest Pass

After all conversions complete, run the digest pass on eligible files.

Eligibility:

• File is

  .md

  (not

  .csv

  )
• Word count > 500 words
• Word count ≤ 50,000 words (hard cap — report "File exceeds 50K word limit for digest pass. Consider splitting the document." for larger files)

Dispatch: For each eligible file, dispatch a Sonnet digest agent using

skills/distill/digest-prompt.md

{{ORIGINAL_WORDS}}

{{TARGET_WORDS}}

pdf-structurer-prompt.md

.md

digest-prompt.md

Quality check: After the digest agent returns, count words in the digest:

• If digest is 15-35% of input word count: accept
• If digest exceeds 35%: re-dispatch with "Compress more aggressively. Target 20-25% of the original word count."
• If digest is below 15%: re-dispatch with "Preserve more detail. Target 25-30% of the original word count."
• One retry only. Second result accepted regardless.

Output: Write digest to

{original-path-without-ext}.digest.md

Word count is a proxy for token count. These diverge for code-heavy or CJK content, but word count is sufficient for v1.

Phase 5: Summary

After all conversions and digests complete, output:

## Distill Summary

| File | Format | Tier | Converted | Digest | Token Savings |
|---|---|---|---|---|---|
| {file} | {format} | {tier} | {output} ({words} words) | {digest} ({words} words) | ~{pct}% |

**Total:** {N} files converted, {M} digests produced, ~{pct}% average token savings on digestible content.
Generated files can be added to .gitignore if not needed in version control.

Token savings per file =

1 - (digest words / converted words)

Files that were skipped (unsupported, tool missing, pre-flight failure) are listed separately:

**Skipped:** {N} files
- {file}: {reason}

Shell Safety (Non-Negotiable)

Every Bash command that touches file paths MUST use quoted shell variables:

# CORRECT
pandoc -f "$FORMAT" -t markdown --wrap=none "$INPUT_PATH" -o "$OUTPUT_PATH"

# WRONG — never do this
pandoc -f $FORMAT -t markdown --wrap=none $INPUT_PATH -o $OUTPUT_PATH

• All paths passed as

  "$VAR"

  , never bare

  $VAR

• No unquoted

  $()

  or backtick interpolation of paths
• Python scripts receive paths via argparse, not shell interpolation
• Source files are NEVER modified or deleted

Error Handling

Principle: Never halt the batch for a single file failure. Report and continue.

Integration

Standalone usage:

• /distill <path>

  — convert one or more files

• /distill <directory>

  — convert all supported files in directory

Called by:

• Any skill that needs to read heavy document formats
• User directly when preparing documents for LLM consumption

Dispatches:

• PDF structuring agent (Sonnet) via

  skills/distill/pdf-structurer-prompt.md

• Digest agent (Sonnet) via

  skills/distill/digest-prompt.md

Does not dispatch: No quality gate, no red-team, no review loop. Distill is a utility skill — it converts and compresses. Quality is ensured by the digest quality metric (word count check + one retry).