documentation-indexing: Unified Full-Text Search + Ranking

Status: SAD STATE → IMPLEMENTATION 🌟 Information Energy: 0.87 (High aspiration, maximum sadness) Trit Assignment: 0 (COORDINATOR - Balances generators and validators) GF(3) Color: #49EE54 (Green - Equilibrium point)

Purpose

Provide full-text search, semantic indexing, and relevance ranking across all documentation:

• Skill registry (69 skills)
• Language docs (llms.txt standard)
• Blog posts / tutorials
• Source code docstrings
• DuckDB database schemas

Key capabilities:

1. Full-Text Search: Keyword + fuzzy matching (BM25 algorithm)
2. Semantic Ranking: TF-IDF + recency + community signals
3. Multi-Source Indexing: Consolidate docs from heterogeneous sources
4. Metadata Extraction: Automatically parse headers, links, code blocks
5. Bi-Directional Navigation: Move between docs ↔ implementations

Architecture

┌──────────────────────────────────────────────────────────────────┐
│              DOCUMENTATION INDEXING (GREEN COORDINATOR)          │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┬──────────────┬──────────────┐                 │
│  │   llms.txt   │   README.md  │  Code Docs   │                 │
│  │  Discovery   │   Crawlers   │  Extractors  │                 │
│  └──────┬───────┴──────┬───────┴──────┬───────┘                 │
│         │               │               │                       │
│         ▼──────────────▼───────────────▼                        │
│  ┌────────────────────────────────────────┐                    │
│  │    Metadata Normalization Layer        │                    │
│  │  • Title extraction (H1 → h1)          │                    │
│  │  • Link parsing (Markdown → URL)       │                    │
│  │  • Code fence detection ([```] → ...)  │                    │
│  │  • Authority scoring (stars, forks)    │                    │
│  └────────────┬──────────────────────────┘                     │
│               │                                                 │
│               ▼                                                 │
│  ┌────────────────────────────────────────┐                    │
│  │      Inverted Index (DuckDB)           │                    │
│  │  ┌──────────────────────────────────┐  │                    │
│  │  │ term_id | term | doc_id | rank  │  │                    │
│  │  │ 1       | gay  | 42     | 0.89  │  │                    │
│  │  │ 2       | mcp  | 42     | 0.76  │  │                    │
│  │  │ 3       | api  | 71     | 0.65  │  │                    │
│  │  └──────────────────────────────────┘  │                    │
│  └────────────┬───────────────────────────┘                    │
│               │                                                 │
│               ▼                                                 │
│  ┌────────────────────────────────────────┐                    │
│  │    BM25 Ranker + Result Aggregator     │                    │
│  │  • Cross-language result merging       │                    │
│  │  • Deduplication (canonical URLs)      │                    │
│  │  • Community signals (upvotes, stars)  │                    │
│  └────────────┬───────────────────────────┘                    │
│               │                                                 │
│               ▼                                                 │
│  ┌────────────────────────────────────────┐                    │
│  │   Result Cache (< 1 second latency)    │                    │
│  │   [query → ranked results + metadata]  │                    │
│  └────────────────────────────────────────┘                    │
│                                                                  │
│  GF(3) BALANCE: (-1 extractor) ⊗ (0 indexer) ⊗ (+1 ranker)     │
└──────────────────────────────────────────────────────────────────┘

Data Model

Documents Index

CREATE TABLE documentation_index (
  doc_id INT PRIMARY KEY,
  source VARCHAR,                 -- 'github', 'llms-txt', 'blog'
  repo_id VARCHAR,               -- 'bmorphism/Gay.jl'
  title VARCHAR,
  url VARCHAR,
  body TEXT,                      -- Full document text
  headers TEXT[],                 -- H1, H2, H3 hierarchy
  links TEXT[],                   -- Embedded links
  code_blocks TEXT[],             -- [```lang ... ```]
  stars INT,                      -- GitHub stars
  forks INT,                       -- GitHub forks
  updated_at TIMESTAMP,
  indexed_at TIMESTAMP,
  trit TINYINT                    -- GF(3) assigned (0)
);

Terms Inverted Index

CREATE TABLE term_index (
  term_id INT PRIMARY KEY,
  term VARCHAR,
  term_lower VARCHAR,
  frequency INT,                  -- TF (term frequency)
  doc_count INT,                  -- DF (document frequency)
  bm25_idf FLOAT,                 -- Precomputed IDF
  created_at TIMESTAMP
);

CREATE TABLE term_doc_map (
  term_id INT,
  doc_id INT,
  frequency INT,                  -- TF in this doc
  position INT[],                 -- Token positions
  context VARCHAR,                -- Surrounding text
  relevance_score FLOAT,          -- BM25(tf, idf, doc_len)
  PRIMARY KEY (term_id, doc_id)
);

API / Interfaces

Simple Text Search

;; Search docs
(search-docs {:query "gay"
              :type :keyword
              :limit 10
              :min-score 0.3})
→ [{:title "Gay.jl"
    :url "https://github.com/bmorphism/Gay.jl"
    :score 0.95
    :snippet "Gay.jl: Deterministic color generation..."}
   ...]

;; Fuzzy search (typo tolerance)
(search-docs {:query "gey"              ; typo
              :fuzzy true
              :distance 1})
→ (Results for "gay" with edit distance ≤ 1)

;; Advanced: Boolean search
(search-docs {:query "(gay OR color) AND julia"
              :type :boolean})

Metadata Search

;; Find docs by category
(search-by-metadata {:source "github"
                     :stars {:min 100 :max 1000}})
→ [Gay.jl, ACSets.jl, Duck, ...]

;; Find recent updates
(search-by-metadata {:updated-after "2025-12-01"
                     :source "blog"})
→ [Latest blog posts, ...]

;; Filter by language
(search-by-metadata {:languages ["Julia" "Clojure" "Babashka"]})

Bidirectional Navigation

;; Find implementations of a doc
(doc-implementations {:doc-id 42})
→ [{:file "src/gay.jl"
    :lines [1 42]
    :snippet "function seed!(rng)..."}]

;; Find docs for implementation
(implementation-docs {:file "src/gay.jl"
                      :line 10})
→ [{:doc-id 42
    :title "Gay.jl API Reference"
    :section "seed! function"}]

;; Find related docs
(related-docs {:doc-id 42
               :semantic true})
→ [ACSets.jl, GF(3) docs, ...]

GF(3) Trit Assignment

documentation-indexing → 0 (COORDINATOR)
  Balances extraction (-1) ↔ ranking/generation (+1)
  Maintains middle ground for all doc types

Triadic system:
  source-extractor (-1 validator)   → pulls raw docs
  indexing (0 coordinator)          → organizes & indexes
  result-ranker (+1 generator)      → produces ranked results

  Sum: (-1) + (0) + (+1) = 0 ✓ GF(3) CONSERVED

Implementation Strategy

Stage 1: Term Extraction (Days 1-2)

Create

/Users/bob/iii/duck/asi-skills/documentation-indexing/extractor.bb

:

• Markdown parser → extract H1-H4, links, code blocks
• Tokenizer → split text into terms
• Stopword filter → remove common words
• Store in DuckDB

  term_index

Stage 2: Inverted Index Builder (Days 3-4)

Create

indexer.bb

:

• BM25 IDF calculation
• TF per document
• Relevance scoring
• Populate

  term_doc_map

Stage 3: Search Engine (Days 5-6)

Create

searcher.bb

:

• Boolean query parser
• Fuzzy matching (Levenshtein)
• Result ranking by score
• Caching layer

Stage 4: Multi-Source Integration (Days 7-8)

• Crawl llms.txt repositories
• Index GitHub README files
• Extract docstrings from source
• Verify GF(3) balance across sources

Example: Semantic Search Pipeline

;; User query
(search-docs {:query "how to generate deterministic colors in julia"})

Step 1: Extract terms (-1 validator)
  ["deterministic" "colors" "julia"]

Step 2: Index lookup (0 coordinator)
  Fetch docs matching all terms
  Calculate BM25 score per doc

Step 3: Rank and aggregate (+1 generator)
  1. Gay.jl (0.95)
  2. GF(3) Integration (0.72)
  3. Color Theory (0.68)

Result: Σ(-1, 0, +1) = 0 ✓

Success Metrics

Metric	Target	Status
Docs indexed	500+ (skills + readmes + blogs)	⏳ Pending
Search latency	<100ms p95	⏳ Pending
Precision (top-5)	≥0.8	⏳ Pending
Recall	≥0.75	⏳ Pending
Fuzzy tolerance	Edit distance ≤ 2	⏳ Pending
GF(3) balance	All pipeline stages ≡ 0 (mod 3)	⏳ Pending

Related Skills

Dependencies:

• skill-taxonomy

  - Registry of docs to index

• acsets

  - Schema for index structure

• llms-txt-discovery

  - Crawl source docs

Dependents:

• polyglot-orchestration

  - Search polyglot docs

• skill-dispatch

  - Route searches to relevant skills

• world-knowledge-base

  - Unified doc interface

References

• BM25 algorithm: https://en.wikipedia.org/wiki/Okapi_BM25
• Inverted index: Classic IR data structure
• Levenshtein distance: Fuzzy string matching
• TF-IDF: Term weighting scheme
• DuckDB FTS: Full-text search extension

---

Status: 😢 SAD STATE → 🌟 IMPLEMENTING Color: #49EE54 (Green Coordinator) Next: Create

extractor.bb

(term extraction) Owner: GREEN AGENT (0) Created: 2026-01-04