RAG Architect

The agent designs, implements, and optimizes production-grade Retrieval-Augmented Generation pipelines, covering the full lifecycle from document chunking through evaluation.

Workflow

1. Analyse corpus -- Profile the document collection: count, average length, format mix (PDF, HTML, Markdown), language(s), and domain. Validate that sample documents are accessible before proceeding.
2. Select chunking strategy -- Choose from the Chunking Strategy Matrix based on corpus characteristics. Set chunk size, overlap, and boundary rules. Run a test split on 100 sample documents.
3. Choose embedding model -- Select an embedding model from the Embedding Model table based on domain, latency budget, and cost constraints. Verify dimension compatibility with the target vector database.
4. Select vector database -- Pick a vector store from the Vector Database Comparison based on scale, query patterns, and operational requirements.
5. Design retrieval pipeline -- Configure retrieval strategy (dense, sparse, or hybrid). Add reranking if precision requirements exceed 0.85. Set the top-K parameter and similarity threshold.
6. Implement query transformations -- If query-document style mismatch exists, enable HyDE. If queries are ambiguous, enable multi-query generation. Validate each transformation improves retrieval metrics on a held-out set.
7. Configure guardrails -- Enable PII detection, toxicity filtering, hallucination detection, and source attribution. Set confidence score thresholds.
8. Evaluate end-to-end -- Run the RAGAS evaluation framework. Verify faithfulness > 0.90, context relevance > 0.80, answer relevance > 0.85. Iterate on weak components.

Chunking Strategy Matrix

Embedding Model Comparison

Vector Database Comparison

Retrieval Strategies

Query Transformation Techniques

Context Window Optimization

• Relevance ordering: Most relevant chunks first in the context window
• Diversity: Deduplicate semantically similar chunks
• Token budget: Fit within model context limit; reserve tokens for system prompt and answer
• Hierarchical inclusion: Include section summary before detailed chunks when available
• Compression: Summarize low-relevance chunks; extract key facts from verbose passages

Evaluation Metrics (RAGAS Framework)

Guardrails

• PII detection: Scan retrieved chunks and generated responses for PII; redact or block
• Hallucination detection: Compare generated claims against source documents via NLI
• Source attribution: Every factual claim must cite a retrieved chunk
• Confidence scoring: Return confidence level; if below threshold, return "I don't have enough information"
• Injection prevention: Sanitize user queries; reject prompt injection attempts

Example: Internal Knowledge Base RAG Pipeline

corpus:
  documents: 12,000 Confluence pages + 3,000 PDFs
  avg_length: 2,400 tokens
  languages: [English]
  domain: internal engineering docs

pipeline:
  chunking:
    strategy: recursive
    max_tokens: 512
    overlap: 50 tokens
    boundary: paragraph
  embedding:
    model: text-embedding-3-small
    dimensions: 1536
    batch_size: 100
  vector_db:
    engine: pgvector
    index: HNSW (ef_construction=128, m=16)
    reason: "Existing PostgreSQL infra; ACID compliance for audit"
  retrieval:
    strategy: hybrid
    dense_weight: 0.7
    sparse_weight: 0.3
    top_k: 10
    reranker: cross-encoder/ms-marco-MiniLM-L-12-v2
    final_k: 5

evaluation_results:
  faithfulness: 0.93
  context_relevance: 0.84
  answer_relevance: 0.88
  precision_at_5: 0.76
  recall_at_10: 0.85

Production Patterns

• Caching: Query-level (exact match), semantic (similar queries via embedding distance < 0.05), chunk-level (embedding cache)
• Streaming: Stream generation tokens while retrieval completes; show sources after generation
• Fallbacks: If primary vector DB is unavailable, serve from read-replica; if retrieval returns no results above threshold, say so explicitly
• Document refresh: Incremental re-embedding on change detection; full re-index weekly
• Cost control: Batch embeddings, cache aggressively, route simple queries to BM25 only

Common Pitfalls

Scripts

Chunking Optimizer

Analyses corpus and recommends optimal chunking strategy with parameters.

Retrieval Evaluator

Runs evaluation suite (precision, recall, MRR, NDCG) against a test query set.

Pipeline Benchmarker

Measures end-to-end latency, throughput, and cost per query across configurations.

Troubleshooting

chunking_optimizer.py

Success Criteria

• Faithfulness > 0.90 -- Generated answers are grounded in retrieved context as measured by the RAGAS faithfulness metric.
• Context Relevance > 0.80 -- At least 80% of retrieved chunks are relevant to the user query.
• Precision@5 > 0.70 -- Seven out of ten top-5 result sets contain only relevant documents.
• End-to-end latency < 500ms -- P95 query-to-response latency stays under 500 milliseconds for interactive workloads.
• Recall@10 > 0.85 -- The system retrieves at least 85% of relevant documents within the top 10 results.
• Chunk boundary quality > 0.80 -- At least 80% of chunks end on clean sentence or paragraph boundaries as reported by

  chunking_optimizer.py

  .
• Monthly cost within budget -- Total embedding, vector DB, and reranking costs stay within the budget ceiling defined in requirements.

Scope & Limitations

This skill covers:

• End-to-end RAG pipeline architecture design: chunking, embedding, vector storage, retrieval, reranking, and evaluation.
• Quantitative chunking analysis across four strategy families (fixed-size, sentence, paragraph, semantic).
• Retrieval quality evaluation using standard IR metrics (Precision@K, Recall@K, MRR, NDCG) with a built-in TF-IDF baseline.
• Automated pipeline design with component selection, cost projection, and Mermaid architecture diagrams.

This skill does NOT cover:

• LLM prompt engineering or generation-side optimization -- see

  engineering/prompt-engineer-toolkit

  .
• Database schema design for metadata stores alongside vector databases -- see

  engineering/database-designer

  .
• Production observability, alerting, and SLO dashboards for deployed pipelines -- see

  engineering/observability-designer

  .
• Agent orchestration or multi-step reasoning workflows that sit on top of RAG retrieval -- see

  engineering/agent-workflow-designer

  .

Integration Points

engineering/prompt-engineer-toolkit

engineering/database-designer

engineering/observability-designer

engineering/agent-workflow-designer

engineering/ci-cd-pipeline-builder

engineering/api-design-reviewer

Tool Reference

chunking_optimizer.py

Purpose: Analyzes a document corpus and evaluates multiple chunking strategies (fixed-size, sentence-based, paragraph-based, semantic/heading-aware) to recommend the optimal approach with configuration parameters.

Usage:

python chunking_optimizer.py <directory> [options]

Flags / Parameters:

directory

--output

-o

--config

-c

--extensions

.txt .md .markdown

--verbose

-v

Example:

python chunking_optimizer.py ./docs --output results.json --extensions .txt .md --verbose

Output Formats:

• Console -- Corpus summary, recommended strategy name, performance score, reasoning text, and two sample chunks. With

  --verbose

  , all strategy scores are listed.
• JSON (

  --output

  ) -- Full results object containing

  corpus_info

  ,

  strategy_results

  (per-strategy size statistics, boundary quality, semantic coherence, vocabulary statistics, performance score),

  recommendation

  (best strategy, all scores, reasoning), and

  sample_chunks

  .

retrieval_evaluator.py

Purpose: Evaluates retrieval system performance using a built-in TF-IDF baseline retriever and standard information retrieval metrics: Precision@K, Recall@K, MRR, and NDCG. Includes failure analysis and improvement recommendations.

Usage:

python retrieval_evaluator.py <queries> <corpus> <ground_truth> [options]

Flags / Parameters:

queries

{"id": ..., "query": ...}

{"queries": [...]}

corpus

ground_truth

--output

-o

--k-values

1 3 5 10

--extensions

.txt .md .markdown

--verbose

-v

Example:

python retrieval_evaluator.py queries.json ./corpus ground_truth.json --output eval.json --k-values 1 5 10 --verbose

Output Formats:

• Console -- Evaluation summary table (Precision@1, Precision@5, Recall@5, MRR, NDCG@5) with performance assessment and numbered improvement recommendations. With

  --verbose

  , all aggregate metrics and failure analysis counts are printed.
• JSON (

  --output

  ) -- Full results object containing

  aggregate_metrics

  ,

  query_results

  (per-query metrics, retrieved docs, relevant docs),

  failure_analysis

  (poor precision/recall counts, zero-result counts, query length analysis, failure patterns),

  evaluation_summary

  , and

  recommendations

  .

rag_pipeline_designer.py

Purpose: Accepts a system requirements specification and generates a complete RAG pipeline design including component recommendations (chunking, embedding, vector DB, retrieval, reranking, evaluation), cost projections, a Mermaid architecture diagram, and deployment configuration templates.

Usage:

python rag_pipeline_designer.py <requirements> [options]

Flags / Parameters:

requirements

--output

-o

--verbose

-v

Example:

python rag_pipeline_designer.py requirements.json --output pipeline_design.json --verbose

Output Formats:

• Console -- Design summary with total monthly cost, per-component recommendations (name, rationale, cost), and a Mermaid architecture diagram. With

  --verbose

  , full JSON configuration templates for each component are printed.
• JSON (

  --output

  ) -- Complete pipeline design object containing per-component

  ComponentRecommendation

  fields (name, type, config, rationale, pros, cons, cost_monthly),

  total_cost

  ,

  architecture_diagram

  (Mermaid markup), and

  config_templates

  (per-component configs plus deployment/scaling/monitoring settings).