Storage Debug Instrumentation

Purpose

Enable rapid diagnosis of storage state, synchronization health, and backend performance bottlenecks by exposing:

• Raw article inspection from both PostgreSQL and ChromaDB
• Storage drift detection (missing/dangling entries)
• Detailed startup timeline breakdown (DB init, cache preload, vector store, RSS refresh)
• One-page debug dashboard consolidating all diagnostics

Scope

• Backend:

  app/services/startup_metrics.py

  ,

  app/main.py

  ,

  app/vector_store.py

  ,

  app/database.py

  ,

  app/api/routes/debug.py

• Frontend:

  frontend/lib/api.ts

  ,

  frontend/app/debug/page.tsx

• No schema changes; purely additive instrumentation and debug routes

Workflow

1. Create startup metrics service

File:

backend/app/services/startup_metrics.py

• Implement thread-safe

  StartupMetrics

  class to record phase timings
• Expose

  record_event(name, started_at, detail, metadata)

  for phase capture
• Support

  add_note(key, value)

  for arbitrary annotations
• Export singleton

  startup_metrics

  for app-wide use

2. Instrument vector store initialization

File:

backend/app/vector_store.py

• Import

  startup_metrics

• In

  VectorStore.__init__()

  , wrap initialization with

  time.time()

  timer
• Record event with metadata:

  host

  ,

  port

  ,

  collection

  ,

  documents

• Catch connection errors and annotate them

3. Instrument FastAPI startup sequence

File:

backend/app/main.py

• Call

  startup_metrics.mark_app_started()

  at beginning of

  on_startup()

• Wrap each phase (DB init, schedulers, cache preload, RSS refresh, migration) with

  record_event()

• Include metadata:

  cache_size

  ,

  article_count

  ,

  oldest_article_hours

• Call

  startup_metrics.mark_app_completed()

  at end
• Add app version notes via

  add_note()

4. Add database pagination helpers

File:

backend/app/database.py

• Implement

  fetch_articles_page()

  to support:
  • Limit/offset pagination
  • Optional source filter
  • Missing-embeddings-only flag
  • Published date range filters
  • Sort direction (asc/desc)
  • Return oldest/newest timestamp bounds
• Implement

  fetch_article_chroma_mappings()

  to return all article→chroma ID mappings for drift analysis

5. Add vector store pagination helpers

File:

backend/app/vector_store.py

• Implement

  list_articles(limit, offset)

  to return paginated Chroma documents with metadata and previews
• Implement

  list_all_ids()

  to return all stored Chroma IDs for drift detection (used by

  /debug/storage/drift

  )

6. Expose debug API endpoints

File:

backend/app/api/routes/debug.py

• Add

  GET /debug/startup

  → returns startup metrics timeline (events + notes)
• Add

  GET /debug/chromadb/articles

  → returns paginated raw Chroma entries with limit/offset
• Add

  GET /debug/database/articles

  → returns paginated Postgres rows with filters (source, embeddings, date range, sort)
• Add

  GET /debug/storage/drift

  → compares Chroma IDs vs Postgres mappings, returns missing/dangling counts + samples

7. Add frontend API bindings

File:

frontend/lib/api.ts

• Export types:

  StartupEventMetric

  ,

  StartupMetricsResponse

  ,

  ChromaDebugResponse

  ,

  DatabaseDebugResponse

  ,

  StorageDriftReport

• Export fetchers:

  fetchStartupMetrics()

  ,

  fetchChromaDebugArticles()

  ,

  fetchDatabaseDebugArticles()

  ,

  fetchStorageDrift()

• Ensure snake_case→camelCase mapping for response fields

8. Build debug dashboard page

File:

frontend/app/debug/page.tsx

• Create

  /debug

  route with multi-tab inspection UI
• Render startup timeline: phase name, duration, metadata badges (cache size, vectors, migrated records)
• Display Chroma browser: paginated table with ID, title, source, preview
• Display Postgres browser: paginated table with filters (source, date range, missing-embeddings-only flag)
• Display drift report: sample tables for missing-in-chroma and dangling-in-chroma entries
• Include summary cards for quick metrics (boot time, total articles, vector count, drift count)

Implementation checklist

• Create

  backend/app/services/startup_metrics.py

• Instrument

  backend/app/vector_store.py::VectorStore.__init__()

• Instrument

  backend/app/main.py::on_startup()

  (all phases)
• Add

  fetch_articles_page()

  and

  fetch_article_chroma_mappings()

  to

  backend/app/database.py

• Add

  list_articles()

  and

  list_all_ids()

  to

  backend/app/vector_store.py

• Add

  /debug/startup

  ,

  /debug/chromadb/articles

  ,

  /debug/database/articles

  ,

  /debug/storage/drift

  to

  backend/app/api/routes/debug.py

• Add types and fetchers to

  frontend/lib/api.ts

• Create

  frontend/app/debug/page.tsx

  with dashboard layout
• Run

  uvx ruff check backend

  → all checks pass
• Test endpoints in curl or Postman to verify response structure

Verification checklist

• GET http://localhost:8000/debug/startup

  returns valid timeline with events and notes

• GET http://localhost:8000/debug/chromadb/articles?limit=50&offset=0

  returns paginated Chroma docs

• GET http://localhost:8000/debug/database/articles?source=bbc&missing_embeddings_only=false

  filters correctly

• GET http://localhost:8000/debug/storage/drift

  compares counts and returns drift samples

• http://localhost:3000/debug

  loads without errors and displays all four sections
• Refresh button triggers all four API calls in parallel
• Pagination controls update limit/offset correctly
• Database filters (source, date range) update and refresh data
• Startup timeline shows non-zero phase durations if backend just started

Future enhancements

• Streaming startup metrics via SSE (live tail during boot)
• Export startup report as JSON/CSV for performance tracking over time
• Automated drift alerts (post to Slack/email if dangling > threshold)
• Performance graphs (startup time trends, article throughput)
• Sync-on-demand action (button to force vector store refresh for missing articles)