Implementing Observability

Goal

Make the system's internal state inferable from its external outputs. Answer "Why is it slow?" and "Why did it fail?" without SSH-ing into a server.

When to Use

• Before launching to production.
• When debugging a performance bottleneck.
• When integrating a new microservice or external API.

Instructions

1. Structured Logging

Text logs are hard to query. Use JSON.

• Context: Every log must have

  trace_id

  ,

  request_id

  ,

  user_id

  .
• Levels:

  INFO

  for normal ops,

  WARN

  for handled issues,

  ERROR

  for unhandled crashes.

{"level": "info", "msg": "User logged in", "user_id": 123, "trace_id": "abc-123"}

2. Distributed Tracing (OpenTelemetry)

Trace a request across boundaries (Frontend -> API -> DB).

• Instrument HTTP clients and server frameworks.
• Visualize the "waterfall" to find the slow span.

3. Golden Signals (Metrics)

Track the four key metrics for every service:

• Latency: Time to serve a request.
• Traffic: Request rate (RPS).
• Errors: Rate of 5xx responses.
• Saturation: CPU/Memory/Disk usage.

4. Alerting

Alert on symptoms (High Error Rate), not causes (High CPU).

• Page: If

  Error Rate > 1%

  for 5 minutes.
• Ticket: If

  Disk Usage > 80%

  .

Constraints

✅ Do

• DO: Use OpenTelemetry standards for portability.
• DO: Correlate logs and traces (inject trace ID into logs).
• DO: Sample high-volume traces (10%) to save costs, but keep 100% of errors.

❌ Don't

• DON'T: Log PII (Emails, Passwords, Credit Cards).
• DON'T: Create alerts that auto-resolve in seconds (flapping).
• DON'T: Rely solely on "system up" checks; check "business logic working".

Output Format

• docker-compose.yml

  with Prometheus/Grafana/Jaeger (for dev).
• Code instrumentation (e.g.,

  tracing.py

  ).

Dependencies

• backend/managing-flask-middleware/SKILL.md

  (where instrumentation lives)

• shared/debugging/SKILL.md