Observability Designer

The agent designs production-ready observability strategies that combine the three pillars (metrics, logs, traces) with SLI/SLO frameworks, golden signals monitoring, and alert optimization.

Workflow

1. Catalogue services -- List every service in scope with its type (request-driven, pipeline, storage), criticality tier (T1-T3), and owning team. Validate that at least one T1 service exists before proceeding.
2. Define SLIs per service -- For each service, select SLIs from the Golden Signals table. Map each SLI to a concrete Prometheus/InfluxDB metric expression.
3. Set SLO targets -- Assign SLO targets based on criticality tier and user expectations. Calculate the corresponding error budget (e.g., 99.9% = 43.8 min/month).
4. Design burn-rate alerts -- Create multi-window burn-rate alert rules for each SLO. Validate that every alert has a clear runbook link and response action.
5. Build dashboards -- Generate dashboard specs following the hierarchy: Overview > Service > Component > Instance. Cap each screen at 7 panels. Include SLO target reference lines.
6. Configure log aggregation -- Define structured log format, set log levels, assign correlation IDs, and configure retention policies per tier.
7. Instrument traces -- Set up distributed tracing with sampling strategy (head-based for dev, tail-based for production). Define span boundaries at service and database call points.
8. Validate coverage -- Confirm every T1 service has metrics, logs, and traces. Confirm every alert has a runbook. Confirm dashboard load time is under 2 seconds.

SLI/SLO Quick Reference

1 - (sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])))

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

sum(rate(grpc_server_handled_total{grpc_code!="OK"}[5m])) / sum(rate(grpc_server_handled_total[5m]))

sum(rate(http_requests_total[5m]))

Error Budget Calculation

Error Budget = 1 - SLO target

Example (99.9% availability):
  Monthly budget = 30d x 24h x 60m x 0.001 = 43.2 minutes
  If 20 minutes consumed, remaining = 23.2 minutes (53.7% left)

Burn-Rate Alert Design

Rule: Every critical alert must have an actionable runbook. If no clear action exists, downgrade to warning.

Alert Classification

Alert Fatigue Prevention

• Hysteresis: Set different thresholds for firing (e.g., > 90% CPU for 5 min) and resolving (e.g., < 80% CPU for 10 min).
• Suppression: Suppress dependent alerts during known outages (e.g., suppress pod alerts when node is down).
• Grouping: Group related alerts into a single notification (e.g., all pods in one deployment).
• Precision over recall: A missed alert that would self-resolve is better than 50 false pages per week.

Golden Signals

Dashboard Design Rules

• Hierarchy: Overview (all services) > Service (one service) > Component (e.g., database) > Instance
• Panel limit: Maximum 7 panels per screen to manage cognitive load
• Reference lines: Always show SLO targets and capacity thresholds
• Time defaults: 4 hours for incident investigation, 7 days for trend analysis
• Role-based views: SRE (operational), Developer (debug), Executive (reliability summary)

Structured Log Format

{
  "timestamp": "2025-11-05T14:30:00Z",
  "level": "ERROR",
  "service": "payment-api",
  "trace_id": "abc123def456",
  "span_id": "789ghi",
  "message": "Payment processing failed",
  "error_code": "PAYMENT_TIMEOUT",
  "duration_ms": 5023,
  "customer_id": "cust_42",
  "environment": "production"
}

Log levels: DEBUG (local dev only), INFO (request lifecycle), WARN (degraded but functional), ERROR (failed operation), FATAL (service cannot continue).

Trace Sampling Strategies

Runbook Template

# Alert: [Alert Name]

## What It Means
[One sentence explaining the alert condition]

## Impact
[User-facing vs internal; affected services]

## Investigation Steps
1. Check dashboard: [link]  (1 min)
2. Review recent deploys: [link]  (2 min)
3. Check dependent services: [list]  (2 min)
4. Review logs: [query]  (3 min)

## Resolution Actions
- If [condition A]: [action]
- If [condition B]: [action]
- If unclear: Escalate to [team] via [channel]

## Post-Incident
- [ ] Update incident timeline
- [ ] File post-mortem if > 5 min user impact

Example: E-Commerce Payment Service Observability

service: payment-api
tier: T1 (revenue-critical)
owner: payments-team

slis:
  availability:
    metric: "1 - rate(http_5xx) / rate(http_total)"
    slo: 99.95%
    error_budget: 21.6 min/month
  latency_p99:
    metric: "histogram_quantile(0.99, http_duration_seconds)"
    slo: < 800ms
  error_rate:
    metric: "rate(payment_failures) / rate(payment_attempts)"
    slo: < 0.5%

alerts:
  - name: PaymentHighErrorRate
    expr: "rate(payment_failures[5m]) / rate(payment_attempts[5m]) > 0.01"
    for: 2m
    severity: critical
    runbook: "https://wiki.internal/runbooks/payment-errors"

dashboard_panels:
  - Payment success rate (gauge)
  - Transaction volume (time series)
  - P50/P95/P99 latency (time series)
  - Error breakdown by type (stacked bar)
  - Downstream dependency health (status map)
  - Error budget remaining (gauge)

Cost Optimization

• Metric retention: 15-day full resolution, 90-day downsampled, 1-year aggregated
• Log sampling: Sample DEBUG/INFO at 10% in high-throughput services; always keep ERROR/FATAL at 100%
• Trace sampling: Tail-based sampling retains only errors and slow requests (> P99)
• Cardinality management: Alert on any metric with > 10K unique label combinations

Scripts

SLO Designer (

slo_designer.py

)

Generates SLI/SLO frameworks from service description JSON. Outputs SLI definitions, SLO targets, error budgets, burn-rate alerts, and SLA recommendations.

Alert Optimizer (

alert_optimizer.py

)

Analyzes existing alert configurations for noise, coverage gaps, and duplicate rules. Outputs an optimization report with improved thresholds.

Dashboard Generator (

dashboard_generator.py

)

Creates Grafana-compatible dashboard JSON from service/system descriptions. Covers golden signals, RED/USE methods, and role-based views.

Integration Points

Troubleshooting

rate()

for:

curl localhost:9090/api/v1/label/__name__/values

metric_relabel_configs

Success Criteria

• Alert noise ratio below 10% -- fewer than 1 in 10 pages should be false positives or non-actionable.
• SLO compliance above 99.5% across all Tier-1 services measured over a rolling 30-day window.
• Mean time to detect (MTTD) under 5 minutes for Tier-1 service degradations via burn-rate alerts.
• Every critical alert has an associated runbook that was reviewed within the last 90 days.
• Dashboard load time under 2 seconds with default time range for all role-based views.
• Trace coverage spans 100% of Tier-1 service boundaries with tail-based sampling retaining all error and P99+ latency spans.
• Error budget consumption is reviewed weekly by the owning team with documented decisions on whether to freeze or proceed with deployments.

Scope & Limitations

Covers:

• SLI/SLO framework design for request-driven, pipeline, storage, and ML services.
• Multi-window burn-rate alert generation and alert noise optimization.
• Grafana-compatible dashboard specification with role-based layouts (SRE, Developer, Executive, Ops).
• Structured logging format, trace sampling strategy selection, and cost-optimization guidance.

Does NOT cover:

• Infrastructure provisioning or Terraform/Helm configuration for Prometheus, Grafana, or Jaeger -- see

  ci-cd-pipeline-builder

  for deployment pipelines.
• Incident response workflow orchestration or post-mortem facilitation -- see

  runbook-generator

  for runbook authoring.
• Application Performance Management (APM) agent installation or vendor-specific SDK integration.
• Security monitoring, SIEM rule design, or compliance audit logging -- see

  skill-security-auditor

  for security-focused analysis.

Integration Points

runbook-generator

ci-cd-pipeline-builder

performance-profiler

database-designer

tech-debt-tracker

release-manager

Tool Reference

SLO Designer (

scripts/slo_designer.py

)

Purpose: Generates complete SLI/SLO frameworks from service definitions, including SLI metric expressions, SLO targets, error budgets, multi-window burn-rate alerts, and SLA recommendations.

Usage:

python slo_designer.py --input service_definition.json --output slo_framework.json
python slo_designer.py --service-type api --criticality high --user-facing true
python slo_designer.py --service-type web --criticality critical --user-facing true --summary-only

Flags/Parameters:

--input

-i

--output

-o

{service_name}_slo_framework.json

--service-type

api

web

database

queue

batch

ml

--criticality

critical

high

medium

low

--user-facing

true

false

--service-name

{service_type}_service

--summary-only

*Either

--input

--service-type

--criticality

--user-facing

Example:

python slo_designer.py --service-type api --criticality high --user-facing true --service-name payment-api --output payment_slo.json

Output Formats: JSON file containing

metadata

slis

slos

error_budgets

sla_recommendations

monitoring_recommendations

implementation_guide

Alert Optimizer (

scripts/alert_optimizer.py

)

Purpose: Analyzes existing alert configurations to identify noisy alerts, coverage gaps, duplicate rules, poor thresholds, missing runbooks, and routing issues. Generates an optimization report and optionally an improved configuration.

Usage:

python alert_optimizer.py --input alerts.json --analyze-only
python alert_optimizer.py --input alerts.json --output optimized_alerts.json
python alert_optimizer.py --input alerts.json --report report.html --format html

Flags/Parameters:

--input

-i

--output

-o

optimized_alerts.json

--report

-r

--format

json

html

--analyze-only

Example:

python alert_optimizer.py --input prod_alerts.json --analyze-only --report analysis.json --format json

Output Formats: JSON or HTML report containing noise analysis (scored alerts with reasons and recommendations), coverage gap analysis (missing categories and golden signals), duplicate detection (exact and semantic duplicates), and optimization recommendations. When not using

--analyze-only

Dashboard Generator (

scripts/dashboard_generator.py

)

Purpose: Creates Grafana-compatible dashboard JSON specifications from service definitions. Covers golden signals, RED/USE methods, role-based views (SRE, Developer, Executive, Ops), and drill-down paths for troubleshooting workflows.

Usage:

python dashboard_generator.py --input service_definition.json --output dashboard_spec.json
python dashboard_generator.py --service-type api --name "Payment Service" --output payment_dashboard.json
python dashboard_generator.py --service-type web --name "Frontend" --role developer --format grafana

Flags/Parameters:

--input

-i

--output

-o

{service_name}_dashboard.json

--service-type

api

web

database

queue

batch

ml

--name

--criticality

critical

high

medium

low

--role

sre

developer

executive

ops

--format

json

grafana

--doc-output

--summary-only

*Either

--input

--service-type

--name

Example:

python dashboard_generator.py --service-type database --name "orders-db" --criticality high --role sre --format grafana --output orders_db_dashboard.json

Output Formats: JSON specification or Grafana-compatible JSON containing

metadata

configuration

layout

panels

variables

alerts_integration

drill_down_paths

--doc-output