Observability

Comparable to: Datadog, Grafana, Honeycomb, Jaeger

Key Concepts

Use the concepts below when they fit the task. Not every worker needs custom spans or metrics.

• Built-in OpenTelemetry support across all SDKs — every function invocation is automatically traced
• The engine exports traces, metrics, and logs via OTLP to any compatible collector
• Workers propagate W3C trace context automatically across function invocations
• Prometheus metrics are exposed on port 9464

• registerWorker()

  with

  otel

  config enables telemetry per worker
• Custom spans via

  withSpan(name, opts, fn)

  wrap async work with trace context
• Custom metrics via

  getMeter()

  create counters and histograms

Architecture

The worker SDK generates spans, metrics, and logs during function execution. These flow to the engine, which exports them via OTLP to a collector (Jaeger, Grafana, Datadog). The engine also exposes a Prometheus endpoint on port 9464 for scraping.

iii Primitives Used

registerWorker(url, { otel })

withSpan(name, opts, fn)

getTracer()

getMeter()

currentTraceId()

injectTraceparent()

onLog(callback, { level })

shutdown_otel()

Reference Implementation

See ../references/observability.js for the full working example — a worker with custom spans,

Also available in Python: ../references/observability.py

Also available in Rust: ../references/observability.rs metrics counters, trace propagation, and log subscriptions connected to an OTel collector.

Common Patterns

Code using this pattern commonly includes, when relevant:

• registerWorker('ws://localhost:49134', { otel: { enabled: true, serviceName: 'my-svc' } })

  — enable telemetry

• withSpan('validate-order', {}, async (span) => { span.setAttribute('order.id', id); ... })

  — custom span

• getMeter().createCounter('orders.processed')

  — custom counter metric

• getMeter().createHistogram('request.duration')

  — custom histogram metric

• onLog((log) => { ... }, { level: 'warn' })

  — subscribe to warnings and above

• currentTraceId()

  — get active trace ID for correlation with external systems

• injectTraceparent()

  — propagate trace context to outbound HTTP calls
• Disable telemetry:

  registerWorker(url, { otel: { enabled: false } })

  or

  OTEL_ENABLED=false

Adapting This Pattern

Use the adaptations below when they apply to the task.

• Enable

  otel

  in

  registerWorker()

  config to start collecting traces automatically
• Add custom spans around expensive operations (DB queries, LLM calls, external APIs)
• Create domain-specific metrics (orders processed, payment failures, queue depth)
• Use

  currentTraceId()

  to correlate iii traces with external system logs
• Configure

  iii-observability

  in iii-config.yaml for engine-side exporter, sampling ratio, and alerts
• Point the OTLP endpoint at your collector (Jaeger, Grafana Tempo, Datadog Agent)

Engine Configuration

iii-observability must be enabled in iii-config.yaml for engine-side traces, metrics, and logs. See ../references/iii-config.yaml for the full annotated config reference.

Pattern Boundaries

• For engine-side iii-observability YAML configuration, prefer

  iii-engine-config

  .
• For SDK init options and function registration, prefer

  iii-functions-and-triggers

  .
• Stay with

  iii-observability

  when the primary problem is SDK-level telemetry: spans, metrics, logs, and trace propagation.

When to Use

• Use this skill when the task is primarily about

  iii-observability

  in the iii engine.
• Triggers when the request directly asks for this pattern or an equivalent implementation.

Boundaries

• Never use this skill as a generic fallback for unrelated tasks.
• You must not apply this skill when a more specific iii skill is a better fit.
• Always verify environment and safety constraints before applying examples from this skill.