System Architecture (Canonical Topology)

This skill is the single source of truth for joelclaw system wiring. Use it for:

• "why did this run / not run"
• "which worker handles this function"
• "what is listening on port X"
• "how does event Y flow"
• full-stack routing/debug across CLI → Inngest → workers → gateway → telemetry

Ground-Truth Scope + Evidence Snapshot

This document is grounded in direct reads of:

• apps/docs-api/src/index.ts

• packages/restate/Dockerfile

• packages/restate/src/index.ts

• packages/restate/src/workflows/dag-orchestrator.ts

• packages/agent-execution/src/microvm.ts

• packages/system-bus/src/serve.ts

• packages/system-bus/src/inngest/functions/index.host.ts

• packages/system-bus/src/inngest/functions/index.cluster.ts

• packages/system-bus/src/inngest/client.ts

• infra/worker-supervisor/src/main.rs

• ~/Library/LaunchAgents/com.joel*.plist

• k8s/*

  (all files)

• infra/pds/values.yaml

• packages/gateway/src/daemon.ts

• packages/gateway/src/channels/*.ts

• ~/.joelclaw/gateway/AGENTS.md

• ~/.joelclaw/gateway/.pi/settings.json

• ~/.local/caddy/Caddyfile

• ~/.colima/default/colima.yaml

  +

  colima status --json

• packages/cli/src/cli.ts

  ,

  packages/cli/src/config.ts

  ,

  packages/cli/src/inngest.ts

• packages/system-bus/src/observability/*

  (key files:

  emit.ts

  ,

  otel-event.ts

  ,

  store.ts

  )

• packages/telemetry/src/emitter.ts

• packages/system-bus/src/lib/langfuse.ts

• packages/inference-router/src/tracing.ts

• ADRs in

  ~/Vault/docs/decisions/

  (required + topology-adjacent)
• last 50 lines of

  ~/Vault/system/system-log.jsonl

Related docs verified

• docs/architecture.md

  — Restate/Firecracker runtime + workload execution flow

• docs/deploy.md

  — Restate worker deploy + auth/identity/PVC procedures

• docs/cli.md

  — workload command tree + runtime bridge

• docs/observability.md

  — not inspected in this update

1) Physical Topology

Mac Mini "Panda" (host macOS)
├─ launchd services (gateway, worker supervisor, caddy, talon, agent-mail, etc.)
├─ Colima VM (driver: VZ, arch: aarch64, runtime: docker, VM IP: 192.168.64.2)
│  └─ Talos node: joelclaw-controlplane-1 (k8s v1.35.0, internal IP 10.5.0.2)
│     ├─ namespace: joelclaw
│     │  ├─ inngest (StatefulSet + NodePort 8288/8289)
│     │  ├─ redis (StatefulSet + NodePort 6379)
│     │  ├─ typesense (StatefulSet + ClusterIP 8108)
│     │  ├─ restate (StatefulSet + NodePort 8080/9070/9071)
│     │  ├─ system-bus-worker (Deployment + ClusterIP 3111)
│     │  ├─ restate-worker (Deployment + ClusterIP 9080; full agent image + Firecracker)
│     │  ├─ dkron (StatefulSet + ClusterIP 8080)
│     │  ├─ docs-api (Deployment + NodePort 3838)
│     │  ├─ livekit-server (Deployment + NodePort 7880/7881)
│     │  ├─ bluesky-pds (Deployment + NodePort 3000)
│     │  └─ minio (StatefulSet + NodePort 30900/30901)
│     └─ namespace: aistor
│        ├─ aistor operator (Deployments: adminjob-operator, object-store-operator)
│        └─ aistor-s3 object store (StatefulSet + NodePort 31000/31001)
├─ Caddy reverse proxy (tailnet HTTPS fan-in)
├─ Gateway daemon (embedded pi session)
├─ Firecracker substrate (requires Colima nestedVirtualization=true for /dev/kvm; OFF by default — unstable under load)
└─ NAS "three-body" (NFS tiers per ADR-0088)

Known runtime endpoints

• Colima VM IP:

  192.168.64.2

  (

  colima status --json

  )
• Kubernetes API (local forward):

  https://127.0.0.1:64784

  (

  kubectl cluster-info

  )
• Tailnet hostnames seen in config:

  • panda.tail7af24.ts.net

    (Caddy routes)

  • pds.panda.tail7af24.ts.net

    (PDS values)

Tailscale mesh state

• tailscale status --json

  failed in this environment: UNKNOWN — needs manual verification

2) Process Inventory (Long-Running)

Host launchd inventory (snapshot)

Snapshot source:

launchctl print gui/$(id -u)/<label>

and plist inspection.

com.joel.system-bus-worker

worker-supervisor

com.joel.restate-worker

scripts/restate/start.sh

deployment/restate-worker

com.joel.gateway

packages/gateway/src/daemon.ts

:3018

com.joel.caddy

com.joel.talon

127.0.0.1:9999

com.joel.agent-secrets

com.joel.imsg-rpc

/tmp/imsg.sock

com.joel.typesense-portforward

kubectl port-forward svc/typesense 8108:8108

com.joel.voice-agent

com.joel.local-sandbox-janitor

scripts/local-sandbox-janitor.sh

joelclaw workload sandboxes janitor

/tmp/joelclaw/local-sandbox-janitor.{log,err}

com.joelclaw.agent-mail

127.0.0.1:8765

com.joel.colima

com.joel.k8s-reboot-heal

com.joel.system-bus-sync

com.joel.gateway-tripwire

com.joel.content-sync-watcher

com.joel.vault-log-sync

Process supervision behavior:

worker-supervisor

Source:

infra/worker-supervisor/src/main.rs

• Default config:
  • worker dir:

    ~/Code/joelhooks/joelclaw/packages/system-bus

  • command:

    bun run src/serve.ts

  • port:

    3111

  • health endpoint:

    /api/inngest

  • sync endpoint:

    /api/inngest

    (PUT)
  • health interval: 30s
  • restart after 3 consecutive health failures
  • restart backoff: 1s → 30s max
• Pre-start kills stale process on port 3111.
• Runs host import preflight before spawn:

  • bun --eval "await import('./src/inngest/functions/index.host.ts');"

  • on failure, skips spawn and retries with exponential backoff
• Loads env from

  ~/.config/system-bus.env

  plus leased secrets.
• Forces

  WORKER_ROLE=host

  for the supervised host worker.
• Emits OTEL events via CLI on supervisor failures/restarts:

  • worker.supervisor.preflight.failed

  • worker.supervisor.worker_exit

  • worker.supervisor.health_check.restart

Worker supervision split note

• Talon is running (

  com.joel.talon

  ), but host worker is still launched via

  com.joel.system-bus-worker

  ->

  worker-supervisor

  .
• ADR + system-log indicate Talon can defer worker supervision during coexistence.

Kubernetes process inventory

Node

• joelclaw-controlplane-1

  (Talos v1.12.4, k8s v1.35.0, internal IP

  10.5.0.2

  )

Core services

inngest

inngest-svc

redis

typesense

restate

dagOrchestrator

dagWorker

dkron-svc

dkron-peer

aistor

aistor

Restate / Firecracker runtime note

• deployment/restate-worker

  is the current durable execution worker. The image bundles Bun + Node +

  pi

  +

  codex

  , the full repo checkout, and 76 symlinked skills.
• Runtime auth/identity come from

  secret/pi-auth

  and

  configmap/agent-identity

  , which recreate

  /root/.pi/agent/auth.json

  plus the joelclaw identity chain inside the pod.
• Firecracker is enabled in-pod via privileged access to

  /dev/kvm

  on Colima VZ. The

  /dev/kvm

  hostPath mount uses type

  ""

  (optional) so the pod starts without it when nestedVirtualization is off.
• Persistent microVM assets live on PVC

  firecracker-images

  , mounted at

  /tmp/firecracker-test

  for kernel, rootfs, and snapshot files.
• Retry caps (2026-03-17): dagWorker maxAttempts=5, dagOrchestrator maxAttempts=3. Prevents Restate journal poisoning from infinite retries after code changes or infrastructure failures.
• Colima stability: nestedVirtualization is OFF by default (crashes VM under Docker build load). Toggle ON only for Firecracker testing sessions, then toggle OFF. See k8s skill for recovery procedures.

Control-plane access

• kube API exposed locally at

  127.0.0.1:64784

  (forwarded)
• additional forwarded control ports observed:

  64785

  ,

  9627

  (exact ownership mapping UNKNOWN — needs manual verification)

3) Worker Architecture (Role Split + Registration)

Source files:

• packages/system-bus/src/serve.ts

• packages/system-bus/src/inngest/functions/index.host.ts

• packages/system-bus/src/inngest/functions/index.cluster.ts

• packages/system-bus/src/inngest/client.ts

Role model

• WORKER_ROLE

  parsed as

  host

  (default) or

  cluster

  .
• Registered function set is role-dependent:
  • host uses

    hostFunctionDefinitions

  • cluster uses

    clusterFunctionDefinitions

Ground-truth counts

• Host function set: 101
• Cluster function set: 12
• Cluster subset functions:

  • approvalRequest

    ,

    approvalResolve

  • todoistCommentAdded

    ,

    todoistTaskCompleted

    ,

    todoistTaskCreated

  • frontMessageReceived

    ,

    frontMessageSent

    ,

    frontAssigneeChanged

  • todoistMemoryReviewBridge

  • githubWorkflowRunCompleted

    ,

    githubPackagePublished

  • webhookSubscriptionDispatchGithubWorkflowRunCompleted

App registration isolation

From

inngest/client.ts

• app id resolves to:

  • system-bus-host

    when role is host

  • system-bus-cluster

    when role is cluster
• explicit

  INNGEST_APP_ID

  overrides role-derived id.

This prevents host and cluster workers from overwriting each other’s function graphs.

serveHost behavior

From

serve.ts

• host role default

  serveHost

  :

  http://host.docker.internal:3111

• cluster role default

  serveHost

  : unset (connect-mode default)

• INNGEST_SERVE_HOST

  overrides either role.

Kubernetes cluster worker manifest sets:

• INNGEST_BASE_URL=http://inngest-svc:8288

• INNGEST_SERVE_HOST=http://system-bus-worker:3111

Registration mechanics

• Worker exposes

  GET|POST|PUT /api/inngest

  .
• Worker sends a delayed self-sync

  PUT /api/inngest

  ~5s after startup.

• worker-supervisor

  also performs startup PUT sync.

Host is primary today

From index comments + function lists:

• ADR-0089 transition: host remains authoritative for broad function ownership.
• Cluster is intentionally limited to cluster-safe subset (12 functions).

4) Event Flow (CLI → Inngest → Worker → Completion)

Canonical flow:

joelclaw send

1. CLI

   joelclaw send <event>

   calls

   Inngest.send()

   .

2. Inngest.send()

   POSTs event JSON to:

   • ${INNGEST_URL}/e/${INNGEST_EVENT_KEY}

   • default:

     http://localhost:8288/e/<key>

3. Inngest server persists the event and resolves matching function triggers.
4. Inngest dispatches function steps to the worker app graph that owns that function ID:
   • host app (

     system-bus-host

     ) for 101-host set
   • cluster app (

     system-bus-cluster

     ) for 12-cluster subset
5. Worker handles callbacks via

   /api/inngest

   (Hono +

   inngest/hono

   handler).
6. Each

   step.run

   result is memoized by Inngest; next step executes when prior completes.
7. Completion/failure is queryable via GraphQL (

   /v0/gql

   ) and CLI commands (

   runs

   ,

   run

   ,

   event

   ,

   events

   ).

Queue flow:

joelclaw queue emit

→ Restate drainer → durable dispatch

1. CLI

   joelclaw queue emit <event>

   persists a

   QueueEventEnvelope

   into Redis stream

   joelclaw:queue:events

   and indexes it in sorted set

   joelclaw:queue:priority

   .
2. The

   restate-worker

   k8s deployment (

   packages/restate/src/index.ts

   ) starts a deterministic queue drainer beside the channel callback listener.
3. On startup, the drainer claims pending + never-delivered entries via

   @joelclaw/queue#getUnacked()

   , reindexes replayable entries, and emits OTEL replay evidence.
4. Each drain tick selects the next priority candidate from the sorted set, resolves its static registry target from

   packages/queue/src/registry.ts

   , and POSTs a one-node DAG request to Restate

   /dagOrchestrator/{workflowId}/run/send

   .
5. When backlog remains and a dispatch slot frees, the drainer self-pulses immediately instead of waiting for the next

   QUEUE_DRAIN_INTERVAL_MS

   heartbeat. That interval is now the idle poll / retry cadence, not a mandatory 2-second tax between successful sends.
6. The current Story-3 bridge re-emits the queue item to its registered Inngest event target inside that one-node DAG request. This is deliberate: the deterministic queue/drainer is proven first; per-family Restate cutovers remain Story 4 work.
7. On accepted Restate dispatch, the drainer acks the queue message; on failure it leaves the message in Redis, applies retry cooldown, and emits

   queue.dispatch.failed

   OTEL evidence.
8. If backlog remains in Redis but the drainer stops making progress past

   QUEUE_DRAIN_STALL_AFTER_MS

   , it emits

   queue.drainer.stalled

   and exits non-zero so k8s restarts

   deployment/restate-worker

   . That is the self-heal path for a wedged drainer inside an otherwise-running Bun process.
9. Crash recovery comes from the Redis stream + consumer-group replay path, not from vibes: restart the

   restate-worker

   pod, let

   getUnacked()

   reclaim the inflight entries, then drain resumes.

Workload flow:

joelclaw workload run

→ Redis → Restate DAG → execution

1. joelclaw workload plan ... --stages-from <file>

   can load an explicit stage DAG, validate unknown deps/self-deps/duplicates/cycles, and preserve per-stage acceptance gates.

2. joelclaw workload run <plan-artifact>

   normalizes the selected stage into the canonical

   workload/requested

   runtime request.
3. Queue admission writes the request into Redis, where the deterministic drainer forwards it into Restate as a

   dagOrchestrator/{workflowId}/run/send

   request.

4. dagOrchestrator

   executes dependency waves: ready nodes in parallel, chained nodes only after every

   dependsOn

   node has terminal output.

5. dagWorker

   executes the node handler:

   • shell

     → subprocess work inside the

     restate-worker

     pod

   • infer

     →

     pi -p --no-session --no-extensions

     inside the pod, using the mounted auth + identity + skill set

   • microvm

     → Firecracker boot/restore through

     /dev/kvm

     with kernel/rootfs/snapshot files on PVC

     firecracker-images

6. Each node emits OTEL (

   dag.node.*

   ), and the workflow emits

   dag.workflow.*

   so queue → Restate → execution remains observable.
7. Current truthful limit: the microVM runtime boots and restores snapshots in-cluster, but the broader exec-in-VM workspace drive protocol is still incomplete for general coding slices.

Webhook flow

1. External service posts to

   /webhooks/:provider

   .
2. Caddy routes

   /webhooks/*

   on

   localhost:8443

   to worker

   localhost:3111

   .

3. webhookApp

   verifies signature, normalizes payload, emits Inngest events (

   provider/event

   ).
4. Inngest executes subscribed functions.

"Why did this run / not run" trace recipe

1. joelclaw send <event> -d '<payload>'

2. joelclaw events --prefix <event-prefix> --hours 1

3. joelclaw event <event-id>

   (fan-out to function runs)

4. joelclaw run <run-id>

   (step trace + errors)

5. joelclaw runs --count 20 --hours 1

6. joelclaw otel search "<component/action>" --hours 1

7. Validate function ownership in

   index.host.ts

   /

   index.cluster.ts

   .

5) Port Map (Canonical)

Exposure sources: k8s service manifests, Caddyfile,

kubectl get svc

,

lsof

listeners.

/

/api/inngest

/webhooks

/observability/emit

restate-worker

dagOrchestrator

dagWorker

kubectl port-forward

svc/dkron-svc:8080

aistor

aistor

localhost:3111

localhost:3838

localhost:7880

localhost:8288

localhost:8289

127.0.0.1

127.0.0.1

Notes

• Host NodePort exposure appears through an

  ssh

  listener process (Colima portForwarder=ssh).
• Exact per-port ssh forward command line is UNKNOWN — needs manual verification (process introspection restricted in this environment).

6) Storage Topology

Redis

• Runtime: k8s StatefulSet (

  redis:7-alpine

  , appendonly enabled).
• Primary uses:
  • gateway queue/session keys (

    joelclaw:events:*

    ,

    joelclaw:notify:*

    ,

    joelclaw:gateway:sessions

    )
  • webhook subscriptions (

    joelclaw:webhook:*

    )
  • gateway health mute/streak keys (

    gateway:health:*

    )

Typesense

From observability code:

• otel_events

  collection (canonical telemetry event store)

• memory_observations

  collection (vector-aware memory index; schema validated at startup)
• docs-api also points at

  http://typesense:8108

  for docs search/index surfaces.

Firecracker runtime storage

• PVC:

  firecracker-images

• Mounted in

  deployment/restate-worker

  at

  /tmp/firecracker-test

• Stores:
  • kernel (

    vmlinux

    )
  • rootfs (

    agent-rootfs.ext4

    )
  • snapshots (

    snapshots/vm.snap

    ,

    snapshots/vm.mem

    )
• Firecracker snapshot restore is currently operator-proven at ~9ms on the Colima VZ nested-virt path.

Inngest state

• StatefulSet PVC mounted at

  /data

• INNGEST_SQLITE_DIR=/data

docs-api surface

• Deployment:

  docs-api

  on NodePort

  3838

• Route count: 11 endpoints including

  /health

• Key routes:

  • GET /search

    — hybrid chunk search with

    concept

    ,

    concepts

    ,

    doc_id

    ,

    expand

    , and

    assemble

  • GET /docs/search

  • GET /docs

  • GET /docs/:id

  • GET /docs/:id/toc

  • GET /docs/:id/chunks

  • GET /chunks/:id

  • GET /concepts

  • GET /concepts/:id

  • GET /concepts/:id/docs

• Taxonomy surface: 21-concept SKOS graph (10 parents + 11 sub-concepts) with

  broader

  ,

  narrower

  , and

  related

  edges.

NAS (ADR-0088 + ADR-0187)

Tiering policy:

• Tier 1 local SSD (hot runtime state)
• Tier 2 NAS NVMe (

  /Volumes/nas-nvme

  ↔

  /volume2/data

  )
• Tier 3 NAS HDD (

  /Volumes/three-body

  )

Access paths

/Volumes/nas-nvme

/Volumes/three-body

nas-nvme

nas-hdd

/Volumes/nas-nvme

/Volumes/three-body

k8s ↔ NAS networking

k8s pods reach the NAS via a LAN route through the Colima col0 bridge:

Talos → Docker NAT → VM col0 → macOS (ip.forwarding=1) → LAN → NAS

The VZ NAT on eth0 does NOT forward LAN traffic. Route persisted in Colima provision + colima-tunnel script:

ip route replace 192.168.1.0/24 via 192.168.64.1 dev col0

Always use IP 192.168.1.163, never hostname three-body — DNS doesn't resolve from k8s.

Degradation contract (ADR-0187):

• writes must fallback

  local -> remote -> queued

• queue spool default:

  /tmp/joelclaw/nas-queue

Vault

• Obsidian vault at

  /Users/joel/Vault

• system log file:

  /Users/joel/Vault/system/system-log.jsonl

7) Networking Topology

Caddy reverse proxy routes (from

~/.local/caddy/Caddyfile

)

• https://panda.tail7af24.ts.net:9443

  ->

  localhost:8288

  (Inngest)

• https://panda.tail7af24.ts.net:8290

  ->

  localhost:8289

  (Inngest connect)

• https://panda.tail7af24.ts.net:3443

  ->

  localhost:3111

  (worker)

• https://panda.tail7af24.ts.net:5443

  ->

  localhost:3838

  (docs-api)

• https://panda.tail7af24.ts.net:7443

  ->

  localhost:7880

  (LiveKit)

• https://panda.tail7af24.ts.net:6443

  ->

  localhost:6333

  (Qdrant)

• http://localhost:8443

  path router:

  • /webhooks/*

    ->

    localhost:3111

  • fallback ->

    localhost:8288

Tailscale + Funnel

• Config comments and ADR-0051 describe Funnel path

  :443 -> localhost:8443

  .
• Runtime

  tailscale status

  unavailable here: UNKNOWN — needs manual verification.

External webhook ingress

Expected path:

1. Internet provider -> Tailscale Funnel :443
2. Funnel -> local

   :8443

3. Caddy path route

   /webhooks/*

   -> worker

   :3111

4. worker

   /webhooks/:provider

   verifies + emits Inngest event

8) CLI Wiring (Command Tree → Endpoint Surface)

Primary command tree root:

packages/cli/src/cli.ts

Endpoint map by command family

send

POST /e/<event-key>

runs

run

functions

event

events

POST /v0/gql

status

gateway *

workload *

dagOrchestrator

dagWorker

docs *

/search

/docs/*

/chunks/*

/concepts*

restate cron *

--base-url

kubectl port-forward

svc/dkron-svc

otel *

otel_events

recall *

mail *

127.0.0.1:8765

inngest *

Config source:

• ~/.config/system-bus.env

  (plus env overrides)
• defaults:

  • INNGEST_URL=http://localhost:8288

  • INNGEST_WORKER_URL=http://localhost:3111

9) Observability + Tracing Topology

OTEL event pipeline

• Worker emits via

  emitOtelEvent()

  /

  emitMeasuredOtelEvent()

  .
• Gateway emits via

  @joelclaw/telemetry

  (

  emitGatewayOtel

  ) to:
  • default

    OTEL_EMIT_URL=http://localhost:3111/observability/emit

• Worker endpoint

  /observability/emit

  validates token (

  x-otel-emit-token

  ) if configured.
• Store path (

  storeOtelEvent

  ):
  1. Typesense

     otel_events

     (primary)
  2. optional Convex mirror for high-severity recent window
  3. optional Sentry forward for

     warn/error/fatal

Langfuse integration points

• Gateway boot:

  packages/gateway/src/daemon.ts

  calls

  initTracing({})

  from inference-router.
• Inference router traces model-route decisions:

  • packages/inference-router/src/tracing.ts

  • used from

    packages/inference-router/src/router.ts

• System-bus LLM traces:

  • packages/system-bus/src/lib/langfuse.ts

    (

    traceLlmGeneration

    )
  • called by

    packages/system-bus/src/lib/inference.ts

    and

    channel-message-classify.ts

10) Key ADR Topology Decisions

/webhooks/:provider

@joelclaw/*

localhost -> vm -> svc_dns

aistor

10.1) Sandbox Execution Contract (@joelclaw/agent-execution)

Package:

packages/agent-execution/

Contract Types

Request:

SandboxExecutionRequest

• workflowId

  ,

  requestId

  ,

  storyId

  : identifiers

• task

  : story prompt/task to execute

• agent

  :

  { name, variant?, model?, program? }

• sandbox

  :

  "workspace-write" | "danger-full-access"

• baseSha

  : git SHA before execution

• cwd?

  : working directory

• timeoutSeconds?

  : timeout

• verificationCommands?

  : post-execution verification

• sessionId?

  : tracking identifier

Result:

SandboxExecutionResult

• requestId

  : correlation ID

• state

  :

  "pending" | "running" | "completed" | "failed" | "cancelled"

• startedAt

  ,

  completedAt?

  ,

  durationMs?

  : timing

• artifacts?

  : execution artifacts (see below)

• error?

  : error message (failed state)

• output?

  : stdout/stderr output

Artifacts:

ExecutionArtifacts

• headSha

  : git SHA after execution

• touchedFiles

  : list of modified/untracked files from

  git status --porcelain

• patch?

  : git patch content (format-patch or diff)

• verification?

  :

  { commands, success, output }

• logs?

  :

  { executionLog?, verificationLog? }

Repo Materialization (Story 3)

Function:

materializeRepo(targetPath, baseSha, options)

Behavior:

• Clone repo if target path doesn't exist (requires

  remoteUrl

  )
• Fetch + checkout if target path exists
• SHA verification after checkout
• Automatic unshallow if SHA not in shallow clone
• Isolated sandbox-local workspace (host worktree untouched)

Returns:

{ path, sha, freshClone, durationMs }

Key options:

• remoteUrl?

  : remote URL for fresh clone

• branch?

  : branch/ref to fetch (default:

  "main"

  )

• depth?

  : shallow clone depth (default:

  1

  )

• includeSubmodules?

  : include submodules

• timeoutSeconds?

  : timeout (default:

  300

  )

Artifact Export (Story 3)

Function:

generatePatchArtifact(options)

Behavior:

• Captures touched-file inventory via

  getTouchedFiles()

• Generates git patch from

  baseSha..headSha

  :
  • Uses

    git format-patch

    if commits exist in range
  • Uses

    git diff

    if only uncommitted changes
• Optionally includes untracked files as patch content
• Embeds verification summary and log references
• Serializable to JSON via

  writeArtifactBundle()

Key options:

• repoPath

  : path to git repo

• baseSha

  : base SHA (start of diff range)

• headSha?

  : head SHA (default: HEAD)

• includeUntracked?

  : include untracked files (default:

  true

  )

• verificationCommands?

  ,

  verificationSuccess?

  ,

  verificationOutput?

  : verification data

• executionLogPath?

  ,

  verificationLogPath?

  : log references

• timeoutSeconds?

  : timeout (default:

  60

  )

Returns:

ExecutionArtifacts

Promotion Boundary (Phase 1)

Authoritative output is patch bundle + metadata.

Sandbox runs do not merge to main or push to remote. The runtime:

1. Materializes repo at

   baseSha

   in sandbox-local workspace
2. Executes agent task
3. Runs verification commands
4. Exports patch artifact with touched files and verification results
5. Emits

   SandboxExecutionResult

   event with

   ExecutionArtifacts

Promotion is a separate operator decision:

• Restate workflow receives

  ExecutionArtifacts

• Operator reviews patch + verification summary
• Operator applies patch to host repo (or discards)
• Operator commits and pushes (if approved)

This keeps sandbox runs isolated and reversible.

k8s Job Integration

Job spec generation:

generateJobSpec(request, options)

Cold k8s Jobs for isolated story execution:

• Deterministic Job naming keyed by

  requestId

• Runtime image contract: Git, Bun, agent tooling,

  /workspace

  directory
• Environment-driven config:

  WORKFLOW_ID

  ,

  REQUEST_ID

  ,

  STORY_ID

  ,

  TASK_PROMPT_B64

  ,

  BASE_SHA

  , etc.
• Resource limits:

  500m-2

  CPU,

  1-4Gi

  memory (configurable)
• TTL cleanup: auto-delete after 5 minutes (default)
• Active deadline: 1 hour max runtime (default)
• No automatic retries (

  backoffLimit: 0

  )
• Security: non-root (UID 1000), no privilege escalation, capabilities dropped

Runtime contract:

1. Decode

   TASK_PROMPT_B64

   from env
2. Call

   materializeRepo()

   at

   BASE_SHA

3. Execute agent with task
4. Run verification commands (if

   VERIFICATION_COMMANDS_B64

   set)
5. Call

   generatePatchArtifact()

   with results
6. Emit

   SandboxExecutionResult

   event with

   ExecutionArtifacts

7. Exit 0 (success) or non-zero (failure)

Cancellation: Delete Job resource (SIGTERM to container)

Job deletion:

generateJobDeletion(requestId)

{ name, namespace, propagationPolicy }

See

k8s/agent-runner.yaml

Topology Impact

• Story 2: Added contract types and Job spec generation
• Story 3: Added repo materialization and artifact export helpers
• ADR-0221 phase 1: added explicit local sandbox isolation primitives — deterministic sandbox identity, deterministic local sandbox paths, per-sandbox env materialization, minimal/full mode vocabulary, and a JSON registry helper for host-worker sandboxes
• ADR-0221 phase 2: wired those local helpers into the real host-worker

  system/agent-dispatch

  local backend so sandbox runs now allocate deterministic paths under

  ~/.joelclaw/sandboxes/

  , materialize

  .sandbox.env

  , persist registry state, and carry

  localSandbox

  metadata in inbox snapshots
• ADR-0221 phase 3/4/5/6: phase 3 added terminal retention/cleanup policy (

  cleanupAfter

  + registry metadata), opportunistic pruning of expired local sandboxes on new-run startup, copy-first

  .devcontainer

  materialization helpers with exclusion rules for env/secret junk, live sandbox env injection so the agent process actually sees the reserved runtime identity, a hash-preserving sandbox identity fix after live dogfood exposed path collisions from long shared requestId prefixes, abbreviated-

  baseSha

  acceptance during repo materialization, truthful failed inbox snapshots when dispatch crashes before normal terminal writeback, and a repeatable operator probe at

  bun scripts/verify-local-sandbox-dispatch.ts

  ; phase 4 adds

  sandboxMode=minimal|full

  through the workload front door, requested-cwd mapping inside the cloned checkout, compose-backed full local mode startup, the reality that stale Restate workers can reject

  workload/requested

  until restarted and reloaded, a recursion guard because sandboxed stage runs were able to call

  scripts/verify-workload-full-mode.ts

  /

  joelclaw workload run

  from inside the sandbox and spawn nested canaries instead of terminating honestly, and a guarded workflow-rig proof run (

  WR_20260310_013158

  ) that completes terminally with healthy compose startup plus clean teardown; phase 5 adds the operator-facing CLI surface

  joelclaw workload sandboxes list|cleanup|janitor

  so retained sandboxes can be inspected and janitored on demand instead of only during startup opportunistic pruning, and the operator surfaces now reconcile registry entries against per-sandbox metadata before reporting or deleting so old partial writeback residue stops lying about terminal state; phase 6 makes janitoring scheduled instead of purely manual via repo-managed launchd service

  com.joel.local-sandbox-janitor

  , which runs

  scripts/local-sandbox-janitor.sh

  →

  joelclaw workload sandboxes janitor

  at load and every 30 minutes
• Future: Runtime image build, hot-image CronJob, warm-pool scheduler, Restate integration

Current state: the host-worker local sandbox path is now using the local-isolation helpers in production code, the package has a concurrent proof that two local sandboxes keep distinct compose identity plus copied devcontainer state, guarded full-mode workflow-rig dogfood closes terminally, and cleanup now has both on-demand CLI surfaces and scheduled launchd janitoring. Follow-on work is now about deeper runtime ergonomics and debugging any remaining non-terminal stale residues, not missing basic cleanup automation.

11) Verification Commands (Health + Wiring)

Core topology

# Colima + VM IP
colima status --json

# Kubernetes control plane + node
kubectl cluster-info
kubectl get nodes -o wide

# Core workloads
kubectl get pods -n joelclaw -o wide
kubectl get svc -n joelclaw -o wide

Host supervision

# Worker supervisor launchd state
launchctl print gui/$(id -u)/com.joel.system-bus-worker | rg "state =|pid =|last exit code"

# Gateway / Caddy / Talon
launchctl print gui/$(id -u)/com.joel.gateway | rg "state =|pid ="
launchctl print gui/$(id -u)/com.joel.caddy | rg "state =|pid ="
launchctl print gui/$(id -u)/com.joel.talon | rg "state =|pid ="

# Talon health
curl -s http://127.0.0.1:9999/health

Worker role split

# Parse role counts directly from source lists
python - <<'PY'
import re
from pathlib import Path
for f,name in [('packages/system-bus/src/inngest/functions/index.host.ts','host'),('packages/system-bus/src/inngest/functions/index.cluster.ts','cluster')]:
    txt=Path(f).read_text()
    body=re.search(rf'export const {name}FunctionDefinitions = \[(.*?)\];', txt, re.S).group(1)
    count=sum(1 for line in body.splitlines() if line.strip() and not line.strip().startswith('//'))
    print(name, count)
PY

# Inngest app ID derivation logic
rg -n "INNGEST_APP_ID|system-bus-host|system-bus-cluster|WORKER_ROLE" packages/system-bus/src/inngest/client.ts

Event flow trace

# Send event
joelclaw send <event> -d '<json>'

# Trace event and resulting runs
joelclaw events --prefix <event-prefix> --hours 1 --count 20
joelclaw event <event-id>
joelclaw runs --hours 1 --count 20
joelclaw run <run-id>

# Telemetry correlation
joelclaw otel search "<component_or_action>" --hours 1

Networking

# Caddy route config
caddy validate --config ~/.local/caddy/Caddyfile

# Listening ports snapshot
/usr/sbin/lsof -iTCP -sTCP:LISTEN -n -P

# Tailscale runtime (if daemon available)
tailscale status --json

12) Known Unknowns (Do Not Guess)

• Tailscale daemon state is not readable in this environment.

  • tailscale status --json

    -> failed to connect.
  • UNKNOWN — needs manual verification

• docs/architecture.md

  ,

  docs/deploy.md

  ,

  docs/observability.md

  are absent in-repo.
  • UNKNOWN — needs manual verification
• Exact command-line ownership of all Colima ssh forwarding ports (

  64784

  ,

  64785

  ,

  9627

  , etc.)
  • UNKNOWN — needs manual verification
• Ingress controller runtime status for

  k8s/docs-api-ingress.yaml

  • UNKNOWN — needs manual verification

13) Mandatory Update Policy (Non-Optional)

Update this skill in the same change whenever any of these change:

1. Worker runtime wiring

   • serve.ts

     ,

     client.ts

     ,

     index.host.ts

     ,

     index.cluster.ts

   • WORKER_ROLE

     , app IDs, serveHost behavior, registration path
2. Supervision/process topology
   • any

     ~/Library/LaunchAgents/com.joel*.plist

   • infra/worker-supervisor/*

     , Talon behavior, gateway launch script/label
3. Kubernetes topology
   • any file under

     k8s/

   • Helm values affecting core services (

     livekit

     ,

     pds

     , etc.)
   • Service type/port changes (NodePort/ClusterIP)
4. Networking/ingress
   • Caddyfile route/port changes
   • Tailscale/Funnel hostnames or ingress path changes
   • Colima/VM networking model changes
5. Storage topology
   • Redis keyspace contracts for gateway/webhook routing
   • Typesense telemetry collection/schema changes
   • NAS mount/fallback/queue contract changes
6. Observability/tracing
   • OTEL emit endpoint/token behavior
   • telemetry storage path changes (Typesense/Convex/Sentry)
   • Langfuse integration points
7. CLI control-plane routing
   • command families moved to different endpoints/services
8. ADR status changes affecting topology
   • especially ADR-0048, 0088, 0089, 0144, 0155, 0156, 0159, 0182, 0187

If any item above changed and this skill was not updated, this skill is stale and non-canonical.