# Observability Instrumentation + Control API Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use /ship (recommended) or /executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Ship additive observability and control APIs while old plans/journals preserve current runtime behavior. **Architecture:** Centralize optional-field normalization and capability detection in `src/observability.js`. Add deep state modules behind `src/control-api.js`; keep journal authoritative; use durable attempt/control/submission/revision identities. One strict multi-wave DAG serializes shared-file edits and gates legacy compatibility after each layer. **Tech Stack:** Node.js CommonJS, SQLite journal, Unix-socket HTTP/SSE control API, Bash integration tests, Rust TUI build gate. --- ## Wave Plan | Wave | Tasks | Files touched | Safe to parallelize? | |---|---|---|---| | 1 | Task 1 | `src/observability.js`, unit test, compatibility manifest | single foundation | | 2 | Tasks 2–7 | producer files; six distinct modules/tests | yes — zero file overlap; all depend only on Task 1 | | 3 | Task 8 | `src/control-api.js`, control API contract test | single integration hotspot | | 4 | Task 9 | legacy end-to-end fixtures/test | single compatibility gate | | 5 | Task 10 | runtime verification test | single terminal gate | Base branch: `feat/cursor-codex-preset`. It contains current `main` plus required `presets/cursor-codex.json`; verified with `git ls-tree`. Preset: `cursor-codex`. Gate mode: `strict`. ## File map - `src/observability.js`: sole optional-field compatibility and capability boundary. - `src/runner.js`, `src/engine/gates.js`: attempt lifecycle and enriched producers. - `src/event-feed.js`: journal/run-log event projection, composite cursor, SSE replay/tail. - `src/task-control.js`: task-scoped pause/resume/kill state machine. - `src/control-launch.js`: authenticated idempotent daemon submission. - `src/control-config.js`: config catalog/provenance and concurrency-safe mutation. - `src/plan-revisions.js`: validated durable revision state machine. - `src/control-api.js`: route wiring only; modules own behavior. - `spec/OBSERVABILITY-COMPAT.md`: versioned legacy deletion manifest. ### Task 1: Compatibility boundary + deletion manifest **Wave:** 1 **Blocks:** Tasks 2–7 **Blocked by:** — **Files:** - Create: `src/observability.js` - Create: `src/observability.test.js` - Create: `spec/OBSERVABILITY-COMPAT.md` **Contract:** - Export `createAttemptContext`, `normalizeTaskObservability`, `normalizeEventObservability`, `observabilityCapabilities`, `redactObservabilityValue` per design §1. - Normalization MUST accept records with every new field absent and preserve old execution meaning. - Missing `attemptId` MUST remain `null`; never synthesize correlation. - Redaction MUST remove credentials, auth headers, tokens, secret-like keys, and environment values recursively without mutating input. - Manifest MUST list legacy-only functions/branches, fixture paths, capability keys, inventory proof format, and canonical enriched code that survives retirement. **Behavior:** malformed optional substructures become unavailable capability, not run failure; newly authored structured fields still validate at author boundary. Functions remain pure except attempt-ID mint input. **Acceptance:** - Run: `node --test src/observability.test.js` - Expected: PASS; legacy-empty, enriched, partial, redaction, immutability, and capability fixtures green. - [ ] Write failing tests. - [ ] Implement contracts. - [ ] Write deletion manifest. - [ ] Run acceptance. - [ ] Commit: `git add src/observability.js src/observability.test.js spec/OBSERVABILITY-COMPAT.md && git commit -m "feat: add observability compatibility boundary"` ### Task 2: Attempt lifecycle + producer enrichment + acceptance **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Modify: `src/runner.js` - Modify: `src/engine/gates.js` - Modify: `spec/events.schema.json` - Modify: `src/test/runner-transcript.test.js` - Modify: `src/test/engine-gates.test.js` - Create: `src/test/observability-producers.test.js` **Contract:** - Append `attempt.started {task,phase,ordinal,attemptId,leaseId,processGeneration}` before dispatch; reuse only same reconciled lease; mint on new lease/redispatch/fixer/phase. - Stamp attempt ID on kinds listed in design §2. - Preserve every existing kind/payload field. - Enrich `gate0.fail`, review completion, `task.duration`, `task.diff` exactly per design. - Add optional task `acceptance[]`; missing field preserves current behavior. Emit `acceptance.result` only for structured criteria. - Terminal metrics use `(task,attemptId,phase)` idempotency and grouped append. **Behavior:** monotonic duration; actual gate argv/cwd; review parser failure emits no fake findings; git measurement failure omits diff without failing task; legacy free-text acceptance unchanged. **Acceptance:** - Run: `node --test src/test/observability-producers.test.js src/test/runner-transcript.test.js src/test/engine-gates.test.js` - Expected: PASS; retry/resume IDs, exact-once terminal metrics, enriched payloads, parser failure, and no-field legacy path green. - [ ] Write failing tests. - [ ] Implement lifecycle and enrichment. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: emit correlated attempt observability`. ### Task 3: Authoritative event feed + SSE replay **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Create: `src/event-feed.js` - Create: `src/event-feed.test.js` **Contract:** - Export `readEvents`, `openTaskEventStream`, `encodeEventCursor`, `decodeEventCursor`. - Cursor: opaque base64url `{v:1,j,l}`; event IDs `j:` and `l::`. - Deterministic order `(ts,sourcePriority,sourceCursor)`; journal state/facts and run-log transcript/activity remain source-partitioned. - Default/max page `500/2000`; high-water replay→tail handoff; replay-gap `409`; malformed cursor `400`. - SSE honors `Last-Event-ID` over `since`, heartbeat cadence, terminal close, byte/record backpressure bounds. **Behavior:** exact task filter; legacy task-only stream works without attempt ID; all output passes shared redaction; no counterfeit journal sequences or cross-source dedup. **Acceptance:** - Run: `node --test src/event-feed.test.js` - Expected: PASS; pagination, merge ties, cursor skew/gap, reconnect, no-gap handoff, slow consumer, terminal close, legacy fixture, secret canaries green. - [ ] Write failing tests. - [ ] Implement feed. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: add resumable event feed`. ### Task 4: Task-scoped control state machine **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Create: `src/task-control.js` - Create: `src/task-control.test.js` **Contract:** - Export `controlTask({run,taskId,attemptId?,verb,requestId})` for `kill|pause|resume`. - Journal requested/applied/failed payloads from design §7. - Compare-and-act under task mutex + lease CAS; immediately revalidate lease, generation, PID, OS start identity. - Missing attempt ID targets authoritative current task attempt for legacy callers; supplied stale ID → `409`. **Behavior:** repeated request returns prior result; crash reconciliation never re-signals unverified PID; task pause blocks only target; no run-scoped fallback; no active task → `409`. **Acceptance:** - Run: `node --test src/task-control.test.js` - Expected: PASS; task isolation, PID reuse, TOCTOU lease replacement, idempotence, crash replay, legacy no-ID, and all verbs green. - [ ] Write failing tests. - [ ] Implement state machine. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: add task-scoped controls`. ### Task 5: Idempotent run launch service **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Create: `src/control-launch.js` - Create: `src/control-launch.test.js` **Contract:** - Export `submitRun(request,{idempotencyKey,allowedRoots})`. - Reuse exact CLI plan/preset/config/preflight entrypoints. - Confine project to allowed roots and plan realpath to project `docs/plans`. - Atomically persist unique `{key,requestHash,runId,state:'accepted'}` before spawn/queue; resume accepted record idempotently. **Behavior:** same key/request returns result; changed request `409`; invalid preflight structured `422`; path escape `403`; `202` only after durable acceptance; overrides redacted. **Acceptance:** - Run: `node --test src/control-launch.test.js` - Expected: PASS; CLI parity with old plan, duplicate/crash matrix, confinement, preflight failures, and secret canaries green. - [ ] Write failing tests. - [ ] Implement service. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: add idempotent run launch`. ### Task 6: Config read + concurrency-safe mutation **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Create: `src/control-config.js` - Create: `src/control-config.test.js` - Modify: `src/config.js` - Modify: `src/config.test.js` **Contract:** - Export preset catalog, field schema, defaults read/write, effective run config, and mid-run patch services used by routes in design §9. - Reuse existing resolver/provenance; expose `{value,source,immutable,mutationClass}`. - Require `If-Match`; stale → `412`; unknown/locked fields reject whole request; defaults atomic and future-runs-only. **Behavior:** no parallel config model; missing legacy provenance becomes `source:'unknown'`; secrets never returned; mutation failure leaves prior file/state byte-equivalent. **Acceptance:** - Run: `node --test src/control-config.test.js src/config.test.js` - Expected: PASS; provenance, classification, CAS conflict, atomic failure, locked field, legacy unknown, and secret canaries green. - [ ] Write failing tests. - [ ] Extend resolver seam minimally. - [ ] Implement service. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: expose mutable config contracts`. ### Task 7: Durable plan revision service **Wave:** 2 **Blocks:** Task 8 **Blocked by:** Task 1 **Files:** - Create: `src/plan-revisions.js` - Create: `src/plan-revisions.test.js` **Contract:** - Export `readEffectivePlan` and `revisePlan({ops,ifMatch,idempotencyKey})`. - Implement closed `PlanRevisionOp` union, request hash, revision IDs, numbered snapshots, requested/applied journal protocol, cache hash reconciliation. - Only unstarted tasks mutate. Meaningful started-task update creates deterministic `followup::` with `afterTerminal` any-terminal predecessor. - Preserve unknown legacy fields byte-semantically through read/rewrite. **Behavior:** stop dispatch then drain without kill; drain failure leaves paused; stale revision `412`; same key/different hash `409`; requested-only crash cases replay exactly per design; corrupt hash halts. **Acceptance:** - Run: `node --test src/plan-revisions.test.js` - Expected: PASS; every op, DAG validation, started follow-up after failed/cancelled source, drain failure, snapshot, crash matrix, idempotence, and legacy round-trip green. - [ ] Write failing tests. - [ ] Implement service. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: add durable plan revisions`. ### Task 8: Control API route integration **Wave:** 3 **Blocks:** Task 9 **Blocked by:** Tasks 2–7 **Files:** - Modify: `src/control-api.js` - Modify: `src/test/control-api.test.js` **Contract:** - Add B1–B6 routes exactly as design §§4–10. - Reuse existing Bearer timing-safe auth and JSON/error response conventions. - Route handlers validate HTTP boundary only, then delegate to Tasks 3–7 modules. - Decisions expose stored richness + capabilities without inventing meanings. **Behavior:** preserve all existing routes/statuses; missing optional fields are unavailable, never errors; enforce `400/403/404/409/412/422/500` model; redact every response/SSE frame. **Acceptance:** - Run: `node --test src/test/control-api.test.js` - Expected: PASS; old routes unchanged; every new route auth/status/body; legacy/enriched decisions/events; SSE; controls; launch/config/revision wiring green. - [ ] Write failing route contracts. - [ ] Wire modules. - [ ] Run acceptance. - [ ] Commit declared files only with message `feat: expose observability control API`. ### Task 9: Legacy active-run compatibility gate **Wave:** 4 **Blocks:** Task 10 **Blocked by:** Task 8 **Files:** - Create: `test/fixtures/observability-legacy/` — old plan/journal/run-log/config fixtures with all new fields absent. - Create: `test/observability-legacy.test.sh` **Contract:** - Exercise old plan through initial dispatch, retry, warm resume, cold resume, API reads, no-attempt task control, and unsupported revision rejection. - Assert scheduling, gate, review, resume, and existing control results match pre-upgrade fixture expectations. - Inventory manifest fixture paths MUST match `spec/OBSERVABILITY-COMPAT.md` exactly. **Behavior:** test uses temporary repo/home/runstate only; never touches live runstate or active processes. Any new-field requirement fails test. **Acceptance:** - Run: `bash test/observability-legacy.test.sh` - Expected: PASS with explicit checks for every compatibility lifecycle. - [ ] Build fixtures from existing pre-upgrade shapes. - [ ] Write integration gate. - [ ] Run acceptance. - [ ] Commit declared files only with message `test: lock legacy run compatibility`. ### Task 10: Full build + launcher runtime probe **Wave:** 5 **Blocks:** — **Blocked by:** Task 9 **Files:** - Create: `test/observability-runtime.test.sh` **Contract:** - Probe actual `bin/runplan` launcher with legacy fixture and `cursor-codex` preset from integration branch. - Probe control API starts against updated modules and serves authenticated legacy + enriched event reads. - Assert launcher resolves repository `src/runner.js`, not alternate/debug artifact. **Acceptance:** - Run: `bash test/observability-runtime.test.sh && make test` - Expected: PASS; Rust release build/clippy/tests, complete Node/Bash suite, launcher/runtime probes, zero warnings. - [ ] Write runtime probe. - [ ] Run targeted probe. - [ ] Run full acceptance. - [ ] Commit: `git add test/observability-runtime.test.sh && git commit -m "test: verify observability runtime"` ## Decision enumeration - `g-protected-observability` is already resolved by user's explicit execution request. Tasks modify only declared `src/` and `spec/` paths. - No irreversible, graph-changing, external-input, or unresolved architecture decision remains. - Legacy retirement remains deferred; it is a separate explicitly scheduled plan after inventory reaches zero. ## Self-review - Source brief A1–A7 and B1–B6 map to Tasks 2 and 5–8. - Backward compatibility maps to Tasks 1, 2, 8, 9, 10. - Same-wave file overlap: none. - Every referenced seam is defined here or in design spec. - No implementation body embedded; tasks pin contracts and executable acceptance.