# Observability Instrumentation + Control API Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use /ship (recommended) or /executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Ship additive observability and control APIs while old plans/journals preserve current runtime behavior.

**Architecture:** Centralize optional-field normalization and capability detection in `src/observability.js`. Add deep state modules behind `src/control-api.js`; keep journal authoritative; use durable attempt/control/submission/revision identities. One strict multi-wave DAG serializes shared-file edits and gates legacy compatibility after each layer.

**Tech Stack:** Node.js CommonJS, SQLite journal, Unix-socket HTTP/SSE control API, Bash integration tests, Rust TUI build gate.

---

## Wave Plan

| Wave | Tasks | Files touched | Safe to parallelize? |
|---|---|---|---|
| 1 | Task 1 | `src/observability.js`, unit test, compatibility manifest | single foundation |
| 2 | Tasks 2–7 | producer files; six distinct modules/tests | yes — zero file overlap; all depend only on Task 1 |
| 3 | Task 8 | `src/control-api.js`, control API contract test | single integration hotspot |
| 4 | Task 9 | legacy end-to-end fixtures/test | single compatibility gate |
| 5 | Task 10 | runtime verification test | single terminal gate |

Base branch: `feat/cursor-codex-preset`. It contains current `main` plus required `presets/cursor-codex.json`; verified with `git ls-tree`. Preset: `cursor-codex`. Gate mode: `strict`.

## File map

- `src/observability.js`: sole optional-field compatibility and capability boundary.
- `src/runner.js`, `src/engine/gates.js`: attempt lifecycle and enriched producers.
- `src/event-feed.js`: journal/run-log event projection, composite cursor, SSE replay/tail.
- `src/task-control.js`: task-scoped pause/resume/kill state machine.
- `src/control-launch.js`: authenticated idempotent daemon submission.
- `src/control-config.js`: config catalog/provenance and concurrency-safe mutation.
- `src/plan-revisions.js`: validated durable revision state machine.
- `src/control-api.js`: route wiring only; modules own behavior.
- `spec/OBSERVABILITY-COMPAT.md`: versioned legacy deletion manifest.

### Task 1: Compatibility boundary + deletion manifest

**Wave:** 1
**Blocks:** Tasks 2–7
**Blocked by:** —

**Files:**
- Create: `src/observability.js`
- Create: `src/observability.test.js`
- Create: `spec/OBSERVABILITY-COMPAT.md`

**Contract:**
- Export `createAttemptContext`, `normalizeTaskObservability`, `normalizeEventObservability`, `observabilityCapabilities`, `redactObservabilityValue` per design §1.
- Normalization MUST accept records with every new field absent and preserve old execution meaning.
- Missing `attemptId` MUST remain `null`; never synthesize correlation.
- Redaction MUST remove credentials, auth headers, tokens, secret-like keys, and environment values recursively without mutating input.
- Manifest MUST list legacy-only functions/branches, fixture paths, capability keys, inventory proof format, and canonical enriched code that survives retirement.

**Behavior:** malformed optional substructures become unavailable capability, not run failure; newly authored structured fields still validate at author boundary. Functions remain pure except attempt-ID mint input.

**Acceptance:**
- Run: `node --test src/observability.test.js`
- Expected: PASS; legacy-empty, enriched, partial, redaction, immutability, and capability fixtures green.

- [ ] Write failing tests.
- [ ] Implement contracts.
- [ ] Write deletion manifest.
- [ ] Run acceptance.
- [ ] Commit: `git add src/observability.js src/observability.test.js spec/OBSERVABILITY-COMPAT.md && git commit -m "feat: add observability compatibility boundary"`

### Task 2: Attempt lifecycle + producer enrichment + acceptance

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Modify: `src/runner.js`
- Modify: `src/engine/gates.js`
- Modify: `spec/events.schema.json`
- Modify: `src/test/runner-transcript.test.js`
- Modify: `src/test/engine-gates.test.js`
- Create: `src/test/observability-producers.test.js`

**Contract:**
- Append `attempt.started {task,phase,ordinal,attemptId,leaseId,processGeneration}` before dispatch; reuse only same reconciled lease; mint on new lease/redispatch/fixer/phase.
- Stamp attempt ID on kinds listed in design §2.
- Preserve every existing kind/payload field.
- Enrich `gate0.fail`, review completion, `task.duration`, `task.diff` exactly per design.
- Add optional task `acceptance[]`; missing field preserves current behavior. Emit `acceptance.result` only for structured criteria.
- Terminal metrics use `(task,attemptId,phase)` idempotency and grouped append.

**Behavior:** monotonic duration; actual gate argv/cwd; review parser failure emits no fake findings; git measurement failure omits diff without failing task; legacy free-text acceptance unchanged.

**Acceptance:**
- Run: `node --test src/test/observability-producers.test.js src/test/runner-transcript.test.js src/test/engine-gates.test.js`
- Expected: PASS; retry/resume IDs, exact-once terminal metrics, enriched payloads, parser failure, and no-field legacy path green.

- [ ] Write failing tests.
- [ ] Implement lifecycle and enrichment.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: emit correlated attempt observability`.

### Task 3: Authoritative event feed + SSE replay

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Create: `src/event-feed.js`
- Create: `src/event-feed.test.js`

**Contract:**
- Export `readEvents`, `openTaskEventStream`, `encodeEventCursor`, `decodeEventCursor`.
- Cursor: opaque base64url `{v:1,j,l}`; event IDs `j:<seq>` and `l:<offset>:<length>`.
- Deterministic order `(ts,sourcePriority,sourceCursor)`; journal state/facts and run-log transcript/activity remain source-partitioned.
- Default/max page `500/2000`; high-water replay→tail handoff; replay-gap `409`; malformed cursor `400`.
- SSE honors `Last-Event-ID` over `since`, heartbeat cadence, terminal close, byte/record backpressure bounds.

**Behavior:** exact task filter; legacy task-only stream works without attempt ID; all output passes shared redaction; no counterfeit journal sequences or cross-source dedup.

**Acceptance:**
- Run: `node --test src/event-feed.test.js`
- Expected: PASS; pagination, merge ties, cursor skew/gap, reconnect, no-gap handoff, slow consumer, terminal close, legacy fixture, secret canaries green.

- [ ] Write failing tests.
- [ ] Implement feed.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: add resumable event feed`.

### Task 4: Task-scoped control state machine

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Create: `src/task-control.js`
- Create: `src/task-control.test.js`

**Contract:**
- Export `controlTask({run,taskId,attemptId?,verb,requestId})` for `kill|pause|resume`.
- Journal requested/applied/failed payloads from design §7.
- Compare-and-act under task mutex + lease CAS; immediately revalidate lease, generation, PID, OS start identity.
- Missing attempt ID targets authoritative current task attempt for legacy callers; supplied stale ID → `409`.

**Behavior:** repeated request returns prior result; crash reconciliation never re-signals unverified PID; task pause blocks only target; no run-scoped fallback; no active task → `409`.

**Acceptance:**
- Run: `node --test src/task-control.test.js`
- Expected: PASS; task isolation, PID reuse, TOCTOU lease replacement, idempotence, crash replay, legacy no-ID, and all verbs green.

- [ ] Write failing tests.
- [ ] Implement state machine.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: add task-scoped controls`.

### Task 5: Idempotent run launch service

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Create: `src/control-launch.js`
- Create: `src/control-launch.test.js`

**Contract:**
- Export `submitRun(request,{idempotencyKey,allowedRoots})`.
- Reuse exact CLI plan/preset/config/preflight entrypoints.
- Confine project to allowed roots and plan realpath to project `docs/plans`.
- Atomically persist unique `{key,requestHash,runId,state:'accepted'}` before spawn/queue; resume accepted record idempotently.

**Behavior:** same key/request returns result; changed request `409`; invalid preflight structured `422`; path escape `403`; `202` only after durable acceptance; overrides redacted.

**Acceptance:**
- Run: `node --test src/control-launch.test.js`
- Expected: PASS; CLI parity with old plan, duplicate/crash matrix, confinement, preflight failures, and secret canaries green.

- [ ] Write failing tests.
- [ ] Implement service.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: add idempotent run launch`.

### Task 6: Config read + concurrency-safe mutation

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Create: `src/control-config.js`
- Create: `src/control-config.test.js`
- Modify: `src/config.js`
- Modify: `src/config.test.js`

**Contract:**
- Export preset catalog, field schema, defaults read/write, effective run config, and mid-run patch services used by routes in design §9.
- Reuse existing resolver/provenance; expose `{value,source,immutable,mutationClass}`.
- Require `If-Match`; stale → `412`; unknown/locked fields reject whole request; defaults atomic and future-runs-only.

**Behavior:** no parallel config model; missing legacy provenance becomes `source:'unknown'`; secrets never returned; mutation failure leaves prior file/state byte-equivalent.

**Acceptance:**
- Run: `node --test src/control-config.test.js src/config.test.js`
- Expected: PASS; provenance, classification, CAS conflict, atomic failure, locked field, legacy unknown, and secret canaries green.

- [ ] Write failing tests.
- [ ] Extend resolver seam minimally.
- [ ] Implement service.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: expose mutable config contracts`.

### Task 7: Durable plan revision service

**Wave:** 2
**Blocks:** Task 8
**Blocked by:** Task 1

**Files:**
- Create: `src/plan-revisions.js`
- Create: `src/plan-revisions.test.js`

**Contract:**
- Export `readEffectivePlan` and `revisePlan({ops,ifMatch,idempotencyKey})`.
- Implement closed `PlanRevisionOp` union, request hash, revision IDs, numbered snapshots, requested/applied journal protocol, cache hash reconciliation.
- Only unstarted tasks mutate. Meaningful started-task update creates deterministic `followup:<revisionId>:<taskId>` with `afterTerminal` any-terminal predecessor.
- Preserve unknown legacy fields byte-semantically through read/rewrite.

**Behavior:** stop dispatch then drain without kill; drain failure leaves paused; stale revision `412`; same key/different hash `409`; requested-only crash cases replay exactly per design; corrupt hash halts.

**Acceptance:**
- Run: `node --test src/plan-revisions.test.js`
- Expected: PASS; every op, DAG validation, started follow-up after failed/cancelled source, drain failure, snapshot, crash matrix, idempotence, and legacy round-trip green.

- [ ] Write failing tests.
- [ ] Implement service.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: add durable plan revisions`.

### Task 8: Control API route integration

**Wave:** 3
**Blocks:** Task 9
**Blocked by:** Tasks 2–7

**Files:**
- Modify: `src/control-api.js`
- Modify: `src/test/control-api.test.js`

**Contract:**
- Add B1–B6 routes exactly as design §§4–10.
- Reuse existing Bearer timing-safe auth and JSON/error response conventions.
- Route handlers validate HTTP boundary only, then delegate to Tasks 3–7 modules.
- Decisions expose stored richness + capabilities without inventing meanings.

**Behavior:** preserve all existing routes/statuses; missing optional fields are unavailable, never errors; enforce `400/403/404/409/412/422/500` model; redact every response/SSE frame.

**Acceptance:**
- Run: `node --test src/test/control-api.test.js`
- Expected: PASS; old routes unchanged; every new route auth/status/body; legacy/enriched decisions/events; SSE; controls; launch/config/revision wiring green.

- [ ] Write failing route contracts.
- [ ] Wire modules.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `feat: expose observability control API`.

### Task 9: Legacy active-run compatibility gate

**Wave:** 4
**Blocks:** Task 10
**Blocked by:** Task 8

**Files:**
- Create: `test/fixtures/observability-legacy/` — old plan/journal/run-log/config fixtures with all new fields absent.
- Create: `test/observability-legacy.test.sh`

**Contract:**
- Exercise old plan through initial dispatch, retry, warm resume, cold resume, API reads, no-attempt task control, and unsupported revision rejection.
- Assert scheduling, gate, review, resume, and existing control results match pre-upgrade fixture expectations.
- Inventory manifest fixture paths MUST match `spec/OBSERVABILITY-COMPAT.md` exactly.

**Behavior:** test uses temporary repo/home/runstate only; never touches live runstate or active processes. Any new-field requirement fails test.

**Acceptance:**
- Run: `bash test/observability-legacy.test.sh`
- Expected: PASS with explicit checks for every compatibility lifecycle.

- [ ] Build fixtures from existing pre-upgrade shapes.
- [ ] Write integration gate.
- [ ] Run acceptance.
- [ ] Commit declared files only with message `test: lock legacy run compatibility`.

### Task 10: Full build + launcher runtime probe

**Wave:** 5
**Blocks:** —
**Blocked by:** Task 9

**Files:**
- Create: `test/observability-runtime.test.sh`

**Contract:**
- Probe actual `bin/runplan` launcher with legacy fixture and `cursor-codex` preset from integration branch.
- Probe control API starts against updated modules and serves authenticated legacy + enriched event reads.
- Assert launcher resolves repository `src/runner.js`, not alternate/debug artifact.

**Acceptance:**
- Run: `bash test/observability-runtime.test.sh && make test`
- Expected: PASS; Rust release build/clippy/tests, complete Node/Bash suite, launcher/runtime probes, zero warnings.

- [ ] Write runtime probe.
- [ ] Run targeted probe.
- [ ] Run full acceptance.
- [ ] Commit: `git add test/observability-runtime.test.sh && git commit -m "test: verify observability runtime"`

## Decision enumeration

- `g-protected-observability` is already resolved by user's explicit execution request. Tasks modify only declared `src/` and `spec/` paths.
- No irreversible, graph-changing, external-input, or unresolved architecture decision remains.
- Legacy retirement remains deferred; it is a separate explicitly scheduled plan after inventory reaches zero.

## Self-review

- Source brief A1–A7 and B1–B6 map to Tasks 2 and 5–8.
- Backward compatibility maps to Tasks 1, 2, 8, 9, 10.
- Same-wave file overlap: none.
- Every referenced seam is defined here or in design spec.
- No implementation body embedded; tasks pin contracts and executable acceptance.
