# Observability Instrumentation + Control API — Design

audience: AI coding agents first. Contract-level: seams + decisions, NOT code bodies.

slug: `observability-control-api` · date: 2026-07-18 · execution preset: `cursor-codex`

## Goal

Add lossless attempt observability and operator control without changing behavior for plans and journals that omit new fields. Execute as one fail-closed DAG. Keep legacy compatibility in one removable boundary until old runs retire.

Source brief: `docs/specs/2026-07-18-observability-instrumentation-and-control-api.md`.

## Non-negotiable compatibility contract

1. Existing plan JSONL MUST parse and run unchanged.
2. New plan fields MUST remain optional. Missing fields MUST preserve current execution, scheduling, review, gate, resume, and control behavior.
3. Existing journal events MUST remain readable. Missing enriched fields MUST produce `null`, omitted output, or existing fallback output; they MUST NOT fail a run or fabricate evidence.
4. New producers MUST emit enriched fields when context exists. Producer enrichment MUST NOT require migration of an active run.
5. Consumers MUST detect field presence/capability, never infer compatibility from date, slug, plan path, or journal version.
6. Compatibility logic MUST live in `src/observability.js` plus explicit legacy fixtures. Core runner/control routes MUST NOT accumulate scattered `legacy` branches.
7. Retirement MUST require deleting `src/observability.js` legacy normalization paths and legacy fixtures only. Execution semantics MUST remain unchanged after deletion.

## Selected architecture

Use one long DAG with contract gates between layers:

```text
legacy/new plan + journal
          |
          v
  observability boundary ----> normalized optional projection
          |                              |
          v                              v
 enriched producers              control API readers
          |                              |
          +------ authoritative journal-+
                         |
                         v
              stream/control/config/plan APIs
```

Each gate runs legacy fixtures plus new-contract tests. Tasks sharing `src/runner.js` or `src/control-api.js` MUST execute in separate waves. File-disjoint tests/modules MAY run in parallel.

## Contracts

### 1. Compatibility + attempt context — `src/observability.js`

Own optional-field normalization, attempt identity, enriched event projection, and capability reporting.

Seams:

```text
createAttemptContext({ runId, taskId, phase, ordinal }): AttemptContext
normalizeTaskObservability(task): NormalizedTaskObservability
normalizeEventObservability(event): NormalizedEventObservability
observabilityCapabilities({ task, events }): ObservabilityCapabilities
```

`AttemptContext.attemptId` MUST be stable inside one execution attempt and unique across retries/phases. Resume MUST reuse an attempt ID only when resuming that same leased attempt; a redispatch/retry MUST mint a new ID.

Normalized optional fields:

- task acceptance: absent → `[]`; execution unchanged.
- event `attemptId`: absent → `null`; no heuristic correlation.
- gate detail, review findings, duration, diff: absent → unavailable; preserve existing message/state.
- decision detail: preserve existing record; missing optional detail remains unavailable.

`ObservabilityCapabilities` MUST report field-level availability. API consumers use it to hide unsupported detail without rejecting legacy runs.

### 2. Event producer enrichment — `src/runner.js`, `src/engine/gates.js`

Stamp `attemptId` on related events when attempt context exists:

- `task.dispatch`
- `dispatch.start`
- `dispatch.done`
- `task.usage`
- review-complete event
- `fixer.attempt`
- `task.duration`
- `task.diff`

Do NOT rename or remove existing kinds. Existing payload fields remain unchanged.

Required additive payloads:

```text
gate0.fail { task, message, failClass?, command?, cwd? }
review complete { verdict?, findings? }
task.duration { task, attemptId, phase, durationMs }
task.diff { task, attemptId, filesTouched, insertions, deletions }
```

`failClass`: `build|typecheck|lint|test|timeout|other`. `command` MUST be actual argv, not reconstructed shell text. Unknown classification → `other`.

Review findings:

```text
{ severity: 'blocker'|'major'|'minor', file?: string, line?: number, message: string }
```

Review parser failure MUST retain current review result/message and emit no fabricated findings.

Duration MUST use monotonic elapsed time. Emit exactly once per terminal attempt outcome, including failed/fixed/reviewed attempts where current orchestration considers attempt complete.

Diff MUST describe attempt branch delta against its attempt start/base. Git measurement failure MUST preserve task flow, omit `task.diff`, and emit existing diagnostic path; it MUST NOT report empty diff as evidence.

### 3. Acceptance criteria — plan parser + runner

Optional task seam:

```text
acceptance?: Array<{ id: string, text: string, kind: 'auto'|'manual' }>
```

Rules:

- Missing `acceptance` → current plan behavior.
- IDs MUST be unique within task when field exists.
- Auto criteria emit `acceptance.result { task, criterionId, result, evidenceRef? }` when mapped check completes.
- Manual criteria start `pending`; no automatic pass.
- Existing free-text acceptance remains accepted and MUST NOT be rewritten during plan load.
- Invalid structured acceptance on a newly authored plan fails validation before dispatch. Legacy free-text does not.

### 4. Decision projection

`GET /runs/:id/decisions` and `GET /runs/:id/decisions/:decisionId` MUST expose stored gated fields without inventing meaning:

```text
{ id, category?, needs?, why?, blast_radius?, options, task?, status }
```

Each option MAY be a legacy string or enriched `{ value, meaning }`. Preserve strings for old records. API MUST return capability metadata indicating whether option meanings and irreversible blast radius are present.

### 5. Raw events API

`GET /runs/:id/events?since=<seq>&task=<taskId>` returns authoritative journal order:

```text
{ events: Array<{ seq, source, kind, ts, taskId?, attemptId?, payload }>, nextSince, capabilities }
```

Rules:

- `since` is exclusive. Invalid/negative cursor → `400`.
- `task` filters exact task ID after authoritative decode.
- Journal remains authority. Derived run-log records MAY fill transcript-only gaps but MUST be labeled `source:'runlog'` and MUST NOT override journal facts.
- Sequence references MUST remain stable. Run-log-only records need a separate opaque cursor namespace; never counterfeit journal sequence numbers.
- Legacy events return `attemptId` absent/null and remain queryable.

### 6. Per-task live stream

`GET /runs/:id/tasks/:taskId/stream` uses SSE with resumable opaque cursor via `Last-Event-ID` or `since`.

- Stream only matching task activity/transcript records.
- Preserve source granularity; do not split stored chunks into fake tokens.
- Emit heartbeat comments using existing stream cadence.
- Slow consumers MUST be bounded and disconnected rather than growing memory.
- Terminal task emits final state then closes.
- Legacy runs without attempt IDs still stream by task ID.

### 7. Per-task attempt controls

Routes:

```text
POST /runs/:id/tasks/:taskId/kill
POST /runs/:id/tasks/:taskId/pause
POST /runs/:id/tasks/:taskId/resume
```

Resolve current active attempt through authoritative lease/process ownership. `attemptId` MAY be supplied as an optimistic concurrency guard; mismatch → `409`. Missing `attemptId` preserves task-scoped legacy targeting.

- No active task → `409` with current state.
- Kill targets owned process group for that task only and journals intent/result.
- Pause stops new work for target attempt without pausing whole run.
- Resume only resumes a task paused through this seam.
- Never fall through to run-scoped kill/pause/resume.

### 8. Run launch

`POST /runs` request:

```text
{ project, plan, preset, concurrency, overrides? }
```

Validate repository, plan, preset, concurrency, compatibility, and normal preflight before daemon submission. Return `202 { runId }` only after durable submission. Preflight failure → `422 { failures:[{ code, message, path? }] }`. Authentication and timing-safe token comparison reuse existing control API boundary.

Old plan launch MUST succeed when current CLI launch succeeds. API MUST NOT require structured acceptance or observability fields.

### 9. Config API

Routes:

```text
GET /presets
GET /config/schema
GET /config/defaults
PUT /config/defaults
GET /runs/:id/config
PATCH /runs/:id/config
```

Use existing config resolver/provenance. Do not create a parallel config model.

- Effective fields expose `{ value, source, immutable, mutationClass }`.
- `mutationClass`: `launch|mid-run|global`.
- Patch accepts only known `mid-run` fields. Unknown/locked fields fail whole request atomically with `409` or `422`; no partial mutation.
- Defaults mutation affects future runs only and uses atomic write.
- Secrets MUST never be returned.
- Missing legacy provenance returns current effective value with `source:'unknown'`; run continues.

### 10. Plan read + revision

Routes:

```text
GET /runs/:id/plan
POST /runs/:id/plan/revisions { ops: PlanRevisionOp[] }
```

Revision state machine:

1. Stop new dispatch.
2. Drain running tasks; NEVER kill.
3. Validate revision against current authoritative plan + journal.
4. Archive numbered snapshot atomically.
5. Append revision intent/result to journal.
6. Persist revised plan cache.
7. Reconcile and resume dispatch.

If drain fails, leave run paused and return explicit failure. Started tasks MUST NOT be rewritten; represent requested change as follow-up task. Only not-started tasks may be edited/reordered/removed. Preserve unknown legacy task fields round-trip. Revision replay MUST reconstruct effective plan from base + journaled revisions.

## Error model

- Authentication failure: existing status/body contract.
- Malformed JSON/query: `400`.
- Unknown run/task/decision: `404`.
- State conflict or stale `attemptId`: `409`.
- Semantically invalid launch/config/revision: `422` with structured failures.
- Internal storage/process failure: fail closed, journal diagnostic where authority is available, return `500` without secrets.
- Missing optional observability data is NOT an error.

## DAG gates

1. Compatibility foundation: legacy plan/journal fixtures run unchanged.
2. Attempt correlation: retry/resume uniqueness and propagation tests pass.
3. Producer enrichment: event schema and exact-once duration/diff tests pass.
4. Read surfaces: decisions/events APIs pass legacy + enriched fixtures.
5. Live/control surfaces: SSE resume/backpressure and task isolation tests pass.
6. Acceptance/config/launch: old CLI-equivalent plans launch; atomic validation tests pass.
7. Revision state machine: drain/crash/replay/follow-up tests pass.
8. Full gate: lint, typecheck, complete test suite, production build, launcher runtime probe.

Each gate MUST include at least one plan fixture with every new field absent. A gate failure HALTs DAG; no task auto-skips.

## Testing

- Unit: compatibility normalization, capability detection, attempt ID lifecycle, verdict/finding parsing, config mutation classification, revision validation.
- Contract: every route auth/status/body; legacy and enriched responses.
- Integration: old plan execution through `bin/runplan`; retry/resume; task-only pause/kill; SSE reconnect; API launch submission.
- Crash/replay: revision intent boundaries, config atomic write, launch durable submission.
- Regression: existing event kinds/routes/payload fields unchanged.
- Retirement proof: run suite with legacy adapter enabled; run enriched-only suite with legacy fixtures excluded. Document exact legacy fixture list as deletion manifest.

## Scope

Included: source brief A1–A7 and B1–B6, compatibility boundary, tests, schemas/docs required by contracts.

Excluded: UI implementation, event-kind renames, mandatory migration of active plans/journals, ship/land behavior changes, automatic legacy retirement.

## Architecture decisions

- Accepted: one multi-wave DAG. Gates isolate regressions while preserving end-to-end dependency order.
- Accepted: centralized observability compatibility boundary. Deletion test passes: removing it before retirement scatters optional-field handling across runner/API; after retirement its legacy branches and fixtures delete together.
- Accepted: additive field capability detection. Plan dates/schema guesses cannot represent partially enriched active runs.
- Accepted: task-scoped controls resolve authoritative active ownership. Attempt ID remains optional guard for legacy compatibility.
- Rejected: separate microservice/event store. SQLite journal already owns authority; another store creates split-brain risk.
- Rejected: migrate active plans before launch. Violates old-plan continuity requirement.
- Rejected: scattered null checks. Makes legacy retirement non-local and untestable.

## Retirement trigger

Retire compatibility only after runtime inventory proves zero nonterminal runs/plans lacking required enriched contracts and operator explicitly schedules retirement. Retirement plan MUST delete legacy normalization branches, legacy fixtures, and capability states together; it MUST NOT change runner scheduling/control semantics.