# Harness Observability — Design

audience: AI coding agents first. Contract-level: seams + decisions, NEVER code bodies.

slug: `harness-observability` · date: 2026-08-07 · engine: `modules/harness/v2` · preset: `codex`

## The one sentence

**The engine has two emit paths; collapse them into one.** `report()` (`v2/run.js:71-74`) journals *and* prints; `journal()` only journals. 41 event kinds are recorded, 4 reach the terminal. Every silence the user has complained about for three harness versions is a kind that took the second path. Filter for **display**, never for **recording** — then no future event can be added silently again. This is SSSF's stated rule (`one _emit() for print + trace so terminal and UI cannot drift`), and it is a deletion, not a layer.

## What is already built (do NOT rebuild)

The `2026-07-22-overdeck-observability-b3` plan **landed its consumer half**. Verified present on `origin/main`:

`collector/src/adapters/harness.ts` · `collector/src/server.ts` · `collector/src/actions.ts` · `apps/web/src/pages/api/collector/[...path].ts` · `apps/web/src/lib/collector-client.ts` · `apps/web/src/lib/harness-types.ts` · `packages/deck-ui/src/useSseStream.ts` · `AgentStatusBar.tsx` · `AgentFeed.tsx` · `DistanceToDone.tsx` · `RunCommandBar.tsx` · `apps/web/src/components/plans/AgentApp.tsx` · `PlanRunApp.tsx` · `apps/web/src/lib/panel-data.ts`

That plan's anchor pointed at `mega-plan-harness` (v1, dead) as the **producer** contract. The consumers were written against a producer that no longer exists. **No task in this plan may re-author c1/c2/w1/u1/u2** — an implementer told to "build observability" will rewrite working code. This spec makes the live v2 producer emit what the already-built consumer reads.

The same applies to `modules/harness/docs/specs/2026-07-18-observability-control-api-design.md` — its A-gap taxonomy (A7 note, A8 cost, A9 account, A10 watchdog) is the source of the literal placeholder strings still rendering in `/plans`. Those gap IDs are **retained verbatim** below; this spec closes them rather than inventing a new taxonomy. Its file targets (`src/observability.js`, `src/runner.js`) are v1 and are **void**.

## The user's four questions, mapped to defects

| Question | Defect | Site |
|---|---|---|
| what failed | `task-end` reaches the terminal only after `runPlan` resolves | `v2/bin/runplan.js:208` vs `:167-182` |
| why it failed | `quarantine` reason journaled, never printed; terminal shows only the collapsed `quality-quarantined (review or strict gate did not pass)` | `v2/quality.js:383-386`, `v2/run.js:669` |
| what did the agent do | coder transcript is written to disk and served by nothing — `taskLogPath` matches `/-(dispatch\|gate)-(\d+)\.log$/`, `childLogPath` never writes phase `dispatch` | `v2/control-api.js:554-569` vs `v2/run.js:1074-1076` |
| anything? | `/attempts` has no collector route → `PlanRunApp.tsx:630-638` short-circuits → `AutopsyStrip`, `BurnPanel`, `AttemptTimeline`, `AttemptDetailDrawer` never mount | no route in `collector/src/server.ts`, no regex in `[...path].ts:15-22` |

Rows 3 and 4 are a regex and two routing hops over data the engine **already produces**. They are the highest value-per-line in the set.

## Architecture decisions

**Keep NDJSON + the replay projection. Do NOT adopt SSSF's SQLite mirror.** SSSF's own doc states the DB is a rebuildable mirror ("losing `sssf.db` loses nothing"); its purpose is poll-by-`rowid`. Overdeck already has the raw record (`v2/journal.js`) *and* a replay projection (`v2/control-api.js:60-147`) *and* a landed SSE consumer. A second store buys a cursor we do not need and introduces split-brain. **Rejected** — same reasoning the 2026-07-18 design used to reject a separate event store.

**Lift SSSF's contracts, not its substrate:** gate evidence (`GateCheck{item, ok, note}`), verbatim-unparsed failure tails (`output_tail`), the process/pid table for hung agents, failure-first status defaults, and one-call run settlement.

**Collapse the two emitters** — the sole structural change. Every event is recorded *and* offered to the progress sink; a display policy decides what prints. Passes the deletion test: remove the policy and you get a firehose, not scattered `if (shouldPrint)` at 41 call sites.

**Retain `v2/supervision.js` projections only if they are fed.** `projectHeartbeat` (`:150`) and `projectRestartHistory` (`:171`) read four kinds (`supervision.heartbeat`, `supervision.crashed`, `supervision.restart-scheduled`, `supervision.restart-refused`) that **nothing emits and the schema does not declare**. Both are dead code today. Either the process-table work below feeds them or they are deleted — no third option.

**Schema is fail-closed and stays that way.** `v2/journal.js:289` throws `undeclared journal event kind`. Every new kind ships with its `spec/journal-events.schema.json` entry **in the same commit**. Never relax the validator.

---

## Wave 1 — the four defects (ships alone, answers all four questions)

**Sequencing inside wave 1.** 1.1 and 1.2 both rewrite `v2/run.js` emit sites and MUST NOT run concurrently: 1.1 collapses `report()`/`journal()` into `emit()`, 1.2 then changes what every call site passes. **1.1 runs alone; 1.2, 1.3 and 1.4 run in parallel after it** (1.3 is `control-api.js` + an engine-side constant, 1.4 is `collector` + `apps/web` — disjoint from each other and from 1.2).

### 1.1 One emitter — `v2/run.js`, `v2/bin/runplan.js`

Seam:

```text
emit(event): void            // records to journal AND offers to onProgress — the ONLY emit path
displayPolicy(event): DisplayDecision | null
DisplayDecision = { level: 'info'|'warn'|'error', text: string, detail?: string[] }
```

Behavior:

- `report()` and `journal()` inside `run.js` collapse to `emit()`. No call site may reach the journal without passing the progress sink.
- `displayPolicy` is a single table keyed by event kind, living beside `runplan.js`'s `reportProgress`. The table MUST be exhaustive over every kind declared in `spec/journal-events.schema.json` — a unit test asserts that, so no kind reaches the terminal unclassified in normal operation. Unknown kind → a one-line `info` default (`<kind> <task?>`), **never** silence: a backstop for a schema/table race, not the normal path.
- **Recording is never filtered; only display is.** Every kind still lands in the journal at full fidelity regardless of what the policy decides to print.
- **High-frequency kinds print on edge, not on tick.** `attempt.heartbeat` fires every `ATTEMPT_HEARTBEAT_MS = 60_000` per in-flight attempt (`dispatch.js:12`); at `--concurrency 4` on a 25-minute task that is ~100 lines burying the four the user needs. Contract: heartbeat prints only on threshold crossing — the first heartbeat whose `lastActivity` age exceeds the stall threshold, and the first one after activity resumes. One message per crossing, no repeats, no cooldown timers. Same edge rule for any future per-tick kind.
- `formatFailure` (`runplan.js:184-189`) stops being end-of-run-only. Its three lines (`task <id> blocked: <failureClass> (<cause>)`, `  log: <logPath>`, `  | <logTail>`) are emitted by the display policy at the moment the block is journaled. The end-of-run summary remains, as a recap.

Kinds that MUST print at the moment they occur: `quarantine` (with its `reason`), `verify.failed` (with tail), `retry.attempt`, `budget.exhausted`, `rate-limit.parked`, `gate.failed`, `fix.rung`, `task-end` with a non-success outcome. `attempt.heartbeat` prints on stall/resume edges only, per the rule above.

Fix in the same task: `run.js:654` wraps as `journal({ ...event, task, ...identity })` — the identity spread lands **last** and overwrites `quarantine`'s `phase:'quality'`. Identity MUST NOT clobber caller-supplied fields; spread identity first.

### 1.2 Task correlation — `v2/run.js`, `v2/dispatch.js`, `spec/journal-events.schema.json`

`verify.passed|failed|retry`, `reply.unmarked`, and `budget.exhausted` carry `task` but no `taskId`; `attempt.*` carry neither. `replay`'s `event.taskId &&` gate (`control-api.js:88`) and `eventsOf` (`:474`) drop all of them, so per-task views stay empty **even after 1.1 streams everything**.

Contract: every event that belongs to a task carries `taskId` (canonical) at the emit site. `task` is retained as a deprecated alias for legacy journals; consumers read `taskId ?? task`. `attempt.*` events additionally carry `taskId` and `attemptId`.

### 1.3 Transcript reachability — `v2/control-api.js`

`taskLogPath` (`:554-569`) matches `dispatch|gate`. `childLogPath` (`run.js:1074-1076`) writes phases `coder`, `fallback`, `gate`, `verify`, `verify-fixer`, `verify-retry`, `review`, `fixer`, `dependency-repair`, `stronger-fixer`. **The agent's own transcript is served by nothing.**

Contract: the phase set is a single exported constant in the engine, imported by both writer and reader. The route regex is derived from it, never re-typed. Adding a phase must not require editing a regex in a second file. Both writer and reader live in `v2/`, so the import stays inside the engine bundle — no consumer outside `modules/harness/v2/**` may import it; anything collector-side that needs the phase list receives it over the wire in the capabilities payload. `/tasks/:id/transcript`, `/tasks/:id/activity`, and `/runs/:id/tasks/:task/stream` serve any declared phase; `?phase=` selects, default = the most recent log for that task.

### 1.4 Attempt routing — `collector/src/server.ts`, `apps/web/src/pages/api/collector/[...path].ts`

Rich attempt routes **already exist** at `control-api.js:751-768` (`/runs/:id/attempts`, `/attempts/:id/prompt`, `/attempts/:id/reply`). They are unreachable from the browser: `grep -n attempts` returns zero in both files, and `[...path].ts:125` default-denies.

Contract: add read-only collector routes and matching allowlist regexes for exactly those three paths — no generic harness proxy, no token in any browser-visible response, upstream status/body preserved via the existing `HarnessApiError`. Mutations stay under `/actions/:verb`.

This one task mounts `AutopsyStrip`, `BurnPanel`, `AttemptTimeline`, and `AttemptDetailDrawer` with no UI changes at all.

---

## Wave 2 — honest fields (engine + control-api, file-disjoint from wave 1 UI work)

### 2.1 Timeline segments — `v2/control-api.js:293-328`

- `agentId`: declared in `harness.ts:260`, never produced by `timelineOf`. **This is why the live-feed page is unreachable** — `agentHrefForTask` (`PlanRunApp.tsx:292-304`) needs it, so the link never renders and a direct URL hits `AgentApp.tsx:217` "Known agents: none recorded". `AgentApp` + `useSseStream` are fully built and wired; emitting `agentId` turns them on. Identity = the seat binding that ran the segment (wrapper + model + account), stable per attempt.
- `t0` is a **relative** ms offset (`:302`: `starts.get(taskId) - planStart`) rendered as an absolute epoch at `SegmentDetailDrawer.tsx:26`, producing "1/1/1970". That is fabricated data, not a placeholder. Emit an absolute `startedAt` (the journal already carries `ts`) **alongside** the relative `t0`; the drawer renders `startedAt`, layout keeps `t0`. Never patch the UI to hide it.
- `journalSeq`: emit the journal sequence number so a segment is traceable to its exact record.
- `durationMs`: emitted on every attempt-terminal event. `PlanRunApp.tsx:167-172` already sums it and no event carries it, so `fixLoopShare` reads 0 on every run. The journal already holds attempt start and terminal `ts`, so the engine computes it at the terminal emit — the panel is not changed.
- `note` (**A7**): the segment's terminal cause when one exists — `failureClass` + `cause`, not prose. Absent when the segment succeeded; the UI keeps its honest-gap label rather than inventing text.

### 2.2 Resolver + decision observability — `v2/quality.js`, `v2/control-api.js:732`

`GET /runs/:id/decisions` is a hard stub returning `{decisions: []}`. Standing user requirement, verbatim: *"needs full observability about what the resolver decided and why"*.

Contract — every escalation rung emits a decision record: which rung fired, the seat binding it selected (wrapper + model + account) and **why that binding differed from the failed seat**, the inputs it saw (failure class, gate output tail, diff stat), its verdict, and the resulting action. Projected by the decisions route in the shape the 2026-07-18 design already pinned (`{ id, category?, needs?, why?, blast_radius?, options, task?, status }`), with legacy string options preserved.

### 2.3 Attempt identity in the event stream — `v2/control-api.js:475`

`attemptId: null` on every projected event. Now that 1.2 stamps it at the emit site, project it. Absent in a legacy journal → `null`, never inferred.

### 2.4 Capabilities are computed, not literal — `v2/control-api.js:20`

`CAPABILITIES` is a frozen literal (`events:true, stream:true, steer:false, decisions:false, control:false`). Derive each flag from what the run's journal actually supports, so a run predating this work still reports honestly and a UI gate flips on automatically when the producer catches up.

### 2.5 Rate limits (**A9**) — `v2/control-api.js:720`

`ratelimits: {}` is a stub while `rate-limit.parked` is a real emitted kind. Project observed parks per account: when parked, until when, how many times this run. Never synthesize a quota number the wrapper did not report.

---

## Wave 3 — SSSF contracts (new engine surface; disjoint from waves 1–2 sites)

### 3.1 Gate evidence — `v2/quality.js`, `lib/gates.sh` boundary

Today a gate returns a verdict. SSSF's gates return evidence: `GateCheck{ item: string, ok: boolean, note: string }`, aggregated as `GateReport{ checks, violations, passed }`.

Contract: gate0 emits a per-check list, not a boolean. **A green gate must say what it verified** — that is what makes "the gate passed" believable and what makes a red gate diagnosable. `gate_pass` / `gate_fail` carry `attempt`, `violations`, `checks`.

### 3.2 Verbatim failure tails

SSSF's `QualityCheckResult.output_tail` rides in the envelope **verbatim and unparsed**. Cap: 4000 chars (SSSF's `TAIL_CHARS`), tail-biased. Applies to gate failures, verify failures, and dispatch failures. Never regex-summarize a failure before showing it; a summarizer is exactly what loses the one line that explains it.

### 3.3 Process table — hung-agent detection

SSSF's rationale, adopted: *a hung agent emits nothing, which is exactly when you need its pid*. `v2/dispatch.js` already schedules `attempt.heartbeat` every 60s (`ATTEMPT_HEARTBEAT_MS`, `:12`) carrying `pid elapsedSecs timeoutSecs eventCount lastActivity`. It is journal-only.

Contract: heartbeat becomes a printed kind (wave 1.1 already routes it) and is projected as a live process record — `{ runId, taskId, attemptId, kind, pid, command, startedAt, endedAt? }` — reachable through the control-api. `AgentApp.tsx:304`'s hardcoded `IdleTimer idle={null}` (**A10**) is fed from `lastActivity`. Silence past a threshold renders as *stalled, N minutes since last activity, pid P* — never as "running".

This is what should feed `supervision.js`'s two dead projections, or they are deleted (see Architecture decisions).

### 3.4 Cost + usage (**A8**)

`attempt.usage` (`dispatch.js:192`) is emitted and never surfaced. Project per-run and per-task token/cost totals in SSSF's `UsageBreakdown` shape (input / output / cache-read / cache-write / reasoning, tokens and cost, mergeable). `PlanRunApp.tsx:592-599`'s hardcoded A8/A10 coverage-gap rows are deleted once 3.3 and 3.4 land — a hardcoded gap notice that outlives its gap is a lie.

### 3.5 One-call run settlement

SSSF's `Run.finish()` settles status, banner, and exit code together, with every phase defaulting to `fail`. Contract: a single terminal settlement in `runplan.js` derives the run's status, the banner, and the process exit code from one projection of the journal. A run that dies mid-flight settles as failed by default — success is asserted, never assumed.

---

## Wave 4 — UI defects (`apps/web` only; disjoint from every engine file above)

- `PlanRunApp.tsx:404` — `[]` is truthy, so both the decisions block and its own gap notice (`:438`) are skipped. Branch on length.
- `PlanRunApp.tsx:170` — `fixLoopShare` keys on kind `llm-fixer`, which the engine never emits (it emits `fix.rung`). Key on the real kind.
- `PlanRunApp.tsx:167-172` — sums `payload.durationMs`. The engine emits it per attempt in 2.1, so this panel needs no change; assert against a run that produced it. **Never render a zero as a measurement.**
- `PlanRunApp.tsx:586` — "Run control is not supported by this run" is unconditional because `CAPABILITIES.control` is a literal `false`. Resolved by 2.4; the string becomes conditional.
- `AgentApp.tsx:304` — `IdleTimer idle={null}` hardcoded; fed by 3.3.
- `SegmentDetailDrawer.tsx:26` — renders `startedAt` from 2.1 instead of `t0`.

Every one of these is currently either a fabricated value or a stale gap notice. Honest data is non-negotiable (`.claude/skills/od-ui-dev/SKILL.md`).

---

## Deliberately out of this spec

SSSF's visualizer is 4621 lines; `PhaseDetail.vue` alone is 1265. Its waterfall lanes, per-agent context-window bars, and compiled-prompt viewer are **worth having and are not what unblocks a failing run tonight**. They are a follow-on plan against the same producer surface, which waves 1–3 make complete enough to build on without further engine work. Nothing here is scoped down to avoid work — this ordering exists because wave 1 alone answers all four of the user's questions.

## Testing

- **Unit** — display policy: every declared journal kind maps to a decision or the explicit default; no kind can be silent. Schema: every emitted kind is declared (the validator already enforces this; the test asserts the inverse — no declared kind is unemitted).
- **Contract** — each control-api route against a legacy journal (fields absent → `null`, never fabricated) and an enriched one.
- **Integration** — a canary run (`presets/canary.json`, free, offline, deterministic) whose task is engineered to quarantine. **Acceptance: the terminal shows the failure class, the cause, and the log tail while the run is still going**, and `/plans` shows the same for that task. That single test is the whole point of this spec; if it does not pass, nothing here shipped.
- **Regression** — existing event kinds, routes, and payload fields unchanged; legacy plan fixtures parse and run.

Engine work runs under `HARNESS_ENGINE_DEV=1` **scoped to the subprocess** (`modules/harness/CLAUDE.md`); `run-tests.sh` unsets it deliberately. Landing is not deploying — a bundle bump (`bin/harness-release.sh`) is post-run main-thread work, never a plan task.

## Wave summary

| Wave | Scope | Files | Parallel? |
|---|---|---|---|
| 1a | one emitter (1.1) | `v2/run.js` `v2/bin/runplan.js` | alone — 1.2 rewrites the same emit sites |
| 1b | taskId (1.2), transcript regex (1.3), attempt routing (1.4) | `v2/run.js` `v2/dispatch.js` `spec/journal-events.schema.json` / `v2/control-api.js` / `collector/src/server.ts` `apps/web/src/pages/api/collector/[...path].ts` | ✅ three disjoint file sets |
| 2 | segments, decisions, attemptId, capabilities, ratelimits | `v2/control-api.js` `v2/quality.js` | serial on `control-api.js`, parallel with `quality.js` |
| 3 | gate evidence, tails, process table, usage, settlement | `v2/quality.js` `v2/dispatch.js` `v2/bin/runplan.js` `lib/gates.sh` | file-disjoint per task |
| 4 | UI defects | `apps/web/src/components/plans/*` `packages/deck-ui/src/SegmentDetailDrawer.tsx` | fully parallel |