# Harness Reliability: v1 Lessons and Recovery Plan

audience: AI coding agents first. This file is canonical for harness crash, restart, retry, checkpoint, and caller-completion work. Read it before changing those paths.

**Owner requirements:** Resolve bounded, classified failures first. When completion is genuinely impossible, write terminal evidence, terminate nonzero, and deliver that exit to the launching agent. NEVER add v1-style daemon/supervisor/control-plane complexity.

## 1. Decision

Use one blocking foreground `runplan` invocation. Keep repair inside that process. Persist verified task checkpoints. On coordinator failure, return nonzero; next explicit invocation resumes the same incomplete logical run from valid checkpoints. NEVER automatically relaunch a coordinator. NEVER replay completed tasks. NEVER make UI, registry, timer, daemon, or process liveness execution authority.

This supersedes automatic coordinator restart/reconcile assumptions in `docs/specs/2026-07-29-harness-v2-parity-design.md` §§4, 5.7, and 5.8. It does not prohibit worker process isolation or resource limits.

## 2. Evidence coverage

Evidence inspected 2026-07-30:

- 1,439 unique pre-cutover transcript candidates parsed structurally from `/home/user/.claude/projects/**`; 1,439 account-home aliases deduplicated by content. Transcript audit: `/home/user/.systray-ai/logs/cdx/20260730-175439-1129386.log`.
- 468 handoffs inventoried; 46 dated v1 harness handoffs parsed. Handoff audit: `/home/user/.systray-ai/logs/cdx/20260730-174353-881903.log`.
- Reachable history inventoried: 655 + 4,284 + 7,730 + 42 archived harness commits and 5,001 Overdeck commits. Relevant source, service, test, plan, report, and correction chains inspected. Archive audit: `/home/user/.systray-ai/logs/cdx/20260730-180744-1370010.log`.
- Current v2 bundle `0.1.102`, service units, journal/control API, and exact `golive-completion` incident inspected.

Confidence labels:

- **PROVEN:** direct transcript/tool evidence or source/diff establishes behavior.
- **SUPPORTED:** handoff plus correction commit/source agrees; raw incident artifact is unavailable.
- **UNRESOLVED:** reported symptom or proposed correction lacks terminal verification.

This catalog is complete to accessible narrowed transcript and handoff evidence. It is not a claim that every line of 17,712 historical commits was manually reviewed. Unknown historical details MUST NOT be promoted to fact.

## 3. Architecture chronology

1. Foreground shell `runplan`: journal, wrappers, worktrees, caller-owned lifetime.
2. Stateful Node runner: scheduler, gates, journal, locks, task control.
3. Crash non-recovery prompted babysitter/watchdog work, then long-lived `harnessd` under systemd.
4. `harnessd` accumulated tick scheduling, notification/watchdog semantics, transient scopes, PID/lock/run pointers, restart/reconcile, reaping, and cleanup.
5. Control API, SSE, TUI/web timelines, retries, quarantine, repair ledgers, queues, and delivery receipts added more independently-derived state.
6. Each remediation exposed another ownership, attribution, environment, projection, or lifecycle seam.
7. V1 was deleted on owner order 2026-07-29. V2 initially rejected a daemon but later reintroduced per-run systemd ownership plus watchdog/reconcile timers.
8. Current v2 incident proves those timers relaunch fresh execution instead of resuming scheduler checkpoints.

## 4. Failure catalog

| ID | Status | Observed failure | Violated invariant | Evidence |
|---|---|---|---|---|
| F01 | PROVEN | Coordinator crashed mid-run and did not recover. | Every invocation reaches terminal success/failure. | Transcript `318f80a1-b74f-4619-ada6-d77da2210e09.jsonl`, `2026-07-11T16:19:22.832Z`, UUID `58889dfd-4aba-4dae-9c60-121bbc9e87cc`: “harness crashing mid run and does not recover by itself.” |
| F02 | SUPPORTED | Unhandled spawn error crash-looped `harnessd`; systemd lacked burst limiting. | Child launch failure is data, not daemon death. | Commit `871df6f`; handoff `2026-07-12-318f80a1.md:35`. |
| F03 | SUPPORTED | Failed spawn emitted `error` + `close`, never `exit`; stale child remained in `activeChildren`, hanging drain/shutdown. | Parent settlement follows an event guaranteed for every child path. | Commit `7a515a2`; handoff `2026-07-13-57aceef4.md:71-75`. |
| F04 | PROVEN | Live service resurrected fixture runs and contaminated tests. | Tests and stale records cannot become executable service input. | Transcript above, `2026-07-12T23:51:42.281Z`, UUID `5dbf77a1-cbf5-4c41-9036-ae8fa90ed98c`; hermetic-HOME correction. |
| F05 | SUPPORTED | Daemon children lacked caller PATH/toolchains/dependency tools/Codex resolution. | Execution environment is explicit and identical at launch boundaries. | Commits `afc5622`, `9205cc2`, `3fe5658`; handoff `2026-07-13-318f80a1.md:45-47`. |
| F06 | SUPPORTED | systemd ignored readiness notification emitted by a Perl child. | Readiness protocol matches actual notifier identity. | Commit `75ecf05`; handoff `worktrees-obs-viz/2026-07-17-090c8793.md:29`. |
| F07 | SUPPORTED | Worktree GC inside daemon tick exceeded watchdog interval; async correction then required overlap control. | Liveness heartbeat and maintenance completion are separate; periodic work is bounded and non-reentrant. | Commits `c207fc2`, `24fe532`, `8df10ef`; handoff `2026-07-17-090c8793.md:55`. |
| F08 | SUPPORTED | Daemons started in deleted worktrees escaped cwd-based cleanup; four orphans accumulated. | Ownership identity survives cwd/worktree deletion. | Handoff `worktrees-hardening-stage1/2026-07-16-090c8793.md:56-60`. |
| F09 | SUPPORTED | Corrupt journal error was logged inside watched tree, retriggering watcher about 5 Hz; one process reportedly consumed 10h50m CPU. | Observation never mutates watched input or creates feedback. | Same handoff; commit `883d95e`. |
| F10 | SUPPORTED | Missing journal path created `undefined.db`. | Persistence paths validate before opening storage. | Commit `03ba9b1`; `src/daemon.js:1461-1462`; handoff `2026-07-17-4dd8b6de.md:55`. |
| F11 | PROVEN | Concurrent journal/index mutation could lose or duplicate state; downgrade mutation erased history. | Journal is append-only and single-writer/serialized. | Commits `6cd65fe8`, `0d0ab080`, `4158c785`; archived `src/state/journal.js:22-27,149-176`. |
| F12 | SUPPORTED | Restart advanced state without a receipt, leaving stale pointers and ghost journals. | Restart is not accepted without durable readiness/identity receipt. | Commits `fa8afbfb`, `0c45a610`; handoff `worktrees-audit-remainder/2026-07-26-fe9a8985.md:23-31`. |
| F13 | UNRESOLVED | Reconnect snapshot suppression could lose the terminal event. | Terminal truth is replayable exactly once after reconnect. | Handoff `2026-07-26-fe9a8985.md:93`; related commits `f9b5a75`, `22a3b1a`, `7f8551a`. |
| F14 | SUPPORTED | Timeline fetch races and task segments crossed run boundaries. | Projection is run-scoped, monotonic, and never authority. | Commits `2a93fa9`, `6a09002`; handoff `worktrees-obs-viz-mega-plan-harness-web/2026-07-16-090c8793.md:27`. |
| F15 | SUPPORTED | `freshRetry` used stale task base, so Gate0 inspected sibling-task diffs. | Retry uses the exact task/dependency checkpoint it owns. | Commit `828298d`; handoff `2026-07-09-66a88ab8.md:23`. |
| F16 | SUPPORTED | Fixer prompt lacked failure evidence; reviewer timed out; merge-conflict error was swallowed. | Every failure remains attributable through review, repair, merge, and gate. | Commit `6d5812e`; same handoff `:27`. |
| F17 | SUPPORTED | Task worktree changes were not persisted; resumed dependency recovery and task-worktree dependency repair required later fixes. | Accepted progress has a durable commit/checkpoint before task success. | Commits `8760389`, `4d4b50f`, `fd53f6b`. |
| F18 | SUPPORTED | Untracked spec was invisible to task worktrees. | Plan inputs are immutable, copied, and content-addressed before dispatch. | Commit `14fd168`; handoff `2026-07-13-57aceef4.md:39`. |
| F19 | PROVEN | Generic timeout retry could misclassify infrastructure timeout as task defect; isolated retry existed only for Vitest. | Timeout is non-completion with preserved cause, not code failure. | Transcript `ebc3c467-c322-415d-8762-32de9f0e665b.jsonl`, `2026-07-06T14:53:38.683Z`, UUID `ac1ce666-e4d1-48a6-a5fc-88745fddb607`; commit `594e260`. |
| F20 | UNRESOLVED | `dep-provision-failed` repair reportedly succeeded 6/852 attempts and quarantined 848. | Retry must change the classified cause or stop; unchanged replay is token waste. | Handoff `worktrees-p1-supervision/2026-07-26-fe9a8985.md:73-81`. |
| F21 | PROVEN | Completion path had a known zero-diff false-green class. | Success requires durable task output and expected state transition. | Transcript `2026-07-19T16:36:17.423Z`, UUID `65636a78-024a-425d-a60a-87d0e7d2c315`. |
| F22 | PROVEN | Successful mutation could be reported failed when cleanup failed; concurrent promotion rewrites could lose/duplicate index entries. | Primary outcome and cleanup outcome are recorded separately; promotion serializes. | Transcript subagent `wf_6861794f-a63/agent-a2c3b985fc606c6e1.jsonl`, `2026-06-30T13:45:02.412Z`. |
| F23 | SUPPORTED | Dead task worktrees accumulated 8.3 GB; 248 dirty trees made cleanup unsafe and ambiguous. | Cleanup never owns progress; committed checkpoint precedes safe reclamation. | Handoffs `worktrees-ca-reap/2026-07-26-fe9a8985.md:53-54`, `2026-07-17-9642626f.md:79-82`. |
| F24 | PROVEN | `runplan --tui` was declared done but crashed immediately on `$1` under `set -u`; no real invocation had verified it. | User-visible completion requires end-to-end invocation evidence. | User report in handoff `2026-07-13-318f80a1.md:63-67`; commit `cf5fc10`. |
| F25 | PROVEN | Current v2 worker budget calls `killScope` but promise settles only on `systemd-run` `close`; failed teardown can leave coordinator pending forever. | Deadline independently settles parent result after bounded teardown. | Bundle `0.1.102/v2/child.js:53-100`; especially timer `:53` versus sole success settlement `:77-100`. |
| F26 | PROVEN | Current foreground command was manually backgrounded; launching agent depended on `TaskOutput` polling and could miss terminal failure. | Caller stays attached to one blocking command and receives its exit code. | Current session transcript line 8, UUID `afdfa7be-edc0-4b85-8d44-00056002721b`; exact run `run-3456769-1785373133722`. |
| F27 | PROVEN | Current reconcile/systemd generations produced separate executions at 6/13, 5/13, 2/13, and 1/13; each restarted at t1. | Recovery resumes validated checkpoints; it never creates replay disguised as recovery. | User screenshot report, current transcript line 645, `2026-07-30T11:01:54.785Z`, UUID `0c4b2f0e-2aa5-4445-b858-d2cf79580e78`; bundle `supervision.js:7`, `bin/reconcile-runs.js:187-197`. |
| F28 | PROVEN | Repeated `plan-start` events under one run ID merged generations, while UI also showed separate records. | One logical run has one unambiguous state; attempts/generations are subordinate evidence only. | Exact run journal sequences 1, 123, 208, 221 for `run-3456769-1785373133722`; control projection investigation. |

## 5. Remediation traps: NEVER repeat

1. Crash → add daemon → add service lifecycle → add watchdog → add reconcile → add receipt fencing → add stale-pointer cleanup. This multiplied owners without proving checkpoint resume.
2. Timeout → kill child → wait forever for child close. Teardown request is not settlement.
3. Failure → retry/quarantine. Unchanged cause replay burns tokens; quarantine alone does not complete work.
4. Journal error → log into watched tree → observer spin. Observation must be side-effect-free relative to source.
5. UI staleness → more snapshots/polling/streams → cross-run and reconnect races. UI must project one durable truth.
6. Worktree buildup → daemon reaper/GC tick → watchdog starvation and unsafe dirty-tree decisions. Progress must be committed before cleanup becomes mechanical.
7. Restart → new generation/run → replay from t1. A restart without checkpoint validation is a new failure, not recovery.
8. “Done” from unit tests → untested real CLI crash. Acceptance must exercise the user path.

## 6. Non-negotiable invariants

### Run ownership

- ONE foreground `runplan` process owns orchestration.
- Launching agent MUST invoke it as one blocking command. NEVER `run_in_background`, daemon mode, `&`, `nohup`, or `TaskOutput` polling.
- systemd MAY isolate worker descendants. systemd MUST NOT own, restart, reconcile, or declare coordinator completion.
- No daemon, timer, control API, UI, registry, or watcher may launch/relaunch a run.

### Terminal behavior

- Every worker attempt has a deadline and bounded teardown grace.
- Deadline MUST settle the coordinator-side promise even when child/scope close never arrives.
- Every controlled run termination writes exactly one durable `complete` or `failed` receipt before process exit.
- Abrupt coordinator death leaves an incomplete run; the attached shell still returns nonzero, and the next explicit invocation records the crash before validating resume.
- Exit `0` only when all required tasks, integration, landing policy, and finalization receipts pass.
- Any exhausted repair, invalid state, teardown uncertainty, or finalization error exits nonzero with task, phase, failure class, and evidence paths on stderr.

### Progress and resume

- Scheduler checkpoint key: `{logicalRunId, planDigest, taskId, taskContractDigest, dependencyCheckpointIds, baseCommit, resultCommit}`.
- Task checkpoint becomes valid only after task commit is integrated and required review/gate evidence is durable.
- Resume reconstructs state from append-only journal plus Git objects. Process/registry/UI state is never accepted as progress.
- Valid checkpoints are skipped. Invalid/ambiguous checkpoints fail closed before dispatch.
- Same `runplan <slug>` invocation resumes the sole compatible incomplete logical run. It starts a new run only after prior terminal completion or explicit `--new-run`.
- Multiple compatible incomplete runs are an error requiring explicit run selection; NEVER guess.

### Repair

- Classify before repair: task/code, gate, dependency/environment, provider/rate-limit, wrapper/engine, teardown, invariant corruption.
- Every retry rung MUST change a relevant input: binding, environment receipt, task head, failure fingerprint, or repair action.
- Same fingerprint + same heads + same binding/environment MUST NOT redispatch.
- Each class has a fixed attempt cap. Exhaustion writes terminal failure and exits nonzero.
- `blocked`/`quarantined` are task evidence, not successful plan terminal states.

### Projection

- Journal + Git checkpoint receipts are sole truth.
- UI/API show one logical run and subordinate attempt history. NEVER merge unrelated runs or render attempts as separate progress-owning runs.
- Observer failure cannot alter run outcome or launch work.

## 7. Implementation plan

Do not implement from archived v1 code. Re-implement smallest v2-native seams.

### Phase 1 — Stop replay and reconnect caller

1. Make `--foreground` the only execution mode used by `run-plan`; reject daemon/background launch in wrapper and skill gate.
2. Disable coordinator relaunch in reconcile/watchdog paths. Timers may report stale evidence only until removed; they MUST NOT invoke `runplan`.
3. Remove UI action/wording that presents a fresh launch as retry/resume.
4. Add integration test: kill coordinator after two completed tasks; assert no process relaunches, caller exits nonzero, and only one logical-run record exists.

**Acceptance:** no new `plan-start` after coordinator death; shell receives nonzero; existing journal, Git, and worktree evidence remains untouched.

### Phase 2 — Guarantee worker settlement

1. Refactor `runChild` around one idempotent settle function shared by spawn error, admission failure, normal close, budget expiry, and teardown expiry.
2. On budget: request scope kill; wait fixed grace; verify ownership/cgroup; escalate once; settle `wrapper-timeout`/`teardown-failed` regardless of `close` delivery.
3. Keep late child events harmless after settlement. Persist teardown evidence before returning.
4. Preserve exact stderr/stdout evidence and classify infrastructure separately from task failure.

**Acceptance:** fixture whose `systemd-run` child never emits `close` settles within `budget + grace`, returns nonzero, and leaves no owned descendant. Duplicate late events do not double-settle.

### Phase 3 — Durable task checkpoints

1. Define checkpoint schema from §6; append atomically after integration + required gate/review.
2. Copy plan input into run state and hash task contracts before first dispatch.
3. Reconstruct scheduler state from journal + Git on startup; validate every checkpoint against plan/dependencies/base/result commits.
4. Skip valid completed tasks. Preserve failed task worktree/commit evidence without calling it completed.
5. Fail closed on journal corruption, missing commit, changed contract, dependency mismatch, or ambiguous incomplete run.

**Acceptance:** deterministic mock-agent fixture completes t1–t6, terminates coordinator, invokes the same slug, then observes first dispatched task is next eligible after t6; t1–t6 dispatch counts remain exactly one. No provider account or model quota is used.

### Phase 4 — Bounded classified resolution

1. Centralize failure classification and one transition table. Remove independent retry loops from dispatch, quality, dependency repair, and reconcile paths.
2. Require a changed causal input for every retry.
3. Persist attempt fingerprint and cap before dispatch so coordinator failure cannot reset allowance.
4. Convert exhausted `blocked`/`quarantined` states into plan-level terminal failure.

**Acceptance:** unchanged failure never redispatches; repair success continues; exhaustion emits one failed receipt and exits nonzero.

### Phase 5 — Terminal receipt and projection

1. Write terminal receipt once, fsync/close journal, then print concise stderr summary and exit.
2. Derive CLI/UI/API status only from checkpoint/terminal receipts.
3. Represent attempt history beneath one logical run. Remove generation aggregation as progress semantics.
4. Make observer faults visible but outcome-neutral.

**Acceptance:** success and every failure class deliver exact exit code to a blocking caller; reconnect/UI refresh cannot lose or duplicate terminal state.

### Phase 6 — End-to-end proof before arming

Run every destructive/resume case with deterministic local mock-agent wrappers in isolated fixture repositories. Tests MUST NOT invoke Claude, Codex, Cursor, Grok, Pi, or any provider-backed agent; MUST NOT consume account/model quota:

- mock worker timeout with stuck parent and descendant;
- mock coordinator SIGKILL after each phase boundary;
- corrupted/truncated journal;
- missing result commit;
- changed plan/task digest;
- repeated identical mock gate/provider/environment failure;
- observer/control API unavailable;
- two concurrent same-slug launches;
- mock 13-task fixture interrupted after task 6 and resumed.

Arm only after the mock 13-task fixture proves tasks 1–6 each executed once, task 7 continues, final shell exit is delivered, and no reconcile/service creates another execution. The already-running real plan is not a test target and MUST NOT be interrupted, relaunched, or consumed for acceptance evidence.

## 8. Scope restraint

This plan intentionally does NOT add:

- automatic coordinator restart;
- long-lived daemon;
- new watchdog/reconcile owner;
- queue redesign;
- distributed lease service;
- event-stream protocol rewrite beyond truthful projection;
- worktree GC changes unrelated to checkpoint safety.

If foreground crash + explicit checkpoint resume later proves operationally insufficient, measure that gap first. Do not preemptively restore v1 supervision.

## 9. Field receipts — 2026-08-01 truth campaign (engine 0.1.105) + delivery-controller placement

Moved here from `0 DOCS/GIT_FATIGUE.md` §20 (that file owns the user-shielding contract, never harness internals). Receipts from the observability restoration, landed + deployed 2026-08-01:

- **Liveness evidence, never inference.** Dead runs showed "running" for days. Fix: engine stamps pid + `/proc` startTime at plan start; non-terminal run + dead coordinator → `degraded` with concrete resume instruction. Pid alone proves nothing (pid reuse); identity = pid AND startTime. Durable outcome = terminal receipt; process evidence only classifies the absence of one (invariants §6 confirmed in production).
- **One row per plan.** 11 relaunches rendered as 11 "running" rows. Fix: group by (repo, slug), newest attempt primary, `×N` attempts badge in web UI. Relaunch count is a visible cost signal.
- **No blind retry enforced.** `babysit-runplan` skill branches every failure on `failureClass`; ONLY `wrapper-engine-down` is retriable (max 2 consecutive, then advisor). `stop-loss`/`claim-exceeded` are NEVER relaunched around.
- **Tool-boundary gate + its own tests.** `runplan-blocking-gate` (PreToolUse) denies background runplan, status probing, live-record reads; forensic reads unlock only on coordinator death (same pid+startTime evidence as engine). Writing its tests exposed two DEAD deny paths in the shipped hook (`$`-anchored regex vs joined fields; dir path without trailing slash). **Rule (third confirmation): gate/resilience machinery MUST carry a test proving the guarded path fires.**
- **Self-heal happy path proven live.** Post-deploy, collector detected control-api version drift and replaced the stale service with 0.1.105 unattended.
- **Release hygiene.** `harness-release.sh bump` cuts `~/.harness/engine/versions/<v>/` AND writes `modules/harness/VERSION` into its checkout — run from a landable checkout, NEVER the pristine deploy clone (deploy fails closed on dirty clone; caught live). `CURRENT` pointer flip is the release; build-once/deploy-exact held.

**Delivery-controller placement (DECIDED 2026-08-01, referenced by GIT_FATIGUE §6.1): no new daemon.** GIT_FATIGUE's delivery controller, when built, lands as ledger-driven steps inside the foreground coordinator lifecycle; collector ensure-loop owns crash detection + self-heal; durable state = ledger + terminal receipts. Publication compare-and-swap needs a lock + ledger, not a resident process. Consistent with §1 Decision and §8 scope restraint. MUST NOT resurrect a harnessd-style daemon; a resident process may be reconsidered ONLY if a controller acceptance test is impossible without one — record the failing test here first.