# Harness v2 Phase 2 Supervision and Recovery Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use /ship (recommended) or /executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

audience: AI coding agents first.

**Goal:** Add per-run systemd supervision, durable automatic crash recovery, ownership-safe hung-run termination, and enforced worker resource limits to harness v2.

**Architecture:** Keep coordination in the existing v2 process. One systemd template unit owns one registered run; short timer-invoked scripts reconcile dead coordinators and inspect stale heartbeats from disk. An append-only journal plus atomic run registry is authoritative across process exit and reboot; no long-lived supervisor daemon is introduced.

**Tech Stack:** Node.js CommonJS, systemd user units/timers, append-only NDJSON journals, atomic JSON registry records, `node:test`.

---

## Wave Plan

| Wave | Tasks | Files touched | Safe to parallelize? |
|---|---|---|---|
| 1 | Task 1, Task 2, Task 3 | `modules/harness/v2/supervision.js`, `modules/harness/spec/events.schema.json`, `modules/harness/v2/test/supervision.test.js`; `modules/harness/v2/child.js`, `modules/harness/v2/test/child-resource-limits.test.js`; five distinct `modules/harness/systemd/*` unit files, `modules/harness/v2/test/systemd-phase2.test.js` | ✅ no overlap |
| 2 | Task 4, Task 5 | `modules/harness/v2/bin/reconcile-runs.js`, `modules/harness/v2/test/reconcile-runs.test.js`; `modules/harness/v2/bin/watchdog.js`, `modules/harness/v2/test/watchdog.test.js` | ✅ no overlap |
| 3 | Task 6 | `modules/harness/v2/run.js`, `modules/harness/v2/bin/runplan.js`, `modules/harness/v2/test/run-supervision.test.js` | single task |
| 4 | Task 7 | `modules/harness/v2/test/index.js` | single task |

## File Structure

- `modules/harness/v2/supervision.js` — atomic run registry, generation/ownership validation, heartbeat and restart projections.
- `modules/harness/spec/events.schema.json` — Phase 2 journal event contracts.
- `modules/harness/v2/child.js` — resource-limited worker scopes and foreground ownership fallback.
- `modules/harness/systemd/harness-run@.service` — one coordinator unit per registered run.
- `modules/harness/systemd/harness-reconcile.service` — one-shot reconciliation invocation.
- `modules/harness/systemd/harness-reconcile.timer` — periodic reconciliation trigger.
- `modules/harness/systemd/harness-watchdog.service` — one-shot watchdog invocation.
- `modules/harness/systemd/harness-watchdog.timer` — periodic watchdog trigger.
- `modules/harness/v2/bin/reconcile-runs.js` — idempotent dead-coordinator recovery.
- `modules/harness/v2/bin/watchdog.js` — idempotent stale-heartbeat ownership check and termination.
- `modules/harness/v2/run.js` — supervision admission, durable phase resume, heartbeats, and resource evidence.
- `modules/harness/v2/bin/runplan.js` — registry-token systemd entry and foreground/unattended mode propagation.

### Task 1: Durable Supervision Registry and Journal Projection

**Wave:** 1

**Blocks:** Task 4, Task 5, Task 6

**Blocked by:** —

**Files:**
- Create: `modules/harness/v2/supervision.js` — registry and durable supervision state.
- Modify: `modules/harness/spec/events.schema.json` — Phase 2 event shapes.
- Create: `modules/harness/v2/test/supervision.test.js` — registry, identity, projection, and restart-ledger tests.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- Registry root: `${HARNESS_HOME:-$HOME/.harness}/state/v2/runs/`; one atomic JSON record per opaque run token.
- Registry record required fields: `token`, `repoRoot`, `planPath`, `slug`, `runId`, `generation`, `mode`, `unit`, `coordinator`, `status`, and `updatedAt`.
- `coordinator` and attempt ownership receipts use `{ pid, startTime, processGroup, cgroup, generation }`; missing or mismatched identity is never treated as ownership.
- Export idempotent operations to register/read/list/update a run, record/validate ownership, project the latest heartbeat, and project restart history from `readJournal(...)`.
- Journal kinds: `supervision.registered`, `supervision.heartbeat`, `supervision.crashed`, `supervision.restart-scheduled`, `supervision.restart-refused`, `attempt.ownership`, `watchdog.incident`, and `resource.limit`.
- Restart projection preserves the durable `30s`, `2m`, `8m` schedule and refuses a fourth restart for the same run.
- Atomic writes use same-directory temporary files plus rename; malformed, incomplete, unknown-generation, or identity-mismatched state fails closed.

**Behavior:**
- Process exit or reboot cannot erase registered run identity, restart count, next eligible restart time, or current generation.
- Replaying the same journal and registry yields identical supervision state.
- Registry operations never infer ownership from PID liveness alone.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/supervision.test.js`
- [ ] Event-schema validation accepts every Phase 2 event and rejects incomplete ownership/restart records.
- [ ] Restart projection survives module reload and returns delays `30000`, `120000`, `480000`, then refusal.

### Task 2: Worker Resource Scopes and Ownership Receipts

**Wave:** 1

**Blocks:** Task 6

**Blocked by:** —

**Files:**
- Modify: `modules/harness/v2/child.js` — resource properties, scope admission, ownership callbacks, and explicit foreground fallback.
- Create: `modules/harness/v2/test/child-resource-limits.test.js` — descendant containment and each limit class.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- Extend `runChild(options)` with validated `resourceLimits`, `mode`, `allowSetsidFallback`, and `onOwnership` fields.
- `resourceLimits` maps memory, CPU, process, and file-size ceilings to `MemoryMax=`, `CPUQuota=`, `TasksMax=`, and `LimitFSIZE=` on the transient user scope.
- A systemd scope is admitted only after its unit identity and cgroup are observable; call `onOwnership({ pid, startTime, processGroup, cgroup, generation, unit })` before useful worker execution can proceed.
- `mode: "unattended"` rejects scope creation or identity-read failure. `mode: "foreground"` may use `setsid` only when `allowSetsidFallback === true`, and must still emit a process-group ownership receipt.
- Return resource termination as infrastructure evidence naming the tripped limit; never report task success.
- Retry resolution waits until the owned scope/process group has no surviving descendant.

**Behavior:**
- Every worker and descendant remains inside one owned scope or one explicitly permitted foreground process group.
- Memory, CPU, task-count, and file-size violations are distinguishable in returned evidence.
- Unknown ownership never triggers a signal.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/child-resource-limits.test.js`
- [ ] Integration cases exceed each supported limit and assert named `resource.limit` evidence.
- [ ] Teardown proof shows no descendant before retry is allowed.

### Task 3: Per-Run and Timer systemd Units

**Wave:** 1

**Blocks:** Task 4, Task 5, Task 6

**Blocked by:** —

**Files:**
- Create: `modules/harness/systemd/harness-run@.service` — one registered run per unit.
- Create: `modules/harness/systemd/harness-reconcile.service` — short one-shot reconcile script.
- Create: `modules/harness/systemd/harness-reconcile.timer` — durable periodic reconciliation.
- Create: `modules/harness/systemd/harness-watchdog.service` — short one-shot watchdog script.
- Create: `modules/harness/systemd/harness-watchdog.timer` — durable periodic watchdog.
- Create: `modules/harness/v2/test/systemd-phase2.test.js` — unit syntax and lifecycle contract tests.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- `harness-run@.service` accepts only an opaque registry token as `%i`; scripts resolve all paths and argv from the registry.
- Run unit invokes released `runplan` in supervised foreground mode; `Restart=no` leaves restart authority exclusively to the durable reconcile ledger.
- Reconcile and watchdog services use `Type=oneshot`, invoke only their named short idempotent scripts, and exit.
- Timers use `Persistent=true`; overlapping invocations are harmless because scripts take a bounded single-writer lock and re-read disk state after locking.
- Units do not embed mutable plan paths, shell-evaluated command strings, or a monolithic daemon.

**Behavior:**
- One failed run unit cannot stop unrelated run units or either timer.
- Reboot re-arms timers without inventing a restart receipt.
- Re-running either one-shot service against unchanged state is a no-op.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/systemd-phase2.test.js`
- [ ] `systemd-analyze --user verify modules/harness/systemd/harness-run@.service modules/harness/systemd/harness-reconcile.service modules/harness/systemd/harness-reconcile.timer modules/harness/systemd/harness-watchdog.service modules/harness/systemd/harness-watchdog.timer`

### Task 4: Automatic Crash Reconciliation

**Wave:** 2

**Blocks:** Task 6

**Blocked by:** Task 1, Task 3

**Files:**
- Create: `modules/harness/v2/bin/reconcile-runs.js` — registry/journal reconciliation and bounded restart.
- Create: `modules/harness/v2/test/reconcile-runs.test.js` — adoption, crash, reboot, backoff, and isolation tests.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- `reconcile-runs.js` is a short idempotent process: lock → list registry → reconcile each record with journal and systemd → write receipts/state → exit.
- Adopt an already-live coordinator only when PID start time, process group, cgroup, unit, and generation all match the registry.
- Mark a dead matching coordinator `supervision.crashed`; schedule generations with durable `30s`, `2m`, `8m` backoff; start `harness-run@<token>.service` only after the recorded deadline.
- Refuse the fourth restart with `supervision.restart-refused`.
- Unknown ownership, incomplete identity, future generation, malformed journal, or ambiguous unit state fails closed for that run and does not block unrelated runs.

**Behavior:**
- A rerun before the durable deadline creates no duplicate receipt or unit start.
- Restart state survives reconcile process exit and host reboot.
- Recovery resumes the existing run id and durable phase; it never creates a replacement run.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/reconcile-runs.test.js`
- [ ] Integration test kills coordinators across three generations, reconstructs timer state, observes `30s/2m/8m`, refuses generation four, and keeps an unrelated run progressing.

### Task 5: Ownership-Safe Hung-Run Watchdog

**Wave:** 2

**Blocks:** Task 6

**Blocked by:** Task 1, Task 3

**Files:**
- Create: `modules/harness/v2/bin/watchdog.js` — stale-heartbeat detection, identity proof, full-tree termination, and recovery handoff.
- Create: `modules/harness/v2/test/watchdog.test.js` — descendant kill and PID-reuse protection.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- `watchdog.js` is a short idempotent process: lock → list active registry records → project heartbeat and attempt ownership → verify live identity → act → journal → exit.
- Staleness uses persisted heartbeat time, never PID liveness alone.
- Before signaling, match PID start time, process group, cgroup, generation, and owned unit/scope against the latest receipts.
- Terminate the complete owned systemd scope/process group, wait for descendant teardown, append `watchdog.incident` evidence, mark the coordinator crashed, then leave restart scheduling to Task 4.
- Identity mismatch or incomplete proof appends a fail-closed incident and sends no signal.

**Behavior:**
- Re-running after termination produces no duplicate signal or recovery schedule.
- A reused PID or generation mismatch survives untouched.
- Watchdog never restarts a run directly.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/watchdog.test.js`
- [ ] Integration test hangs a worker with a descendant: entire owned tree exits and reconcile resumes the same run.
- [ ] Reused/mismatched PID case remains alive and records one fail-closed incident.

### Task 6: Integrate Supervision into Launcher and Run Lifecycle

**Wave:** 3

**Blocks:** Task 7

**Blocked by:** Task 1, Task 2, Task 3, Task 4, Task 5

**Files:**
- Modify: `modules/harness/v2/run.js` — registration, generation resume, heartbeats, ownership, phase checkpoints, and resource evidence.
- Modify: `modules/harness/v2/bin/runplan.js` — registry-token systemd entry and mode propagation.
- Create: `modules/harness/v2/test/run-supervision.test.js` — end-to-end crash, hang, resume, and resource-limit lifecycle tests.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- Unattended launch flow: resolve plan/options → register token/generation → start `harness-run@<token>.service` → return; the unit re-enters through the registry token and runs one foreground coordinator.
- Foreground launch remains foreground; `setsid` worker fallback requires explicit support and is never enabled for unattended runs.
- Coordinator heartbeats journal the current durable phase and generation. Attempt ownership is journaled before dispatch/fallback/fix/review work can mutate state.
- Recovery reuses `slug`, `runId`, integration branch, retry ledger, and completed checkpoints; it resumes the first incomplete durable phase without repeating a completed phase.
- Every `runChild` call receives validated resource limits and mode. Ownership and `resource.limit` evidence is appended immediately.
- Existing Phase 1 retry stop-loss remains authoritative across coordinator generations.
- v2 run completion may update only its integration branch; it must not push `main`.

**Behavior:**
- Crash during coder, review, fixer, or gate resumes that durable phase with no duplicate completed checkpoint.
- Hung attempt termination flows watchdog → crashed receipt → reconcile backoff → same-run resume.
- Resource-limit termination is infrastructure failure evidence and enters existing bounded retry logic.

**Acceptance:**
- [ ] `node --test modules/harness/v2/test/run-supervision.test.js`
- [ ] Crash/restart tests cover coder, review, fixer, and gate phases with the same run id and increasing generation.
- [ ] End-to-end test proves three recovery attempts, fourth refusal, unrelated-run isolation, heartbeat staleness handling, and Phase 1 stop-loss preservation.

### Task 7: Register and Run Full Phase 2 Regression Suite

**Wave:** 4

**Blocks:** —

**Blocked by:** Task 1, Task 2, Task 3, Task 4, Task 5, Task 6

**Files:**
- Modify: `modules/harness/v2/test/index.js` — register all new Phase 2 test files.

**Contract (pin EXACTLY — this is the divergence-prone surface):**
- Register `supervision.test.js`, `child-resource-limits.test.js`, `systemd-phase2.test.js`, `reconcile-runs.test.js`, `watchdog.test.js`, and `run-supervision.test.js` exactly once.
- Preserve every existing test registration.
- Run Phase 2 focused tests before the full v2 regression suite.

**Behavior:**
- Full suite verifies Phase 1 trust-core behavior remains unchanged while Phase 2 supervision is active.
- Syntax, unit-file verification, schema validation, and all v2 regressions complete without warnings or skipped mandatory integration coverage.

**Acceptance:**
- [ ] `node --check modules/harness/v2/supervision.js`
- [ ] `node --check modules/harness/v2/bin/reconcile-runs.js`
- [ ] `node --check modules/harness/v2/bin/watchdog.js`
- [ ] `node --check modules/harness/v2/child.js`
- [ ] `node --check modules/harness/v2/run.js`
- [ ] `node --check modules/harness/v2/bin/runplan.js`
- [ ] `node --test modules/harness/v2/test/supervision.test.js modules/harness/v2/test/child-resource-limits.test.js modules/harness/v2/test/systemd-phase2.test.js modules/harness/v2/test/reconcile-runs.test.js modules/harness/v2/test/watchdog.test.js modules/harness/v2/test/run-supervision.test.js`
- [ ] `node modules/harness/v2/test/index.js`
- [ ] `node modules/harness/presets/_validate.mjs`