# Autonomy hardening — plan

Audience: AI coding agents first.

Goal: run a plan and finish it in ONE run — no interruption, no unbounded idle, self-repair of harness-induced faults.

Status anchor: engine `0.1.54` live (`~/.harness/engine/CURRENT`). Wrapper teardown + shared verdict extraction landed `da306923`.

## Ordering (do not reorder)

1. **P1 supervision** — fenced intent→receipt + hierarchical supervisor. Converts existing correct diagnoses into actual acts.
2. **P2 attribution + conformance** — infra faults MUST NOT charge tasks; one vertical conformance test per seam.
3. **P3 blocked-retryable** — split re-entrant state from terminal quarantine; fingerprint-gated attempts.
4. **P4 readiness verification** — pre-gate deterministic checkup + ownership-safe repair.
5. **P5 idle telemetry** — progress vector as evidence only. MUST NOT kill on local silence alone.

Rationale (secondary): every root cause of the last 4 runs was a harness defect, not task code. Supervision and conformance remove that class; readiness and idle do not.

## P1 — supervision

Rule: **NEVER write an actuator event before the act succeeds.**

Protocol per actuation (restart, kill, re-dispatch, requeue):

1. write-ahead `<act>.requested{intent_id, target, reason}`
2. exclusive claim with monotonic **fencing generation**; loser MUST abort, never act
3. spawn
4. child readiness handshake carrying `{run_id, generation, engine_sha}`
5. `<act>.performed{intent_id, generation, pid, started_at}`

- `requested` with no `performed` inside T = detectable open intent; resolver MUST retry or escalate it.
- pid-alive alone is NOT a receipt. Handshake is the receipt.
- Reconcile crash-between-spawn-and-receipt: on supervisor start, scan open intents, probe for an already-running claim at that generation, adopt or fail it.

Ownership hierarchy (hierarchical, NEVER mutual):

```
systemd --> harnessd --> per-run scope --> task scopes
```

- Supervisor MUST live outside the runner process group.
- `bin/runplan` MUST fail closed when no supervisor is reachable; MUST start one (systemd user unit, `Restart=always`).
- `bin/harness-init.sh:41` swallows systemctl failure — MUST fail closed.

## P2 — attribution + conformance

- `src/fail-taxonomy.js` `UNKNOWN_RESULT` scope `'task'` → MUST default to infra pending evidence. Charging unknown faults to tasks caused 5 misattributed gate reds (r2).
- Adapter/transport/wrapper fail classes MUST re-dispatch, never quarantine the task.
- One vertical conformance test per seam, each with a hostile fake:
  - engine↔wrapper — DONE (`wrappers/test/adapter-conformance.test.sh`, reproduces r4 in ~8s)
  - supervisor↔run — SIGKILL supervisor mid-run; assert resume + `performed` receipt
  - resolver↔actuator — assert every decided action produces a receipt or an open-intent fault
- Test discovery MUST match every test dir. `wrappers/test/*.sh` was unmatched and 12 codex-wrapper tests were silently inert — fixed; re-audit globs when adding dirs.

## P3 — blocked-retryable

- Split states: `blocked-retryable` (re-entrant) vs `quarantined` (terminal). Do NOT overload one state.
- Persist precondition fingerprint + strategy fingerprint on entry.
- **Invariant: spend an attempt ONLY when base, env, provider, prompt, scope, or strategy fingerprint changed.** Otherwise no attempt — not a retry.
- Cause routing:
  - infra/adapter → re-dispatch (attempt was never delivered); free ONLY after reconciliation proves no execution
  - environment → deterministic repair, rerun isolated failing check, then gate. NO agent.
  - missing sibling output → re-attempt only after sibling lands, against new head
  - genuine task defect → NO retry; one bounded diagnosis call returning a materially different strategy or `give-up` with reason
- Scope widening MUST amend the task graph/contract. NEVER bypass the scope guard (`fixer-scope-violation` stays enforced).
- Dependents skip only when the dep's artifact is genuinely required. Quarantined state alone MUST NOT cascade skips.

## P4 — readiness verification

Three checkpoints:

1. **Run-level**, keyed by base SHA + env fingerprint: pinned toolchain versions, baseline gate health, disk AND inode headroom, required credentials, registry reachability only when provisioning needs network.
2. **Pre-gate, post-dispatch**: manifest↔lockfile coherence, installed-dependency stamp keyed by (manifests, lockfile, toolchain, platform), executable resolution, writable cache/tmp, worktree ownership + expected HEAD/base, no merge state.
3. **Reserved, NEVER probed**: ports, workspace lease, process/cgroup ownership. Free-port probes are TOCTOU races.

Auto-repair ONLY when deterministic AND ownership-safe:
- stale installed tree → frozen/offline reprovision, then re-verify
- corrupt owned cache → evict that exact entry
- low disk/inodes → remove owned expired worktrees/tmp under identity checks
- owned orphan → kill using PID start-time + cgroup/lease identity

DO NOT auto-repair manifest↔lockfile disagreement — that is task code; repairing it hides a real defect.
DO NOT require a clean worktree after implementation.
DO NOT probe network for offline gates.

Gap this closes: deps are provisioned before dispatch; the agent may edit dependencies during implementation, leaving the installed tree stale at gate time. Nothing re-checks today.

## P5 — idle telemetry

- Existing local signals already in `src/engine/watchdog.js`: CPU ticks (`DEFAULT_CPU_SAMPLE_INTERVAL_MS`), `newestMtimeMs`, git status delta, first-output deadline, per-class idle timeouts.
- **Local silence MUST NOT be kill evidence.** Server-side model computation and a dead request are locally identical; `wchan` cannot separate healthy `ep_poll` from wedged.
- Kill requires provider-correlated evidence: attempt lease on an independent control channel — `attempt_id`, monotonic status/heartbeat, expiry, cancellation receipt.
- Without provider status: classify `provider-indeterminate`, apply conservative observed-tail deadline, then **fenced** cancellation.
- Progress vector ships as telemetry + soft suspicion only.

## Evidence backlog

Survey all real run journals (this repo + other harness-run repos) before finalizing P3/P4 scope. Record per fail class: occurrences, current handling, whether the plan addresses it. Append findings below.

### Findings — measured 2026-07-26

Source: 151 journals across 8 repos (`/home/user/Projects/*/runstate`), 20442 events, 3281 quarantines.

Repair ladder execution, all repos:

| failClass | repair.attempt | success | quarantined | rate |
|---|---|---|---|---|
| dep-provision-failed | 852 | 6 | 848 | 1% |
| gate-not-green-after-fixer | 0 | 0 | 277 | never ran |
| zero-diff | 94 | 94 | 86 | 100% |
| wrapper-engine-down | 7 | 3 | 141 | 43% |
| agent-idle | 59 | 23 | 33 | 39% |
| unknown | 0 | 0 | 57 | never ran |
| merge-conflict | 0 | 0 | 39 | never ran |
| wrapper-rate-limited | 0 | 0 | 35 | never ran |
| protected-path-touched | 0 | 0 | 26 | never ran |
| fixer-scope-violation | 0 | 0 | 24 | never ran |
| wrapper-timeout-repeated | 0 | 0 | 23 | never ran |
| repeated-failure-stop | 0 | 0 | 20 | never ran |
| gate-red-code | 0 | 0 | 19 | never ran |
| lockfile-out-of-sync | 4 | 4 | 4 | 100% |

Load-bearing facts:

- **10 of 18 fail classes have NEVER produced a single repair attempt**, though `src/resolver.js` defines policies for most. Diagnosis exists; actuation does not. Confirms P1 ordering.
- `dep-provision-failed` is the largest class overall (4847 occurrences) and its repair succeeds 1% of the time. Quarantine message: `git snapshot dependency repair failed with exit 1`.
- 966 `task.skipped` carry no failClass — cascade skips from quarantined deps. Confirms P3 cascade rule.
- gate0: 1888 fail vs 445 pass. `fixer.attempt` 965 → `gate-not-green-after-fixer` 900: fixer converts ~7%.
- `notify.failed` 4967 and `run.hung-detected` 3235 dominate event volume — watchdog and notifier emit at spam rate. Treat as separate hygiene defect.
- Stale-deps-at-gate is real but small: `lockfile-out-of-sync` 32 (platform, zync.is), `gate-red-env` 7. Repaired 4/4 when detected. P4 keeps its scope but is NOT the big lever.

Priority correction: P1 stays first, and its acceptance test is behavioral — **every resolver policy action MUST be observed executing at least once against a fake fault**, not merely decided.

### Root cause of "never ran" — proven

Two independent ladders exist and only one actuates in-run:

- `src/engine/repair.js` — in-run repair, hard allowlist: `dep-provision-failed`, `lockfile-out-of-sync`, `gate-red-env`, `wrapper-engine-down`, `journal-append-failed`, `journal-state-unknown`, `orphan-worktree`, `zero-diff`, `agent-idle`, `task-stalled`. This allowlist matches EXACTLY the classes that show `repair.attempt` in the data.
- `src/resolver.js` POLICY_ROWS — richer table (`redo-task`, `reroute`, `park-retry`, `resume`) covering `gate-not-green-after-fixer`, `merge-conflict`, `wrapper-timeout-repeated`, `wrapper-rate-limited`, `gate-red-code`, `fixer-scope-violation`, `protected-path-touched`. **None of these has an in-run actuator.**

Two false-receipt defects, same class as the `run.restarted` lie:

1. `src/runner.js:4427` calls `resolver.execute(decision, options.handlers || {})` — production supplies NO handlers.
2. `src/resolver.js:427` `execute()` returns `{action, skipped: true}` when no handler is registered, and `recordOutcome` computes `success: !outcome || outcome.skipped !== false` → **a skipped action is journaled as a successful resolution.**

Fix rules:
- `execute()` MUST fail closed on a missing handler: return an explicit not-executed result; the caller MUST journal `success: false` and raise an open-intent fault. NEVER journal a skipped action as success.
- Production MUST register a handler for every action in `ACTIONS`.
- Conformance test MUST assert against the PRODUCTION handler set. A test that injects its own handlers proves only the dispatch table and passes while production is empty.

### In flight

Branch `feat/p1-supervision` (worktree `.worktrees/p1-supervision`), commit `c853a82e`:
- `UNKNOWN_RESULT.scope` `'task'` → `'infra'`; `unknown` policy → `['reroute','escalate']`
- restart protocol requested → claim generation → readiness handshake → performed, legacy `run.restarted` still readable
- `bin/harness-init.sh` propagates systemctl failure

Open on that branch:
- `bin/runplan` supervisor fail-closed NOT done
- `src/test/resolver-actuator-conformance.test.js` injects its own handlers — MUST be rewritten against production handlers
- full `bash run-tests.sh` never run
- `resolutionKey` now maps `infra` → RUN_KEY, so two unknown faults anywhere escalate the whole run; key infra attempts per task when `context.taskId` exists