# Chaos Suite Spec — fault-injection acceptance harness

Audience: AI coding agents first. Canonical for P0's `test/chaos/` and PART V test strategy. Plan context: `docs/plans/2026-07-07-design-gap-handoff.md`. This suite GATES every autonomy-phase merge — a feature claiming a scenario must turn that scenario green, and MUST NOT turn others red.

## Ground rules

- Scenarios run against a THROWAWAY git repo fixture (`test/chaos/fixture-repo/` template, copied per scenario into tmp) — NEVER the harness repo itself.
- Fault injection via a **chaos wrapper** (`test/chaos/wrappers/chaos.sh`): a WRAPPER-CONTRACT-conformant adapter whose behavior per task is scripted by env `CHAOS_SCRIPT=<path.json>` mapping taskId → behavior. All agent-side faults are wrapper behaviors — no monkey-patching engine internals.
- Engine-side faults (kills, corruption) injected by the test runner from outside the process.
- Every scenario asserts on: exit code, journal records (kind sequences + payloads), git state (branches/commits/reachability), filesystem residue (worktrees, locks, sidecars), and — where daemon involved — notifications captured by a stub ntfy server.
- Deterministic: chaos scripts are seeded; NO wall-clock dependence (fake/mocked clocks for backoff, windows, watchdog budgets shrunk to seconds via config).
- Runner: `test/chaos/run.sh [scenario-id...]`, exits non-zero on any failure; wired into `run-tests.sh`.

## Chaos wrapper behaviors (closed vocabulary)

`ok` (implement + commit trivially), `gate-red` (commit code failing the fixture gate), `hang` (sleep forever, ignore SIGTERM for `graceDefiance` sec), `orphan-forker` (spawn detached child, then ok), `flood` (emit N MB to stdout), `malformed` (emit schema-invalid event JSON), `noop-exit0` (exit 0, no commit, no artifacts), `quota-error` (emit quota-mapped error), `usage-lie` (omit usage event), `flaky-gate` (fail gate on attempt 1, identical tree passes attempt 2 — implemented via a fixture gate reading an attempt counter file).

## Scenarios

### C1 — Happy path baseline
Plan: 2 waves × 2 tasks, all `ok`. Core assertions (gate from P2): exit 0; every task `committed`; zero residue (no worktrees, no stray branches beyond landed integration, lock released, sidecar cleared). Feature-gated assertions activate with their phase: land intent+done pair (P2, with I4); report file (P3, with X12). C1 is in every phase's gate from P2 onward with the assertions active for that phase.

### C2 — Self-heal ladder
Tasks: one `gate-red` (fixer fixes — chaos fixer scripted to produce green), one `gate-red` with fixer also scripted red (→ quarantine), plus 2 `ok` siblings, one dependent on the quarantined task. Assert: exit 4; first red task lands after fixer; second `task.quarantined` with correct failClass; dependent `task.skipped` (`skipped-quarantined`); siblings landed; `run.done` summary counts match; snapshot bundle (W6) exists for the quarantine. **C2b (activates at P5, not part of the P2 gate):** postmortem record + lesson file emitted; second identical run's fixer prompt carries the lesson id in `attempt.lessons`.

### C3 — Hang + watchdog + resolver
One `hang` task (watchdog budget 5s in config). Assert: pgid killed (orphan check: no chaos processes after run); `task-stalled` classified; resolver attempt 1 `resume` → chaos script switches to `ok` on attempt 2 → task lands; `resolve.decision`/`resolve.outcome` records present with `source:"table"`.

### C4 — Kill matrix (I3) — the load-bearing test
Baseline = C1 plan. Procedure: run once, record journal seq boundaries; then for EVERY seq boundary k: fresh fixture, run with `CHAOS_KILL_AFTER_SEQ=k` (test harness SIGKILLs runner process group immediately after journal reaches seq k), then resume via the resolver path (or direct `runplan` re-invoke pre-daemon). Assert per k: resume converges to C1's terminal state; no task re-implemented whose commit already existed (git commit count per task ≤1 + fixer commits); no duplicate side effects (intent sweep resolutions journaled, none double-applied); no residue. Runtime bound: boundaries of C1 ≈ dozens — MUST run full matrix, no sampling.

### C5 — Daemon crash + re-attach
C1 plan via daemon `--unattended`. Kill daemon (SIGKILL) at each derived run state (queued-on-window with mocked clock, launching, running, resolving via injected C3 hang, blocked-on-decision via C7, landing). Restart daemon. Assert: re-attach per DAEMON spec (adopt live runner / crash path / queue rebuild); terminal state equals uncrashed reference; tick idempotence (journal has no doubled records).

### C6 — Unresolvable + partial run
One task whose every attempt is `gate-red` and fixer red, breaker exhausts. Assert: quarantined (not infinite retries — count `resolve.decision` ≤ table max); escalation `decision.requested` NOT emitted for task-scope breaker (quarantine is terminal for task scope); siblings land; exit 4; digest lists it.

### C7 — Decision flow
Plan with one `requires_decision` task pre-resolved at preflight AND one runtime escalation (forced via `land-failed`: fixture ship.sh scripted to fail once) with `default: {value: "proceed", afterSec: 2}`. Assert at autonomyLevel 2: preflight passes (pre-resolved); land failure escalates; stub ntfy receives actionable notification (buttons match options); default fires after 2s (mock clock); `decision.answered {by: "default"}`; run completes. Re-run at level 1: default does NOT fire; answer via `runplan answer`; double answer → 409.

### C8 — Hostile agents (I5/I6/I7)
Four tasks: `orphan-forker`, `flood` (cap set 1MB), `malformed`, `noop-exit0` (with declared artifacts, X8). Assert respectively: no orphan processes post-run (pgid kill covers fork); log truncated with marker + task still processed; `adapter-contract-violation` attributed to adapter (health degraded, task rerouted or quarantined — per resolver table) NOT `unknown`; `artifacts-missing` at gate before any review dispatched (no review records for it).

### C9 — Journal integrity (JOURNAL-V2, SQLite form)
Unit-level (no full runs): fixture v2 db; (a) crash-kill fuzz — SIGKILL a child appender at randomized points, ≥50 iterations → reopen: `integrity_check` ok, events = exact committed prefix, no partial event; (b) garble bytes mid-file in a fixture copy → `journal corrupt: integrity` fail-closed, no auto-repair; (c) `user_version=3` fixture → version-skew refusal; (d) v1 jsonl fixture → migration correctness incl. jsonl+db-both-present fail-closed.

### C10 — Concurrency (I1)
(a) Second runner same slug while first alive → refused, first unaffected. (b) Stale lock with recycled-pid simulation (write lock with live-but-different-start-time pid) → takeover succeeds. (c) Two different slugs, same machine, parallel → both land, no cross-talk (distinct locks, worktrees, journals).

### C11 — Quota + reroute (X5)
Two adapters enabled (chaos + chaos2). Task 1: chaos emits `quota-error`, chaos2 `ok` → assert reroute journaled, lands via chaos2. Task 2 with both quota-erroring → `quota-parked` with retryAt; mock clock advance → retry → scripted `ok` → lands. All-parked + retry exhausted variant → run paused + escalation.

### C12 — Protected paths + review proof (V1)
(a) Task scripted to modify `.claude/hooks-file` in fixture → `protected-path-touched`, quarantine, NOT fixer-dispatched. (b) Hand-commit an unreviewed change onto a task branch, journal without `review.passed` → land refuses `land-blocked-unreviewed`.

### C13 — Flaky gate (W8)
`flaky-gate` task. Assert: `gate.flaky` journaled; retry did NOT consume a repair rung (repair records absent); lands; stats fixture shows flake. Second flaky occurrence same gate (two flaky tasks) → escalation `fix-flaky-test`.

### C14 — Budget ceilings (V2)
Usage-reporting chaos wrapper with scripted token counts. (a) `maxUsdPerTask` tiny → `budget-exceeded` quarantine after attempt completes (never mid-write). (b) `maxUsdPerRun` → run paused + decision; answer proceed-with-raise → completes.

### C15 — Soak
`test/chaos/soak.sh N` : N cycles of {C2, C3, C11} via daemon with maintenance window enabled between cycles. After N: zero orphan worktrees (`git worktree list` on fixture), zero `plan/*--*` branches unlanded-and-unquarantined, zero chaos processes, lock dir empty, journal archive intact, status cache consistent with journals. Default N=20 nightly; N=3 in CI.

## Scenario ↔ phase gate map

P1: C9(c-d), C12(a-preflight variant), C14 schema side. P2: C1 (core + land-intent assertions), C2 (WITHOUT C2b), C3, C4, C8, C9(a-b), C10, C13. P3: C1 (+report assertion), C5, C6, C7, C11, C14, C15. P4: C4 + diamond-quarantine variants under `scheduler: "dag"` + cross-mode resume (SCHEDULER tests 2/5). P5: C2b. A phase's merge gate = its scenarios green PLUS all previously-green scenarios still green.