# Offload Robustness Phase-1 Follow-on (R1-align, R6, R3, R5, R4, R7) Implementation Plan

Audience: AI coding agents first.

> **For agentic workers:** Execute task-by-task via direct codex dispatch (gpt-5.6-sol, workspace-write, one task per dispatch) in a worktree off `main`; land each wave via the frozen `.claude/scripts/ship.sh`. The mega-plan-harness engine is BYPASSED for this run (agent-idle quarantine unresolved; harness repo mid-refactor). Steps use checkbox (`- [ ]`) syntax.

**Goal:** Complete Phase-1 of the offload control plane in `overdeck/controller/`: config fail-closed alignment (R1), events/metrics seam (R6), per-job workspaces + artifact CAS (R3), capability admission (R5), global scheduler + spill (R4), incident pipeline (R7).

**Architecture:** Extends the landed C1/C2 pilot (`controller/src/` — `ControllerStore` SQLite-WAL store, `TransitionEngine` with all 8 verbs + idempotency + expectedRevision, config loader, token-auth server; `/status` projection lives in `status.ts:buildControllerStatus`, NOT `server.ts`). Simulation/fixture acceptance only — no live host, no `~/.claude` live-infra edits (Phase-2 manual).

**Tech Stack:** Bun + TypeScript + zod (controller conventions: co-located `*.test.ts` bun tests, workspace-gated by root `pnpm -r typecheck` / `pnpm -r test`).

**Contract authority (spec wins over prose here AND over the parent plan):**
- Wire contracts (event shape, `/metrics` + `/api/v1/query` series with EXACT label sets, per-verb transition args, CAS, auth): `docs/specs/2026-07-18-offload-control-plane-spec.md`.
- Task problem statements + invariants: parent plan `docs/plans/2026-07-18-offload-robustness.md` §R1/§R6/§R3/§R4/§R5/§R7 — read the matching section before implementing; do NOT re-derive contracts from this file alone.
- `ControllerStatus` is a CLOSED shape: no task adds fields to `/status` beyond what the spec's normative `ControllerStatus` defines. New state (workspaces, breaker, queue) is projected through the spec's existing fields; anything without a spec home surfaces via events/metrics instead.

---

## Wave Plan

| Wave | Tasks | Files touched (create + modify) | Safe to parallelize? |
|------|-------|--------------------------------|----------------------|
| 0 | Task 0 (R1-align) | modify `config.ts`, `config.test.ts`, `server.ts` | single task |
| 1 | Task 1 (R6) | create `events.ts`, `metrics.ts` (+tests); modify `store.ts`, `transitions.ts`, `server.ts`, `status.ts` | single task |
| 2 | Task 2 (R3) | create `workspace.ts` (+test); modify `store.ts`, `server.ts`, `status.ts` | single task |
| 3 | Task 3 (R5) | create `capability.ts` (+test); modify `store.ts`, `transitions.ts`, `events.ts`, `server.ts`, `status.ts` | single task |
| 4 | Task 4 (R4) | create `scheduler.ts` (+test); modify `store.ts`, `transitions.ts`, `server.ts`, `status.ts` | single task |
| 5 | Task 5 (R7) | create `incidents.ts`, `watchdog.ts` (+tests); modify `store.ts`, `events.ts`, `metrics.ts`, `scheduler.ts`, `server.ts` | single task |

All paths relative to `controller/src/`. Sequential by design: every task extends the shared `store.ts`/`server.ts`/`status.ts` seam — parallel dispatch would conflict. Order = dependency rungs (R6 is the seam A8 reads → first; R4 consumes R3 workspaces AND R5 eligibility → after both; R7 consumes R6 events + R4 scheduler + R5 incident classes → last).

## Execution protocol (every task)

1. Dispatch codex (`gpt-5.6-sol`, effort medium, `-s workspace-write`, cwd = the worktree) with: the parent-plan section, the spec path, this task's contract, and the acceptance commands.
2. Controller gates after each task (run in worktree, via `local-gate`): `cd controller && bun test` all green, `pnpm -r typecheck` 0 errors. No ignored warnings.
3. Commit per task: one short terse imperative sentence. No co-author lines.
4. Land per wave: `bash .claude/scripts/ship.sh land <branch> <worktree>` (merge-to-main + local deploy — overdeck's established method).

---

### Task 0: R1-align — controller config fail-closed semantics match parent §R1 + spec

**Wave:** 0 · **Blocks:** Task 1 · **Blocked by:** —

Landed C2 config diverges from the parent §R1 contract in three testable ways; align the CONTROLLER'S OWN config (`controller.toml`) semantics. (The live gate's `build-remote.json` fail-closed behavior is Phase-2 — the controller does not own that file yet; see Out of scope.)

**Files:**
- Modify: `controller/src/config.ts` — invalid-file preservation + override semantics
- Modify: `controller/src/config.test.ts` — existing tests encode the wrong override behavior; correct them
- Modify: `controller/src/server.ts` — `/health` requires Bearer auth (spec §Auth: every request authenticated; no unauthenticated exception is specced)

**Contract:**
- Parse/schema failure of `controller.toml`: preserve the bad file as `controller.toml.invalid-<epoch>` (never overwrite silently), controller starts `degraded` with exactly one config incident (existing `ConfigIncident` seam) — unchanged fail-closed posture.
- `BUILD_REMOTE_LOCAL_FALLBACK=1` on corrupt config is HONORED as explicit operator intent (parent §R1: "still honored but recorded") — recorded as an operator-override event once Task 1's event seam exists; until then, journaled via the config incident detail. Never silently rejected.
- `/health` returns 401 without valid Bearer; the systemd watchdog/off-laptop watchdog hold the token (Task 5 pins the heartbeat contract accordingly).

**Acceptance:**
- Run: `cd controller && bun test config.test.ts server.test.ts`
- Expected: PASS — corrupt toml → `.invalid-<epoch>` file exists + degraded + one incident; override env honored + recorded on corrupt config; valid `enabled:false`-style disabled state still authorizes local; unauthenticated `/health` → 401, authorized → `{ok:true}`.

- [ ] Failing tests first → implement → gates green → commit → land wave

### Task 1: R6 — structured events + Prometheus metrics + `/api/v1/query`

**Wave:** 1 · **Blocks:** Tasks 2–5 · **Blocked by:** Task 0 landed

**Files:**
- Create: `controller/src/events.ts` — typed event emission + durable event log (store-backed) + JSONL projection
- Create: `controller/src/metrics.ts` — Prometheus text exposition + series registry + host-series republishing
- Create: `controller/src/events.test.ts`, `controller/src/metrics.test.ts`
- Modify: `controller/src/store.ts` — events table + transactional append/read + revision-cursor read API
- Modify: `controller/src/transitions.ts` — every transition emits its typed event in the SAME transaction as its state change
- Modify: `controller/src/server.ts` — `GET /metrics`, `GET /api/v1/query` (Bearer-gated like everything else)
- Modify: `controller/src/status.ts` — expose event-stream cursor per spec's `ControllerStatus` (only if the spec defines it; otherwise no `/status` change)

**Contract:** parent §R6 + spec §Event stream + §Metrics, with these pinned resolutions:
- **Event shape (spec-exact, NO null sentinels):** every event carries the spec's pinned types verbatim — `{ts: string(ISO), job: string, repo: string, host: string, snapshot: string, attempt: number, stage: string, reason: string, rc: number|null, durationSeconds: number}`. ONLY `rc` is nullable. Non-job events (host transitions, config, lease) keep the same shape with job-scoped string fields as `""` and `attempt`/`durationSeconds` as `0` — pin this convention once in the `events.ts` zod schema; zod rejects `null`/missing on any other field.
- **Durability semantics:** event append is transactional with its state change (no event without state change and vice versa); rollback aborts both. Restart replays in strict revision order from any cursor; SQLite→JSONL projection is derived, never the source.
- **Host-series republishing (spec line ~131):** node_exporter-originated host series are REPUBLISHED by the controller under `build_offload_host_*` exact names + label sets (`disk → {host,mountpoint}`, `core load → {host,core}`, `temp → {host,sensor∈pkg|max|crit}`) because the consumer queries ONLY the controller's `/api/v1/query`. Ingestion is behind an injectable sampler interface (fixture-driven in tests; no live node_exporter required).
- Fleet membership derives from `ControllerStatus.hosts`, never from metric series presence.

**Acceptance:**
- Run: `cd controller && bun test events.test.ts metrics.test.ts`
- Expected: PASS — replayed job lifecycle produces the exact event sequence; rollback produces neither state nor event; restart + cursor replay preserves strict revision order; JSONL projection matches store; `/metrics` + `/api/v1/query` return EVERY spec-listed series (multi-mount disk fixture included) with exact label sets, zod-validated; incident fixture (idle builder + stalled queue + 127 storm) exposes queue-oldest > SLO ∧ idle-capacity ∧ 127-count series.

- [ ] Failing tests first → implement → gates green → commit → land wave

### Task 2: R3 — per-job remote workspaces + artifact CAS

**Wave:** 2 · **Blocks:** Task 4 · **Blocked by:** Task 1 landed

**Files:**
- Create: `controller/src/workspace.ts` — workspace lifecycle, artifact manifest, staged pull-back + snapshot-CAS promotion, transport-reattach, TTL-GC
- Create: `controller/src/workspace.test.ts`
- Modify: `controller/src/store.ts` — workspace/manifest/staging records + atomic GLOBAL `max_remote_jobs` reservation (per-host slot reservation is Task 4's, distinct)
- Modify: `controller/src/status.ts` — workspace/publication state projected through the spec's existing `ControllerStatus` fields only
- Modify: `controller/src/server.ts` — route wiring only

**Contract:** parent §R3 + spec CAS section. Hard invariants: promotion is atomic and refused unless (submitted-snapshot digest unchanged ∧ local checkout generation unchanged); refusal preserves staging + emits `artifact-publication-blocked` (severity act) via Task 1 events; NEVER model an rsync-over-live-checkout path. Staged publication is crash-safe: a crash mid-promotion recovers to either fully-promoted or fully-staged, never partial. Transport interruption reattaches exactly once, then fails loud. Completed workspaces are TTL-GC'd (injectable clock). Filesystem effects simulated against temp dirs (no live remote).

**Acceptance:**
- Run: `cd controller && bun test workspace.test.ts`
- Expected: PASS — two concurrent dirty worktrees of one repo complete without cross-contamination; mid-build local edit stays byte-identical after pull-back; stale job cannot publish (CAS blocks); crash mid-promotion recovers cleanly; transport interruption reattaches once; expired workspace GC'd on tick; global reservation never oversubscribes under concurrent admission.

- [ ] Failing tests first → implement → gates green → commit → land wave

### Task 3: R5 — declarative capability admission + quarantine

**Wave:** 3 · **Blocks:** Task 4 · **Blocked by:** Task 1 landed

**Files:**
- Create: `controller/src/capability.ts` — toolchain manifest schema (zod), admission verifier, 126/127 circuit-breaker with PERSISTED `open → half-open → closed` state
- Create: `controller/src/capability.test.ts`
- Modify: `controller/src/store.ts` — per host+command breaker state, manifest records
- Modify: `controller/src/transitions.ts` — `host-quarantine`/`host-unquarantine` route THROUGH the capability service: unquarantine moves the breaker to `half-open`; `closed` (fully restored) ONLY after a successful probe. The landed direct-clear behavior is a contract violation — replace it, update its tests.
- Modify: `controller/src/events.ts` — `capability-missing`, quarantine/restore events
- Modify: `controller/src/status.ts` — breaker/eligibility state projected through the spec's existing `ControllerStatus` fields only
- Modify: `controller/src/server.ts` — route wiring only

**Contract:** parent §R5. Admission check = manifest satisfaction (command presence + version + writable paths + disk + systemd capability), probe execution behind an injectable prober interface (fixtured in tests; no SSH in Phase 1). Repeated 126/127 same command+host → breaker opens → class-quarantine; half-open re-probe restores; breaker state survives restart.

**Acceptance:**
- Run: `cd controller && bun test capability.test.ts transitions.test.ts`
- Expected: PASS — manifest miss → not admitted + one `capability-missing` event (no 127 storm); repeated failure quarantines; `host-unquarantine` yields `half-open`, full restore only after probe pass; breaker state survives restart; all pre-existing transition invariant tests still green.

- [ ] Failing tests first → implement → gates green → commit → land wave

### Task 4: R4 — global cluster scheduler + spill policy

**Wave:** 4 · **Blocks:** Task 5 · **Blocked by:** Tasks 2, 3 landed

**Files:**
- Create: `controller/src/scheduler.ts` — durable global FIFO, least-loaded eligible dispatch, PER-HOST atomic slot reservation, spill lease, recall-spill
- Create: `controller/src/scheduler.test.ts`
- Modify: `controller/src/store.ts` — extend `QueueTicketRecord` with placement; per-host slot reservation records (DISTINCT from Task 2's global `max_remote_jobs` — two counters, two release lifecycles, both pinned in types)
- Modify: `controller/src/transitions.ts` — `recall-spill` + `admission-reconcile` act on scheduler state
- Modify: `controller/src/status.ts` — queue/dispatch state projected through the spec's existing `ControllerStatus` fields only
- Modify: `controller/src/server.ts` — route wiring only

**Contract:** parent §R4. Eligibility = available ∧ capability-passed (Task 3) ∧ not quarantined ∧ per-host slot reserved. Reservation is atomic at placement (no check-then-start), released exactly once on job terminal state, durable across restart (no reservation leak after crash). Spill rule EXACT: laptop admits a build ONLY when (ALL builders overloaded ∧ queue length > builder count); spill always under an expiring lease (C2 `FallbackLease`); ticket identity = PID+starttime/systemd unit — `lsof` MUST NOT appear in any correctness decision. Fleet size dynamic — no hardcoded host list.

**Acceptance:**
- Run: `cd controller && bun test scheduler.test.ts`
- Expected: PASS — N-builder simulation dispatches least-loaded eligible; concurrent placement never oversubscribes a host slot; restart mid-dispatch leaks no reservation; spill engages ONLY at (all-overloaded ∧ queue > builders), not one build sooner; adding a builder rebalances new dispatch; dead ticket reclaimed by starttime.

- [ ] Failing tests first → implement → gates green → commit → land wave

### Task 5: R7 — incident pipeline + off-laptop watchdog contract + critical-temp paging

**Wave:** 5 · **Blocks:** — · **Blocked by:** Tasks 1, 4 landed

**Files:**
- Create: `controller/src/incidents.ts` — reducer/deduper over Task 1 events → durable incident records; injectable notifier sink
- Create: `controller/src/watchdog.ts` — the off-laptop watchdog OBSERVER: standalone-runnable (bun entry, no store import) loop polling controller `/heartbeat` + collector `/health` with Bearer token via injectable fetcher/clock; on liveness failure applies parent §R7's controller/collector-down row: restart attempt (injectable restarter) then page (injectable notifier). Fixture-tested here; debian1 DEPLOYMENT is Phase-2.
- Create: `controller/src/incidents.test.ts`, `controller/src/watchdog.test.ts`
- Modify: `controller/src/store.ts` — incident table `{key, firstSeen, lastSeen, count, affectedJobs, remediation, cooldownUntil, autoResolveCondition, state}` (the existing config-incident seam cannot carry this — new durable schema)
- Modify: `controller/src/events.ts` — incident lifecycle events
- Modify: `controller/src/metrics.ts` — alert-state series exposition
- Modify: `controller/src/scheduler.ts` — crit-temp dispatch pause/clear integration
- Modify: `controller/src/server.ts` — `GET /heartbeat` (watchdog liveness: Bearer-auth per Task 0, returns `{ok, revision, lastEventTs}`; this IS the off-laptop watchdog's observation contract)

**Contract:** parent §R7. The page-vs-auto-heal table in parent §R7 is the EXACT reducer matrix — one reducer rule per row, implemented as data (table-driven), not scattered conditionals. Incidents are NOT added to `/status` (spec's `ControllerStatus` has no incident field; spec wins) — they surface via the notifier sink, alert-state metrics, and incident lifecycle events. Notifier is an injectable interface (test fixture records notifications; real sink wiring is Phase 2). Crit-temp: page ONLY at pkg ≥ crit + pause new dispatch to that host until clear; NO warn-tier temp noise. This task ships the heartbeat endpoint AND the watchdog observer module (`watchdog.ts` — fixture-tested restart-then-page for BOTH controller-down and collector-down, each detected independently); only its debian1 DEPLOYMENT is Phase-2 manual.

**Acceptance:**
- Run: `cd controller && bun test incidents.test.ts watchdog.test.ts`
- Expected: PASS — each parent-table row's fixture produces exactly one deduped incident with remediation + auto-resolve + correct page/no-page decision through the notifier fixture; crit-temp fixture pages + pauses dispatch, normal-temp does not; `/heartbeat` 401 unauthenticated, correct shape authorized; watchdog fixture observes controller-down (fetcher rejects) → one restart attempt then one page; collector-down likewise → one restart attempt then one page, detected independently of controller state; healthy heartbeat → no action.

- [ ] Failing tests first → implement → gates green → commit → land wave

---

## Out of scope (explicit)

- **`build-remote.json` fail-closed semantics (parent §R1's original target):** that file belongs to the LIVE `~/.claude` gate; the controller does not own it until Phase-2 wiring (parent §R2 "owns build-remote.json writes" is a Phase-2 obligation). Task 0 aligns the controller's OWN config to the same semantics. Deferred, not silently dropped.
- Phase 2 (live `~/.claude` engine obeys controller; debian1 watchdog deployment; real notifier sink) — MANUAL, never in this run.
- dynwf-ux follow-up / landing — mega-plan-harness is mid-refactor in a parallel session; author its plan there after the refactor lands.
- Pre-existing `/ci` fixture regression on main (`pages.spec.ts`) — separate fix, not this plan.

## Self-Review (authoring)

1. Review round 1 (gpt-5.6-sol/medium, 2026-07-19): 9 findings, all verified against code/spec and addressed — R1 divergence → Task 0; host-series republishing → Task 1; `status.ts` seam added to every task list + `ControllerStatus` closed-shape rule; quarantine-verb contradiction → Task 3 transitions rework; dual reservation split → Tasks 2/4; event durability/replay/sentinel semantics → Task 1; incident schema/notifier/reducer-matrix/heartbeat → Task 5; reattach+TTL-GC restored → Task 2; `/health` auth → Task 0.
2. Seam consistency: file lists match actual C1/C2 layout incl. `status.ts` projection ownership; every cross-task dependency (reservation split, eligibility, events, scheduler pause) names the owning task.
3. Wave purity: single-task waves; shared-file overlap makes parallelism unsafe — stated, not hidden.