# Build/Offload Robustness Plan

Audience: AI coding agents first. Optimize for activation, not prose.

Scope: the laptop→cluster build-offload **control plane**, built as a NEW overdeck component `controller/` (a Bun service sibling to `collector/`, systemd unit `overdeck-controller`, binds 127.0.0.1:8787). Kills the failure classes from the 2026-07-18 incident + the gpt-5.6-sol review (report `/home/user/codex-build-ci-offload-review.md`). Consumed by Overdeck v1 (`docs/plans/2026-07-17-overdeck-v1.md` tasks A8/U5/X1) — this plan builds the producer A8 already reads at :8787; Overdeck only reads + acts on it.

**Home + phasing (READ FIRST).** The controller is greenfield in `overdeck/controller/`, NOT the live `~/.claude/{bin,lib}` scripts. Two phases, per ANNOYANCE_FATIGUE.md section 8 (migrate, don't break; leave live running until the replacement is verified):
- **Phase 1 (this plan): stand up the controller in overdeck** — real state store, reconciler, the spec's HTTP API (`/status`, `/transition/:verb`, `/api/v1/query`), events, incident pipeline. Acceptance is simulation/fixture based (no live host needed). Closes overdeck A8's consumer-ahead-of-producer gap immediately (:8787 becomes real). Does NOT touch live `~/.claude` infra.
- **Phase 2 (later, MANUAL — not in any autonomous run): wire the live `~/.claude` engine to obey the controller** (`remote-build.mjs` reads desired-state from :8787). Touches live build infra → hand-done + live-verified, then the old scattered scripts retire.

**Execution is pilot-first.** The controller is a large stateful service; the run preset is deliberately cheap (codex-cursor: composer-2.5 coder, gpt-5.6-sol/low reviewer). First run = a small **foundation pilot** (C1 scaffold+server+gate-wiring, C2 store+state-machine+transition-API+`/status`+fail-closed-config) to prove the loop + the hard invariants (revision monotonicity, idempotency replay, lease expiry, crash-recovery). Verify quality, THEN fan out R6 → R3/R4/R5 → R7 in follow-on runs. Task IDs R1–R7 below are the full Phase-1 backlog; the pilot (C1/C2) is R1+the R2 spine split for observability.

Core principle: **one desired-state controller owns all offload state; every mutation goes through its typed API; nothing else edits config/flock/labels.** No policy in operator memory.

**Contract authority (READ FIRST):** the wire contracts R1–R7 build to are pinned in the source-of-truth specs, NOT re-derived here. When a task's prose and a spec disagree, **the spec wins**.
- Control-plane / controller wire contract (ControllerStatus shape, `/status`, `/api/v1/query` metric series + labels, `POST /transition/:verb` per-verb args + idempotency, event stream, artifact-CAS): `docs/specs/2026-07-18-offload-control-plane-spec.md` (co-located in this repo).
- Overdeck consumer seam (what the deck reads/acts): `/home/user/Projects/overdeck/docs/specs/2026-07-18-overdeck-app-spec.md`.
Each spec carries an **Implementation gates** table — the exact consumer-ahead-of-producer surfaces these tasks must close.

Build target: NEW `overdeck/controller/` Bun service (own `package.json`/`tsconfig.json`/`bun.lock`, co-located `*.test.ts` Bun tests, added to `pnpm-workspace.yaml` so root `pnpm -r typecheck`/`pnpm -r test` gate it — mirror `collector/`). All acceptance runs `cd controller && bun test <file>`. Reference-only (the live engine this eventually supersedes in Phase 2, DO NOT edit in Phase 1): `~/.claude/{bin/local-gate,lib/remote-build.mjs,lib/buildslot.sh,build-remote.json}`.

## Wave Plan

Sequenced as pilot-first runs (each row = one codex-cursor run; verify quality before the next):

| Run | Wave | Tasks | Scope | Parallel? |
|-----|------|-------|-------|-----------|
| **PILOT** | 1 | C1 | scaffold `controller/` Bun service + HTTP server + token auth + `pnpm-workspace` gate wiring + empty `/status` | single |
| **PILOT** | 2 | C2 | SQLite-WAL store + per-host state machine + transition API (`/transition/:verb`, expectedRevision+idempotency) + full `/status` + fail-closed config (folds R1) | deps C1 |
| 2 | 1 | R6 | structured events + Prometheus metrics + `/api/v1/query` — the seam overdeck A8 reads | deps C2 |
| 3 | 1 | R3, R4, R5 | per-job workspaces + artifact CAS · global cluster scheduler + spill · capability admission + quarantine | ✅ after C2/R6 |
| 4 | 1 | R7 | loud-failure incident pipeline + off-laptop watchdog + critical-temp paging | deps R6 |

Land mode: **local-merge-to-main — NO PR, NO push.** Overdeck has NO git remote; PR-gating is impossible + unneeded (solo, local). `meta.land_mode = merge-to-main`, `base_branch = main`, `gate0_mode = strict`. The engine lands each task via the frozen `.claude/scripts/ship.sh` (materialized by `ship-init --mode merge-to-main`). This is overdeck's established method (A8/U5 landed to main at 1444fe4).

Ordering rule: stop at the first rung that holds. **R0 landed in `~/.claude` (commit 751fa4c) — out of scope here** (it hardened the live gate, a Phase-2/live concern). Phase-1 execution: **pilot C1 → C2**, then R6 (the seam A8 reads) → R3/R4/R5 → R7 last. R1 (fail-closed config) folds into C2 (the controller owns + validates its config).

---

### Task R0: exit-75 admission-timeout side-channel

**Wave:** 0 · **Blocks:** — · **Blocked by:** —

Problem: `local-gate` reads a child exit `75` as "admission timed out, requeue". A build can legitimately exit `75` (EX_TEMPFAIL) for its own reason → gate reruns it → possible double-execute. Admission timeout MUST NOT travel in the child's exit-code namespace.

**Contract:**
- `buildslot.sh` writes its admission outcome to a status file whose path is passed in env `BUILD_SLOT_STATUS` (gate-created temp, per invocation): `admitted` on slot grant, `timeout` on admission-wait expiry. The status file — NOT the exit code — is authoritative for "did admission time out".
- `local-gate` requeue decision reads `BUILD_SLOT_STATUS == timeout`, NOT `status === 75`. A child's own `75` with `BUILD_SLOT_STATUS == admitted` passes through UNCHANGED (EX_TEMPFAIL to the caller, no rerun).
- Caller-owned `BUILD_SLOT_TIMEOUT` behavior (no injection) unchanged. No status file present (older buildslot) → fall back to current behavior, log a one-line `status-file-missing` warning (fail-safe, not silent).

**Acceptance:** extend `tests/local-gate-requeue.test.sh` — (a) child that itself exits 75 while `BUILD_SLOT_STATUS=admitted` → NOT rerun, rc 75 passes through; (b) real admission timeout writes `timeout` → requeue path fires; (c) missing status file → legacy path + warning line. `bash tests/local-gate-requeue.test.sh` PASS; `bash tests/buildslot-ghost.test.sh` still 8/8.

- [x] **LANDED** commit 751fa4c — failing cases first, implemented, PASS, committed. Excluded from the run.

### Task R1: fail-closed config

**Wave:** 0 · **Blocks:** — · **Blocked by:** —

Problem: invalid/missing `build-remote.json` → `disabled` → gate authorizes LOCAL execution. Under remote-only intent this is fail-open — the guarantee evaporates on control-plane corruption.

**Contract:**
- Schema-validate `build-remote.json` on load (zod/JSON-schema). Valid `enabled:false` is a legitimate disabled state (local allowed). A PARSE/SCHEMA failure is NOT — it MUST fail closed: preserve the bad file (`build-remote.json.invalid-<epoch>`), emit a typed incident event (R6), and refuse to silently authorize local work. `enabled:true` + unparseable → exit with a named config error, never degrade to local.
- Env override `BUILD_REMOTE_LOCAL_FALLBACK=1` still honored (explicit operator intent) but recorded as an event.

**Acceptance:** new `tests/remote-config-failclosed.test.sh` — corrupt JSON with `enabled:true` intent → gate refuses local + names the error + writes `.invalid-*`; valid `enabled:false` → local allowed. PASS.

- [ ] Failing cases first, implement, run → PASS, commit

### Task R2: build controller (desired-state reconciler)

**Wave:** 1 · **Blocks:** R3, R4, R5, R7, overdeck A8/X1 · **Blocked by:** R1

The spine. Single writer of offload state. Replaces scattered flags/flock/labels/operator-memory.

**Contract:**
- States: `available | draining | maintenance | restoring | degraded` per host + a cluster-level desired mode. Transitions are the ONLY way state changes.
- Persistence: transactional store (SQLite WAL) holding desired + observed state, a monotonic `revision`, the local-fallback lease (with expiry), per-host capacity reservation, CI label/service intent. Crash-resumable: reconcile loop re-derives actions from desired vs observed after any restart (laptop reboot included).
- Ownership: owns `build-remote.json` writes, the fallback lease, buildslot capacity, and CI runner label/service state. Direct mutation of those by anything else is DENIED (add a PreToolUse/deny-gate rule + the CLIs call the controller).
- Transition rules (invariants): `draining → maintenance` legal ONLY when remote jobs + CI jobs on that host are idle; `restoring → available` requires capability probe (R5) + storage + runner + offload health all green; fallback lease is ALWAYS expiring (no permanent local-fallback).
- API: typed transition endpoint (`box-drain/restore`, `host-quarantine/unquarantine`, `admission-reconcile`, `job-retry`, `ci-reconcile`, `recall-spill`) taking `expectedRevision` + idempotency key; `GET /status` (desired/observed/revision/lease/capacity/hosts). Fail-closed audit: intent journaled before execution; audit-write failure → transition refused.
- Runs as a systemd user unit with watchdog.

**Acceptance:** `tests/controller.test.*` — kill the controller mid-transition for EVERY transition, restart, assert convergence with no double-eligible runner identity, no forgotten fallback lease, no premature maintenance. Stale `expectedRevision` → 409 no-op. Replayed idempotency key → prior result, no re-exec. Corrupt config (R1) → `degraded` + one incident, never local fail-open.

- [ ] Contract types + failing tests, implement, run → PASS, commit

### Task R6: structured events + metrics/status endpoint

**Wave:** 1 · **Blocks:** overdeck A8 · **Blocked by:** R2

The seam Overdeck reads. No log-scraping downstream.

**Contract:**
- Every state transition + job lifecycle step emits a typed event: `{ts, job, repo, host, snapshot, attempt, stage, reason, rc, duration}`. Events are the monitoring source; `local-gate.log` stays diagnostic only.
- Controller exposes `GET /status` (R2 shape) + Prometheus metrics on 127.0.0.1: cluster queue depth + oldest-age + p95, per-host running/slots/dispatch-accept, per-job stage, 126/127 counts by host+command, epoch/CAS-mismatch count, pull-back bytes+duration, fallback-lease state. Per-host machine metrics (load, per-core %, mem, swap, net up/down, disk df, **cpu temp pkg/max/crit**) come from node_exporter already running — the controller ADDS the queue/job/dispatch/temp-threshold series, does not duplicate host collection.
- Bind localhost only; bearer or unix-socket auth (match Overdeck collector token convention).

**Acceptance:** `tests/controller-metrics.test.*` — a replayed job lifecycle produces the exact event sequence; `/status` + `/metrics` shapes validate; incident fixture (idle builder + stalled queue + 127 storm) exposes queue-oldest > SLO ∧ idle-capacity ∧ 127-count series. PASS.

- [ ] Failing tests, implement, run → PASS, commit

### Task R3: per-job remote workspaces + artifact CAS

**Wave:** 2 · **Blocks:** R4 · **Blocked by:** R2

Kills epoch-race discards AND the unsafe whole-mirror pull-back (review's P0: `syncPull()` rsyncs the remote tree into the LIVE checkout — a local edit during a build can be overwritten).

**Contract:**
- Keep ONE bare object cache per repo. Per job: a unique immutable workspace (worktree/snapshot dir) + job-specific overlay + job-specific output dir. Epochs become publication leases, NOT mutual-interference control — remove shared-mirror epoch fencing from job correctness.
- Publication: remote job writes ONLY a declared artifact manifest into a job staging dir. Pull-back is staged; before atomic promotion into the local checkout, verify (a) submitted-snapshot digest still matches and (b) local checkout generation unchanged (snapshot CAS). Mismatch → do NOT promote, preserve staging, emit `artifact-publication-blocked` (severity act). NEVER rsync a remote working tree over live source.
- TTL-GC completed job workspaces.
- `max_remote_jobs`: atomic reservation (no check-then-start oversubscribe).

**Acceptance:** `tests/remote-workspace.test.*` — two concurrent dirty worktrees from ONE repo both complete with no cancellation/cross-contamination; a local edit made mid-build stays byte-identical after pull-back; a stale job cannot publish (CAS blocks); transport interruption reattaches once. PASS.

- [ ] Failing tests, implement, run → PASS, commit

### Task R4: global cluster scheduler + spill policy

**Wave:** 2 · **Blocks:** — · **Blocked by:** R2, R3

One global queue across debian1..N; laptop is overflow-only.

**Contract:**
- ONE global FIFO queue (durable, in the controller). Dispatch each build to the least-loaded ELIGIBLE builder (available + capability-passed + not quarantined + capacity-reserved). Atomic capacity reservation on placement. Ticket identity = PID+starttime / systemd unit (no PID-reuse misattribution). Remove `lsof` from correctness decisions.
- Spill rule (EXACT): the laptop runs a build ONLY when ALL builders are overloaded AND queue length > builder count. Spill uses an expiring lease (R2); when a builder frees, in-flight spill is NOT force-killed but no new spill is admitted. Recall-spill verb drains stale spill.
- Host join: `debian2..N` enroll via controller (capability probe R5 → eligible). Fleet size is dynamic — no hardcoded host list anywhere.

**Acceptance:** `tests/cluster-scheduler.test.*` — N-builder simulation: dispatch goes to least-loaded; spill engages ONLY at (all-overloaded ∧ queue > builders) and not one build sooner; adding a builder rebalances new dispatch; ghost/dead ticket reclaimed by starttime, not lsof. PASS.

- [ ] Failing tests, implement, run → PASS, commit

### Task R5: declarative capability admission + quarantine

**Wave:** 2 · **Blocks:** — · **Blocked by:** R2

Kills the silent-127 class (missing `tsc`/`playwright` → hours of exit-127).

**Contract:**
- Per repo/command toolchain manifest (declared, versioned). At admission the controller verifies the target host satisfies the manifest (command presence + version + writable paths + disk + systemd capability) — NOT just SSH + runner hash as `remoteDoctor()` does today.
- Repeated `126/127` for the same command+host → circuit-breaker `open` → host quarantined for that class, new admissions stop, `capability-missing` event (R6). Half-open re-probe restores.
- Manual package installs are NOT parity management — manifest drift is a first-class incident.

**Acceptance:** `tests/capability-admission.test.*` — manifest miss → job NOT admitted to that host + `capability-missing` (no 127 storm); repeated failure quarantines; probe pass restores. PASS.

- [ ] Failing tests, implement, run → PASS, commit

### Task R7: loud-failure pipeline + off-laptop watchdog + critical-temp paging

**Wave:** 3 · **Blocks:** — · **Blocked by:** R2, R6

One incident pipeline. Alert on violated invariants, NOT raw log lines.

**Contract:**
- Reducer/deduper over R6 events → Prometheus alerts → notify. Each alert carries incident key, first/last-seen, count, affected jobs, remediation, cooldown, auto-resolution condition.
- Page-vs-auto-heal table (from review): transient transport → reconnect + one retry, no page; dead lease → reconcile, info; repeated 126/127 → quarantine + page; queue-stalled-while-idle → reconcile once + page; fallback-lease-expired / drain-stuck → page; artifact/CAS mismatch → stop publication + high-sev page (never auto-promote); controller/collector down → watchdog restart then page.
- **CPU temp: page ONLY on critical** (host pkg temp ≥ crit threshold) — pause new dispatch to that host until it clears; NO warn-tier temp noise.
- Watchdog runs OFF the laptop (debian1 or an external heartbeat) observing controller + collector liveness, notifying on a channel the laptop does NOT host — a saturated laptop must not silence its own alerting.

**Acceptance:** `tests/incidents.test.*` — each invariant fixture produces exactly one deduped Item with remediation + auto-resolve; crit-temp fixture pages + pauses dispatch, normal-temp does NOT; controller-down observed by the off-laptop watchdog. PASS.

- [ ] Failing tests, implement, run → PASS, commit

---

## Self-Review (authoring)

1. **Incident coverage:** stranded cohort → R0 + R2 lease; ghost wedge → R4 (starttime identity, no lsof) + R2 reconcile; silent-127 → R5; epoch race + unsafe pull-back → R3; fail-open config → R1; no-one-reads-the-log → R6 + R7.
2. **Boundary:** control plane here; Overdeck read-only over R6 `/status`+metrics and acting via R2 transition API (A8/U5/X1). No Overdeck code mutates offload state.
3. **No stubs:** each task pins a seam (status-file protocol, controller API, manifest, CAS) + leaves the body to implementation — correct plan shape, not placeholder source.
4. **Sequence:** R0 shippable now (live bug). R2 spine before R3–R5. R7 last. Stop-at-first-rung ordering.
5. **Contradiction with prior summary logged:** the "flock held with no fd owner" ghost model is unproven (a live flock needs an open fd); R4 makes correctness depend on PID+starttime, not lsof visibility — sidesteps the unresolved root cause.