# Offload Phase-2 — Controller Goes Live Implementation Plan Audience: AI coding agents first. > **For agentic workers:** REQUIRED SUB-SKILL: Use /run-plan (engine) or /ship to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Take the Phase-1 controller from tested-but-inert modules to a LIVE control plane: a running `:8787` service, a job-reality ingest seam fed by the live `~/.claude` SSH spine, deck action verbs wired through, consumer hardening, watchdog deployment artifacts for debian1, and a real notifier sink. **Architecture:** Everything additive behind flags — the live SSH build spine keeps working untouched if the controller is absent or down (spec rule: "bring R2 up additively behind a flag"). Repo code carries ALL logic; live `~/.claude` spine gains only thin hooks applied post-land. Controller observation/reporting failures are fail-open; invalid live-spine config refuses local execution unless explicit `BUILD_REMOTE_LOCAL_FALLBACK=1`, which is honored + recorded per spec invariant 72. Local services run from the standalone deploy clone `$HOME/.local/share/overdeck/deploy` (a pristine detached checkout of origin/main that `packaging/deploy-local.sh` updates + serves — NEVER the dev root, which a parallel session owns). **Tech Stack:** Bun (controller + collector), zod, SQLite-WAL (`ControllerStore`), systemd --user units, ssh/rsync (port 2222, `~/.ssh/id_ed25519_buildbox`) for debian1 deployment. **Source of truth:** `docs/specs/2026-07-18-offload-control-plane-spec.md` (wire contracts; its "Implementation gates" table names every gap this plan closes) and parent plan `docs/plans/2026-07-18-offload-robustness.md` §R1/§R2/§R7. Phase-1 landed modules: `controller/src/{store,scheduler,workspace,capability,events,metrics,incidents,watchdog,transitions,server,status,config,token}.ts` — all tested, nothing binds a port. Before launch, orchestrator lands `docs/plans/2026-07-20-offload-phase2.{md,jsonl}` to `origin/main` in a docs-only commit; every fresh task worktree therefore contains both files, matching tracked `overdeck-v1` and `offload-robustness` plan docs. **Shared Bearer contract:** one secret, one path: `${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/token`. No `controller-token`, `~/.config/offload/token`, or implicit empty-token fallback remains. O1 explicitly provisions a mode-0600 copy to debian1 at `$HOME/.config/overdeck/token`. | Process / consumer | Ownership | Startup contract | |---|---|---| | Collector | creator | `collector/src/index.ts` calls existing `loadOrCreateToken()` before adapter/gateway construction. Absent file → create mode 0600; existing empty file → exact `CollectorFatalError` code `TOKEN_EMPTY`. | | Controller | creator | T0 `loadOrCreateToken()` uses same path before runtime construction. Absent file → create mode 0600; existing empty file → exact startup code `TOKEN_EMPTY`. | | Offload adapter + deck gateway | in-process collector consumers | Receive collector's already-loaded required non-empty token after creation; MUST NOT create or independently re-read it. Absent-token state is unreachable. Defensive empty input → exact `CollectorFatalError` code `TOKEN_EMPTY`. | | T2 spine report/admission CLI | out-of-process, non-creating laptop consumer | Read exact shared path. Missing/empty/read failure makes hook client unavailable under O3's pinned reporting/admission fail-open rules; MUST NOT create a token. | | T5 watchdog binary on debian1 | out-of-process, non-creating remote consumer | Read configured `tokenFile` after O1 provisioning. Absent file → exact `WatchdogFatalError` code `TOKEN_MISSING`; empty trimmed contents → exact code `TOKEN_EMPTY`; both abort startup before probes. MUST NOT create a token. | --- ## Wave Plan | Wave | Tasks | Files touched | Safe to parallelize? | |------|-------|---------------|----------------------| | 0 | T0 controller live entry | `controller/src/{index,index.test,server,server.test,config,config.test,token,capability,capability.test}.ts`, `packaging/{install-controller.sh,overdeck-controller.service,deploy-local.sh}` | single task | | 1 | T1 spine reality ingest, T5 watchdog deployment + collector health | T1: `controller/src/{index,index.test,server,server.test,store,store.test,status,status.test,events,events.test,transitions,transitions.test,capability,capability.test}.ts`; T5: `controller/src/{watchdog,watchdog.test,watchdog-main,watchdog-main.test}.ts`, `collector/src/server.ts`, `collector/test/server.test.ts`, `packaging/{overdeck-watchdog.service,overdeck-tunnel.service,deploy-watchdog.sh}` | ✅ no overlap within wave; both depend only on T0 | | 2 | T2 spine client pkg, T3 gateway offload verbs, T6 real notifier sink | T2: `spine/**` (all new), `pnpm-workspace.yaml`, `pnpm-lock.yaml`; T3: `collector/src/{actions,actions.test,index}.ts`, `collector/src/adapters/{index,offload,offload.test}.ts`, `collector/test/adapters-wiring.test.ts`; T6: `controller/src/{notify,notify.test,config,config.test,index,index.test,server.test,watchdog-main,watchdog-main.test}.ts` | ✅ no overlap within wave; T6's ordered cross-wave `server.test.ts` edit follows T0/T1 | | 3 | T4 offload adapter hardening | `collector/src/adapters/{index,index.test,offload,offload.test}.ts`, `collector/test/adapters-wiring.test.ts` | single task | | 6 | T7 deck offload action wiring | `packages/deck-ui/src/{MachineDetailModal,MachineDetailModal.test,OffloadControl,OffloadControl.test}.tsx`, `apps/web/src/components/{ci/{CiContent,CiContent.test},inbox/ActionConfirmDialog,design-system/gallery-registry}.tsx`, `apps/web/package.json`, `pnpm-lock.yaml` | single task | | Post-land | O1–O5 operational | live machines only — NEVER run before land | manual, in order | T0 and T1 intentionally modify `controller/src/{index,index.test,capability,capability.test}.ts` in different waves; T1's existing dependency on T0 MUST preserve T0 → T1 ordering. T3 and T4 intentionally modify `collector/test/adapters-wiring.test.ts` in different waves; T4's existing dependency on T3 MUST preserve T3 → T4 ordering. T7 depends on T3's live gateway and blocks O5's from-deck acceptance. --- ### Task T0: Controller live entry + service packaging **Wave:** 0 **Blocks:** T1, T5 **Blocked by:** — **Files:** - Create: `controller/src/index.ts` — the ONLY executable entry; binds the server, owns process lifecycle - Create: `controller/src/index.test.ts` - Modify: `controller/src/server.ts`, `controller/src/server.test.ts` — expose one runtime ownership/close seam for the entrypoint; do not construct duplicate stores/schedulers; permit awaited async transition completion - Modify: `controller/src/config.ts`, `controller/src/config.test.ts` — constrain `bindHost` to literal loopback - Modify: `controller/src/token.ts` — reconcile token path with collector - Modify: `controller/src/capability.ts`, `controller/src/capability.test.ts` — async bounded SSH host-enrollment prober + report-exit breaker API - Create: `packaging/install-controller.sh` — sed-template installer for the user unit (model: `packaging/install.sh`) - Modify: `packaging/overdeck-controller.service` — template placeholders consistent with installer - Modify: `packaging/deploy-local.sh` — install controller unit before adding/restarting it in `SERVICES` - Test: `controller/src/index.test.ts` (new) **Contract (pin EXACTLY):** - `index.ts` startup order: call existing `loadConfigResult()` (NOT fatal `loadConfig()`) → call reconciled `loadOrCreateToken()` → await construction/recovery of one controller runtime/store → bind with existing server factory. Missing config is healthy and uses `defaultConfig()`; invalid config is preserved by `config.ts`, starts controller in `degraded`, and `server.ts:applyConfigHealth` emits one typed incident. NEVER abort or silently authorize local fallback because config is invalid. - Capability construction: export exact `HOST_ENROLLMENT_MANIFEST = {repo:"host-enrollment",command:"true",version:"ssh-exit-0",writablePaths:[],minimumDiskBytes:0,requiresSystemd:false}` plus `createSshCapabilityProber(exec = spawn, home = homedir(), timeoutMs = 5_000): CapabilityProber` from `capability.ts`. Change `capability.ts:31` contract to `probe(host: string, manifest: ToolchainManifest): Promise` and make `CapabilityService.admit(...)` await it. `index.ts` MUST construct this prober and pass it into the single `createControllerRuntime({capabilityProber})`; server binding MUST consume that returned runtime/capability and MUST NOT take `server.ts:106`'s default `new CapabilityService(store, opts.capabilityProber)` path. Probe accepts only safe hostname tokens (no leading `-`; ASCII hostname grammar), rejects any manifest other than exact `HOST_ENROLLMENT_MANIFEST` without spawning, and invokes exact argv-array transport `spawn("ssh", ["-p","2222","-i",join(home,".ssh/id_ed25519_buildbox"),host,"true"], {shell:false})`, resolved by child `error`/`close` events through a Promise. NEVER use `spawnSync`. Per-probe deadline is exactly 5,000ms; timeout terminates child and settles failed. Exit 0 with no signal/error maps to `{commandPresent:true,version:"ssh-exit-0",writablePaths:[],diskFreeBytes:0,systemd:false}`; nonzero, signal, spawn error, or timeout maps to same shape with `commandPresent:false,version:null` and MUST NOT throw/crash controller. This seam proves connectivity enrollment only; it MUST NOT claim arbitrary repo/command MANIFEST satisfaction from `ssh ... true`. - Breaker exit API: widen exact `CapabilityService.recordExit(host: string, command: string, code: number): CapabilityBreakerRecord` seam and implement this state matrix. `closed`: RC `126`/`127` preserves existing failure-count/threshold/open + one `capability-missing` event behavior; every other numeric RC leaves breaker unchanged. `open`: every RC leaves it open; failures keep it open and RC `0` MUST NOT close it because operator `host-unquarantine` is required. `half-open`: next reported finished RC `0` closes it, resets `failureCount` to `0` + `missingEventEmitted` to `false`, and atomically emits the breaker-closed event through existing `buildCapabilityEvent(host, command, "capability-restored", now)` + `ControllerStore.setCapabilityBreakerAndAppendEvent(...)`; any nonzero RC re-opens it with `failureCount >= FAILURE_THRESHOLD` and emits `capability-missing` only when not already emitted. T1 calls this seam for each accepted finished report. `CapabilityService.admit(...)` MUST NOT close a report-fed half-open breaker; probation closes only through a finished success report. - Probe liveness invariant: `/health`, `/heartbeat`, and systemd heartbeat timer paths NEVER execute probes. Probing MUST NEVER block event loop for ≥1s; health/heartbeat remain serviceable while an admission reconcile awaits SSH. - Token: absent shared token is created mode 0600 at `${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/token`; existing-but-empty token remains `TOKEN_EMPTY` startup failure. This preserves `token.ts` auto-create semantics while satisfying non-empty Bearer auth. - Store path: existing `join(dataDir(configHealth.config), "state.sqlite")` contract; default is `$HOME/.config/overdeck/controller/state.sqlite`. Do not invent an `OVERDECK_STATE_DIR` controller seam. - Loopback invariant: change `controller/src/config.ts` `ConfigSchema.bindHost` to `z.literal(DEFAULT_BIND_HOST).default(DEFAULT_BIND_HOST)`, where `DEFAULT_BIND_HOST === "127.0.0.1"`. `index.ts` binds `configHealth.config.bindHost:port`; non-literal configured hosts are invalid config, preserved/degraded through existing `loadConfigResult()`, then bind the degraded fallback's literal `127.0.0.1`. No wildcard, LAN, tailnet, IPv6, or config bypass. Default remains EXACTLY `127.0.0.1:8787`. - Event-log projection: immediately after constructing the single runtime, `index.ts` calls existing `projectEventLog(runtime.store, join(dataDir(configHealth.config), "events.jsonl"))`, then runs the same projection every 1,000ms. Clear the interval on SIGTERM/SIGINT, run one final projection, then close the store. Initial projection MUST complete before bind/`READY=1`; projection errors are never swallowed. SQLite remains authoritative; startup catches JSONL up to current revision and runtime lag is bounded to 1s. No new projector file: reuse `controller/src/events.ts:projectEventLog`; `controller/src/index.test.ts` covers initial, appended, and shutdown-flush output. - sd_notify executable seam: `type NotifyExecResult = {status:number|null;error?:{code?:string;message?:string}}`; inject `notifyExec(file:string, argv:readonly string[]): NotifyExecResult` into entrypoint startup, defaulting to argv-array `spawnSync`. Only when `NOTIFY_SOCKET` is set, spawn exact file `/usr/bin/systemd-notify` with `['READY=1']` exactly once after initial event projection + successful bind, then `['WATCHDOG=1']` every 10s. NEVER use Node/Bun `dgram`: `NOTIFY_SOCKET` is AF_UNIX datagram, unsupported by their UDP APIs. Clear watchdog timer during shutdown. Missing executable (`ENOENT`) logs once, disables further notify attempts, and NEVER crashes/stops server; tests inject `notifyExec` and MUST NOT contact real systemd. - Graceful shutdown: SIGTERM/SIGINT → stop accepting, close store, exit 0. - `install-controller.sh`: substitutes `__CONTROLLER_DIR__` + `__BUN_BIN__` from its OWN landed checkout, installs to `~/.config/systemd/user/`, `daemon-reload`, does NOT enable/start. - `install-controller.sh`: `OVERDECK_INSTALL_CONTROLLER_DRY_RUN=1` writes the fully substituted unit to stdout and exits 0 before directory creation, file install, or `systemctl`; no placeholder may remain. - `packaging/overdeck-controller.service`: set `Type=notify` (not `simple`) + `NotifyAccess=all` (notification sender is `/usr/bin/systemd-notify` child), retain `WatchdogSec=30`, and remove unused `OVERDECK_STATE_DIR` environment line. `READY=1` and watchdog notifications therefore participate in systemd readiness/liveness. - `deploy-local.sh`: invoke `install-controller.sh` BEFORE adding/restarting `overdeck-controller.service`; first post-land deploy MUST NOT restart a nonexistent unit. - `deploy-local.sh`: preserve existing standalone-clone contract at `${OVERDECK_DEPLOY_DIR:-$HOME/.local/share/overdeck/deploy}`. Add `OVERDECK_DEPLOY_DRY_RUN=1`: print exactly `install-controller:/packaging/install-controller.sh` then `restart:overdeck-collector.service overdeck-web.service overdeck-controller.service`, and exit 0 before Git, install, systemd, or network mutation. This executable seam proves controller-unit insertion order. **Behavior:** absent config/token files use healthy defaults + secure token creation; empty token is named fatal error; invalid config remains live-but-degraded with one incident and literal loopback bind; SQLite-backed JSONL projection is active for process lifetime; double SIGTERM exits cleanly; store directory is created if missing. **Acceptance (one executable check):** - Run: `tmp="$(mktemp -d)" && trap 'rm -rf "$tmp"' EXIT && bash -n packaging/deploy-local.sh packaging/install-controller.sh && OVERDECK_INSTALL_CONTROLLER_DRY_RUN=1 bash packaging/install-controller.sh >"$tmp/controller.service" && grep -Eq '^ExecStart=/.+ run src/index\.ts$' "$tmp/controller.service" && grep -qx 'WatchdogSec=30' "$tmp/controller.service" && grep -qx 'Type=notify' "$tmp/controller.service" && grep -qx 'NotifyAccess=all' "$tmp/controller.service" && ! grep -q '__[A-Z_]*__' "$tmp/controller.service" && test "$(OVERDECK_DEPLOY_DRY_RUN=1 OVERDECK_DEPLOY_DIR=/tmp/overdeck-deploy-fixture bash packaging/deploy-local.sh)" = $'install-controller:/tmp/overdeck-deploy-fixture/packaging/install-controller.sh\nrestart:overdeck-collector.service overdeck-web.service overdeck-controller.service' && (cd controller && bun test index.test.ts server.test.ts config.test.ts capability.test.ts && bun run typecheck)` - Expected: PASS — shell parses; rendered unit proves substituted `ExecStart`, `WatchdogSec=30`, `Type=notify`, and `NotifyAccess=all`; deploy dry-run proves installer-before-restart without mutation; server tests run; literal-loopback schema rejects all other bind hosts; missing config/token boot with defaults + created token; invalid config stays live degraded with one incident; JSONL projection starts, appends in revision order, and flushes on shutdown; empty token fails; injected exec captures exact `/usr/bin/systemd-notify` `READY=1` once + `WATCHDOG=1` at 10s cadence only with `NOTIFY_SOCKET`; injected `ENOENT` logs once while server stays live; injected async capability exec captures exact `ssh -p 2222 -i /.ssh/id_ed25519_buildbox true`, exact 5,000ms timeout, no shell, no `spawnSync`, fail-closed hostname/manifest validation, and non-throwing timeout/nonzero mapping; `recordExit` accepts all numeric RCs and obeys the closed/open/half-open state matrix, including atomic half-open success close + `capability-restored`; startup injects this prober into one runtime and never uses default throwing prober; health/heartbeat stay responsive during an in-flight probe; no AF_UNIX socket/systemd/SSH dependency in tests; SIGTERM closes single store; controller `tsc --noEmit` passes. - [ ] Write failing tests → implement → commit --- ### Task T1: Spine reality ingest (job, admission, config) **Wave:** 1 **Blocks:** T2, T3, T6 **Blocked by:** T0 **Files:** - Modify: `controller/src/index.ts`, `controller/src/index.test.ts` — await async pending-transition recovery before bind/readiness - Modify: `controller/src/server.ts` — new routes + await async transition dispatch + reduce incidents after both report routes - Modify: `controller/src/store.ts` — report transaction seam, upsert methods, config replay record, transition CAS - Modify: `controller/src/status.ts` — reported reality reflected in `/status` - Modify: `controller/src/events.ts` — builders for reported job/config lifecycle events - Modify: `controller/src/transitions.ts` — concurrent async admission reconcile promotes only probe-green enrolling hosts under post-probe CAS - Modify: `controller/src/capability.ts`, `controller/src/capability.test.ts` — decouple host enrollment/manifest capability from command-scoped breakers - Create: `controller/src/status.test.ts` - Test: `controller/src/index.test.ts`, `controller/src/server.test.ts`, `controller/src/store.test.ts`, `controller/src/events.test.ts`, `controller/src/transitions.test.ts`, `controller/src/capability.test.ts` **Contract (pin EXACTLY):** - Route: `POST /jobs/report`, Bearer-auth (same token gate as `/status`). Body (zod-validated, reject unknown keys): ```ts { source: "remote-build", // literal; future sources enumerate here host: string, // e.g. "debian1" | "local" key: string, // spine job key (meta.json "key") mirror: string, // 1..255-char basename token; no path separators repo: string, snapshot: string, // spine snapshot identity; O3 pins String(epoch) argv: string[], attempt: number, // positive producer seed; controller owns re-run advancement stage: "started" | "finished", // normative lifecycle field name rc?: number, // required when stage === "finished" startedAt: string, // ISO-8601 finishedAt?: string, // required when stage === "finished" timeoutSec: number } // positive; copied from validated live remote_job_timeout_sec ``` - Identity: controller computes `id = createHash("sha256").update(key + mirror, "utf8").digest("hex")` — direct UTF-8 concatenation, no separator, lowercase 64-hex. `mirror` is mandatory. Body has no `id`; strict schema rejects caller-supplied `id`. Persist derived `id`, `key`, and `mirror` so replay identity remains auditable. - Mirror trust-boundary rule: controller enforces only syntax it can verify from wire data: `mirror` is non-empty, length ≤255, and contains neither `/` nor U+005C backslash (`z.string().min(1).max(255).refine(...)`). It MUST accept any token satisfying that rule and reject blank, separator-bearing, absolute-path, and >255-character values. Controller MUST NOT claim equality with `mirrorName(projectRoot(cwd))`: project root is absent from report. O3 separately pins trusted producer derivation to live `remote-build.mjs:36` and NEVER sends `mirrorPath` or `meta.json.mirror`. - Host enrollment source (spec `Scheduler / admission` → `Host join`): controller store is authoritative, fleet is dynamic, and no host allowlist exists. Fresh SQLite intentionally has zero hosts. After Bearer auth + strict body validation, first `/jobs/report` for unknown `body.host` MUST call `ControllerStore.upsertHost(...)` with this COMPLETE record, bypassing unsafe `store.ts:767` available/capable defaults: `{hostname:body.host,state:"maintenance",role:"builder",slotsTotal:4,slotsUsed:0,runningJobs:0,ciJobsRunning:0,healthStorage:true,healthRunner:true,healthOffload:true,capabilityOk:false,primary:false,enrolling:true,dispatchPaused:false,quarantinedCommands:[]}`. This is observed-only enrollment: report proves host/runner/storage/offload execution path, but MUST NOT assert toolchain capability or availability. `/status.hosts[body.host]` exists immediately while `hostEligible(...)` returns `{eligible:false,reason:"host-not-available"}`. Subsequent reports MUST NOT change host intent/health/capability fields. Existing `ControllerStore.upsertHost` remains explicit controller enrollment/config-seeding seam. Landed `controller/src/config.ts` has no hosts field, and transitions reject unknown hosts; they mutate enrolled records only and MUST NOT become implicit enrollment paths. - Enrollment promotion trigger is existing `POST /transition/admission-reconcile`; add NO route or verb. Preserve spec-exact strict args schema `{reason?:string}`; NEVER add a host argument. After intent journaling + initial revision guard, capture `acceptedRevision` and reconcile whole dynamic fleet. Add side-effect-free seam `CapabilityService.probeAdmission(host: string, manifest: ToolchainManifest): Promise`; invoke it exactly once for every current `{enrolling:true,state:"maintenance"}` host. Launch all host probes concurrently, never sequentially; each probe has T0's 5s deadline and whole reconcile MUST settle within 8s. `probeAdmission` MUST NOT write manifest, breaker, host, event, revision, journal, or idempotency state. Collect results before synchronous state commit; hold no SQLite transaction while awaiting probes. Extend `ControllerStore.commitTransition(...)` with atomic expected-revision CAS inside its existing immediate transaction: apply probe results, transition event, idempotency result, and journal completion only when current revision still equals `acceptedRevision`. Revision mismatch cancels the pending journal and completes as spec-exact 409 `{error:"stale-revision",currentRevision}` with NO probe-result, host, breaker, manifest, reconcile-event, or idempotency write. Green result sets `capabilityOk:true`; only when `capabilityOk`, `healthStorage`, `healthRunner`, and `healthOffload` are all `true` MUST same transition atomically set that host to `{state:"available",enrolling:false}`. Probe miss/nonzero/timeout leaves that host `maintenance`, `enrolling:true`, `capabilityOk:false`, and ineligible. `TransitionEngine.handle` returns/awaits async completion for this verb; `server.ts:handleTransition` MUST await `engine.handle(...)` before serializing response. Preserve existing global drain + scheduler reconciliation in same CAS commit. Existing `box-restore` remains operator transition path and retains same all-green guard before `available`; successful restore also clears `enrolling`. No report route may probe or promote a host. - Async crash recovery: `TransitionEngine.resumePending()` MUST await async `handle`/committed execution for every persisted pending journal, including `admission-reconcile`; runtime construction MUST await `resumePending()` to completion. `index.ts` MUST await runtime construction before server bind and before emitting `READY=1`. A pending admission reconcile therefore resumes, completes all probes and its atomic commit, and clears/completes its journal before any request can be accepted. Startup MUST NOT bind while recovery is in flight. `index.test.ts` MUST cover restart with a persisted pending `admission-reconcile` and prove recovery completion precedes bind/readiness. - Semantics: controller-derived `id` is permanent across builds. Translate report lifecycle before `upsertJob`: first `started` → stored `stage:"running"`; `finished` with `rc === 0` → `stage:"succeeded"`; `finished` with `rc !== 0` → `stage:"failed"`. Add `"succeeded"` to `SCHEDULER_TERMINAL_STATES`; preserve existing terminal values so `upsertJob`'s terminal slot release at `store.ts:1039` fires. A `started` report against terminal persisted state is a NEW ATTEMPT, not replay/regression: set effective `attempt = persisted.attempt + 1`, `stage:"running"`, `rc:null`, `finishedAt:null`, replace start/report timestamps and current report fields, and emit a new started event using incremented attempt. `finished` for unknown `id` creates a complete terminal `JobRecord` from required report fields using effective `attempt = max(1,body.attempt)`. Persist `startedAt`, `finishedAt`, `lastReportAt`, and `timeoutSec`; `lastReportAt` is controller receipt time for each accepted non-replay report. No controller timeout config key is added. - Replay identity key is EXACTLY `(jobId, report stage, effective attempt)`. Effective attempt is `max(1,body.attempt)` for an unknown job, `persisted.attempt + 1` for terminal→`started`, and `persisted.attempt` for `started` against running or `finished` against running/terminal. Canonical lifecycle payload is EXACTLY `{jobId,stage,attempt,rc,startedAt,finishedAt}`, normalized as `rc:null,finishedAt:null` for `started`; no host/repo/snapshot/argv/timeout field participates in replay equality. T1's persisted `JobRecord` is sufficient: persisted `stage:"running"` proves `started` applied for persisted `attempt`; persisted `stage:"succeeded"|"failed"` proves `finished` applied for persisted `attempt`; compare persisted `rc`, `startedAt`, and `finishedAt` to canonical payload. Add no replay table, report hash, lifecycle-history row, or schema field outside `JobRecord`; no additional replay-specific schema migration is needed. - A report is `DUPLICATE` only when its replay identity key names an already-applied lifecycle step and its canonical lifecycle payload is byte-for-byte equal after normalization. Return 200 `{ok:true,id,revision:}` with no job/host write, event, or revision bump. A `finished` report for a running job is new only when `startedAt === persisted.startedAt`; otherwise it is a conflict. A `started` report against running with the same effective attempt but different canonical fields, or a `finished` report against terminal with the same effective attempt but different `rc`, `startedAt`, or `finishedAt`, is `CONFLICT`: return exact 409 `{error:"job-report-conflict",id,stage,attempt}`, leave `JobRecord` and host state unchanged, and append exactly one conflict event. Malformed → 400 `{error:"invalid-args",detail}`; wrong/absent token → 401. - `JobRecord.infraFailure` ingest: every `/jobs/report` upsert writes literal `false`; report bodies have no trusted infra-failure field, and nonzero `rc` MUST NOT be reclassified as infrastructure failure. Jobs known only through reports—including failed jobs—therefore fail the existing `job-retry` typed-infra guard and are not retryable. Keep retries limited to records set `infraFailure:true` by a future trusted typed producer. - Report-fed breaker: require non-empty `argv` and derive exact command identity as `basename(String(body.argv[0]))`; reject empty basename. This MUST equal O3's pre-normalization admission identity. Every NEW accepted `stage:"finished"` outcome calls singleton `CapabilityService.recordExit(body.host, command, body.rc)` exactly once after replay/conflict classification. Duplicate, conflicting, malformed, and unauthorized reports MUST NOT re-feed breaker state. Closed breaker opens after two distinct accepted RC `126`/`127` outcomes; open breaker ignores success as a close signal; `host-unquarantine` calls exact landed `CapabilityService.halfOpen(host, command)` seam; next accepted finished RC `0` closes that half-open breaker through `recordExit` and emits exact `capability-restored`, while any nonzero RC re-opens it. Report route MUST NOT call `admit` or run a probe. - Command-scoped breaker isolation: `HostRecord.capabilityOk` reflects enrollment/manifest probe capability ONLY. `CapabilityService.admit(...)` sets it from that probe outcome; `recordExit`, `quarantine`, `halfOpen`, and command-breaker close/open event paths MUST NOT derive or mutate it from `listCapabilityBreakers`. Opening breaker `(host,commandA)` changes eligibility only for `commandA`; with otherwise-green host state, `hostEligible(host,commandA)` MUST return `{eligible:false,reason:"command-quarantined"}` while `hostEligible(host,commandB)` MUST return `{eligible:true,reason:"eligible"}` and `capabilityOk` remains `true`. Cover this through controller-level admission-route behavior, not only a `CapabilityService` unit assertion. - Atomic report-ingest seam: generalize existing `ControllerStore.mutate(fn)` (`store.ts` immediate SQLite transaction seam) to `ControllerStore.mutate(fn: () => T): T`; keep `this.db.transaction(fn).immediate()` as its implementation boundary. Each non-duplicate report handler executes exactly ONE outer `mutate` call containing ALL ingest writes: unknown-host enrollment, job/config upsert, `CapabilityService.recordExit` breaker writes, incident/degraded-state writes, replay-state write, and EVERY lifecycle/capability event append. Nested store calls MUST remain inside that outer transaction; do not commit service calls independently. Event appends MAY each advance revision, so one accepted report MAY advance revision multiple times (for example, finished job event plus breaker-open `capability-missing`). Return the FINAL meta revision after every ingest write in that transaction; never require job-event revision, breaker-event revision, persisted meta revision, and response revision to be equal. Any job, host, breaker, incident, replay-state, or event write failure rolls back the WHOLE report transaction and returns exact 500 `{error:"report-write-failed"}`. Exact duplicate returns current unchanged revision without transaction writes. Conflict transaction appends exactly one conflict event and performs no `JobRecord` or host write. No stale write/event/revision occurs during status reads. - Add exact `events.ts` builders: | Report | `ts` | `job` | `repo` | `host` | `snapshot` | `attempt` | `stage` | `reason` | `rc` | `durationSeconds` | |---|---|---|---|---|---|---:|---|---|---|---:| | started | `startedAt` | derived `id` | body `repo` | body `host` | body `snapshot` | effective persisted attempt (incremented for terminal→started) | `"started"` | `"report-started"` | `null` | `0` | | finished | `finishedAt` | derived `id` | body `repo` | body `host` | body `snapshot` | current persisted attempt | `"finished"` | `"report-finished"` | body `rc` | `max(0,(Date.parse(finishedAt)-Date.parse(startedAt))/1000)` | | conflict | controller receipt time | persisted `id` | persisted `repo` | persisted `host` | persisted `snapshot` | effective persisted attempt | `"report-conflict"` | `"started-payload-mismatch"` for conflicting started; `"finished-payload-mismatch"` for conflicting finished | persisted `rc` | `0` | - `/status.jobs` remains exact existing `RemoteJobSchema`: `{id,repo,snapshot,stage,host,rc?,pullBytes?,pullDurationSeconds?,publication?}`. `buildControllerStatus(store, configHealth, now)` derives `stage:"stale"` at read time iff persisted `stage === "running"` and `now() - Date.parse(lastReportAt ?? startedAt) >= timeoutSec * 1000`; ledger stage remains unchanged. No background sweeper, timer, cadence, stale event, or stale revision. - Route: authenticated `GET /admission/eligible?host=&command=` returns advisory `{eligible:boolean,reason:string}` from named `ControllerStore.hostEligible(host,command)`. Evaluate observations in fixed order with exact failure reasons: missing host → `unknown-host`; `HostRecord.role !== "builder"` → `not-builder`; `HostRecord.state !== "available"` → `host-not-available`; independent `HostRecord.capabilityOk !== true` → `capability-failed` (this is `/status` `capability.probes[{name:"capability"}].ok`); `HostRecord.dispatchPaused === true` → `dispatch-paused`; exact `CapabilityBreakerRecord(host,command)` with `state === "open"` → `command-quarantined`; `HostRecord.slotsTotal - HostRecord.slotsUsed <= 0` → `no-capacity`; otherwise `{eligible:true,reason:"eligible"}`. Breaker eligibility is exact: `closed` eligible, `half-open` eligible for probation traffic, `open` ineligible; denying half-open makes recovery impossible because no finished report can close it. Never infer exact-command quarantine from `/status`'s display-only `capability.missingCommand`. `hostEligible` MUST NOT create/read `HostSlotReservationRecord` or call `tryReserveHostSlot`; concurrent advisory approvals may race. Live per-box machine-global FIFO `~/.claude/lib/buildslot.sh` remains capacity owner and blocks excess work until one slot frees. - Route: `POST /spine/config/report`, same auth, strict discriminated body: missing `{source:"remote-build",stage:"missing",configPath,observedAt}`, invalid `{source:"remote-build",stage:"invalid",kind:"config-invalid-json"|"config-invalid-shape",detail,configPath,preservedPath,observedAt,override:boolean}`, or valid `{source:"remote-build",stage:"valid",configPath,observedAt,disabled:boolean,override:boolean}`. `preservedPath` is REQUIRED only for `stage:"invalid"`; `disabled` is REQUIRED only for `stage:"valid"`; `override` is REQUIRED for `invalid` and `valid`, and MUST be absent for `missing`. `disabled:true` means valid normalized config has `enabled:false` OR empty `hosts`; spine is off and local execution is allowed. `override:true` means `BUILD_REMOTE_LOCAL_FALLBACK=1` was honored for valid or invalid config; it NEVER makes invalid config valid. Paths are absolute strings; `observedAt` is ISO-8601. Store seam appends one matching revision-ordered event for every new report; invalid opens/updates one typed spine-config incident + degraded dispatch state, valid resolves matching state even when disabled, and missing never creates/resolves/mutates an incident or degraded state. - Config replay record is pinned EXACTLY: `interface SpineConfigStateRecord { configPath: string; lastAppliedBodyHash: string }`. Persist one record per `configPath`; retain ONLY its last hash, no report history/full body. Compute `lastAppliedBodyHash` as lowercase hex SHA-256 of UTF-8 RFC 8785 canonical JSON for the strict-schema-parsed full body. A config report is duplicate iff computed hash equals that path record's current `lastAppliedBodyHash`; return 200 `{ok:true,revision:}` with no event, incident, degraded-state, hash, or revision write. Any different hash is new, including A→B→A; update `lastAppliedBodyHash` inside the same report transaction only after all report effects succeed. New-report response revision follows the atomic report-ingest contract above: FINAL post-transaction meta revision after all event appends, which MAY be more than one revision after request start. - Degraded-state ownership sentinel: invalid report MUST set `dispatch_detail` to exact `` `spine-config-invalid:${configPath}` `` while setting degraded dispatch. Add and use named store constant `NORMAL_CONTROLLER_META = {desired:"available",observed:"available",dispatch_state:"healthy",dispatch_detail:"",reconciler_healthy:"1"}` matching `setMetaDefaults()`. Valid report MUST restore exactly those five values ONLY when current `dispatch_detail` equals exact sentinel for that same `configPath`; otherwise leave desired/observed/dispatch/reconciler state untouched. This ownership guard preserves operator pauses and unrelated incidents. Missing report never checks or changes sentinel state. - Report-fed incident reduction: after each `/jobs/report` and `/spine/config/report` handler completes, `server.ts` MUST call existing singleton `IncidentReducer.reducePending()` before returning the response, matching existing post-workspace/post-transition routing. Do not add another reducer. Two NEW finished reports for same host + command with RC `126` then `127` MUST append breaker-open `capability-missing`; same request path then reduces it and injected `IncidentNotifier` receives exact page-tier `IncidentNotification` for `repeated-command-not-found::capability-missing`. Event mapping (all fields match strict `EventSchema` at `events.ts:6`): | Report | `ts` | `job` | `repo` | `host` | `snapshot` | `attempt` | `stage` | `reason` | `rc` | `durationSeconds` | |---|---|---|---|---|---|---:|---|---|---|---:| | missing | body `observedAt` | `""` | `""` | `"local"` | `""` | `0` | `"config-missing"` | `"config-absent"` | `null` | `0` | | invalid | body `observedAt` | `""` | `""` | `"local"` | `""` | `0` | `"config-invalid"` | body `override === true` ? `` `authorized-local-fallback:${body.kind}` `` : body `kind` | `null` | `0` | | valid | body `observedAt` | `""` | `""` | `"local"` | `""` | `0` | `"config-valid"` | `` `config-valid:${body.disabled ? "disabled" : "enabled"}:${body.override ? "override" : "default"}` `` | `null` | `0` | Incident mapping (all fields match `IncidentRecord` at `store.ts:67`): | Field | Missing report | Invalid report | Valid report | |---|---|---|---| | all incident fields | no row created; existing matching row remains unchanged | values below | resolve matching open row as below | | `key` | — | `` `spine-config:${configPath}` `` | same derived key | | `firstSeen` | — | body `observedAt` on first report; preserve on repeats | preserve existing; create no row when absent | | `lastSeen` | — | body `observedAt` | body `observedAt` | | `count` | — | `1` on first report; increment once per non-replay invalid report | preserve existing | | `affectedJobs` | — | `[]` | preserve existing | | `remediation` | — | `` `Repair ${configPath}; preserved bytes: ${preservedPath}` `` | preserve existing | | `cooldownUntil` | — | `null` | preserve existing | | `autoResolveCondition` | — | `` `valid report for ${configPath}` `` | preserve existing | | `state` | — | `"open"` regardless of `override`; `override:true` NEVER resolves/suppresses incident | `"resolved"` when open; otherwise no-op | Incident identity is exact `key`; at most one open row exists per `configPath`. Replay uses `SpineConfigStateRecord.lastAppliedBodyHash`; authorized fallback and legitimate disabled state remain auditable while invalid incidents stay open. Missing reports emit only informational lifecycle event above: no preservation, incident, degraded-state mutation, or incident resolution. Valid report auto-resolves any open incident for matching path, but restores controller normal state only under exact ownership-sentinel match. Unrelated dispatch pauses/incidents remain untouched. **Behavior:** job ingest never dispatches or mutates existing host intent; unknown-host ingest establishes dynamic observed-only membership with explicit non-eligible values. Missing config reports append only their lifecycle event; invalid/valid reports mutate only matching incident state and sentinel-owned degraded health. Invalid `override:true` records authorized local fallback but keeps config incident/degraded state open. Valid disabled/override combinations remain valid and are recorded exactly. Any report write failure rolls back every ingest write and returns HTTP 500 with exact JSON body `{error:"report-write-failed"}`; do not expose exception detail. No partial job/breaker/incident/replay/event commit. **Acceptance (one executable check):** - Run: `cd controller && bun test index.test.ts server.test.ts store.test.ts status.test.ts events.test.ts transitions.test.ts capability.test.ts && bun run typecheck` - Expected: PASS — auth; mirror syntax accepts arbitrary 1..255-char separator-free token and rejects blank, `/`, U+005C backslash, absolute `mirrorPath`, and >255 chars without asserting producer derivation; derived-id fixtures + caller-id rejection; non-empty argv/command identity validation; first authenticated report for unknown host writes exact observed-only `HostRecord`, makes `/status.hosts[host]` visible, and returns `host-not-available`; report replay cannot promote it; two distinct finished reports for same host+command with RC `126` then `127` produce breaker closed→open, deny that exact command, run reducer on report path, and deliver page-tier notifier record, while another command remains eligible and `capabilityOk:true`; open → `host-unquarantine` → half-open → accepted finished RC `0` → closed emits exact `capability-restored` and changes `hostEligible` from false → true; half-open is eligible before that probation report, and a half-open nonzero report re-opens it; duplicate/conflict outcomes do not transition breaker state; reason-only global `admission-reconcile` rejects `host`, launches all enrolling-maintenance host probes concurrently, settles whole reconcile within 8s, and promotes only all-green hosts; while probes are delayed, a second accepted transition commits and advances revision, then admission reconcile completes as exact stale-revision 409 with only its journal cancelled and zero probe-result/host/breaker/manifest/reconcile-event/idempotency writes; persisted-pending `admission-reconcile` crash-resume completes before bind/`READY=1` and before traffic; `/health`, `/heartbeat`, and WATCHDOG heartbeat remain responsive during delayed probes; failed/timeout probe leaves its host maintenance, enrolling, capability-failed, and ineligible; guarded `box-restore` sets available and clears enrolling only when all-green; started→finished→started for same derived id yields attempt 2 running with cleared `finishedAt`/`rc` and a fresh attempt-2 started event; canonical running-start/terminal-finish duplicates return current unchanged revision with no write/event/revision; mismatched running-start, running-finish `startedAt`, and terminal-finish payloads return exact 409 conflict body, preserve job/host state, and append exactly one pinned conflict event; started→running and finished→succeeded/failed translation; reported jobs persist `infraFailure:false` and cannot `job-retry`; finished report stored terminal + slot released; exact report event fields; breaker-open report proves multiple event revisions inside one atomic ingest and response returns final post-transaction revision; injected write fault at each job/host/breaker/incident/replay/event seam rolls back whole transaction and returns exact 500 `{error:"report-write-failed"}`; clock-controlled read-time stale projection with zero writes; every advisory admission guard without reservation mutation; missing config rejects `preservedPath`/`disabled`/`override`, emits exact missing event, and creates/mutates/resolves no incident; config record retains only `{configPath,lastAppliedBodyHash}`; same last hash is 200 no-op, A→B→A is new, and responses obey final-revision contract; invalid config requires `preservedPath`/`override`, rejects `disabled`, records exact sentinel, and keeps incident open/degraded; valid report requires `disabled`/`override`, rejects `preservedPath`, records all four disabled/override combinations, auto-resolves matching open incident, restores exact `NORMAL_CONTROLLER_META` only for matching sentinel, and preserves unrelated pause/degraded state; both report routes run existing reducer; controller `tsc --noEmit` green. - [ ] Failing tests first → implement → commit --- ### Task T2: Spine client package (`spine/`) **Wave:** 2 **Blocks:** O3 **Blocked by:** T1 **Files:** - Create: `spine/package.json` — name `overdeck-spine`; scripts include `test: "bun test"` and `typecheck: "tsc --noEmit"`; runtime dependency `zod`; dev deps `@types/bun` + `typescript` - Create: `spine/tsconfig.json` — standalone strict Bun/ESNext contract matching `controller/tsconfig.json` - Create: `spine/src/config-schema.ts` — fail-closed `build-remote.json` validator - Create: `spine/src/report.ts` — job lifecycle reporter client - Create: `spine/src/admission.ts` — host-eligibility client - Create: `spine/src/index.ts` — package-internal re-exports; live Node spine MUST NOT import it - Create: `spine/src/cli.ts` — sole live-spine integration surface - Test: `spine/src/config-schema.test.ts`, `spine/src/report.test.ts`, `spine/src/admission.test.ts`, `spine/src/cli.test.ts` - Modify: `pnpm-workspace.yaml` — add exact workspace entry `'spine'` - Modify: `pnpm-lock.yaml` — frozen lockfile importer for `spine` **Contract (pin EXACTLY):** - `config-schema.ts`: strict zod raw schema accepts CURRENT live `~/.claude/build-remote.json` keys plus every `DEFAULT_REMOTE_CONFIG` key; unknown keys ERROR. Export throwing `parseBuildRemoteConfig(raw)` + typed `safeParseBuildRemoteConfig(raw)`. Both success paths return NORMALIZED config through this exact post-validation pipeline, matching live `~/.claude/lib/remote-build.mjs:25–26`: (1) derive `hosts` from `validated.hosts` when it is an array; otherwise derive `[validated.host]` only when legacy `validated.host` is a non-empty string; otherwise `[]`; therefore an array-valued `hosts`, including `[]`, takes precedence and legacy `host` is ignored when both are present; (2) merge field defaults as `{...DEFAULT_REMOTE_CONFIG, ...validated}`; (3) override merged `hosts` with derived hosts after filtering non-string/empty entries and deduplicating by first occurrence, preserving input order. Do not use per-field schema defaults. Pin `DEFAULT_REMOTE_CONFIG` values EXACTLY to live `~/.claude/lib/remote-build.mjs`: ```ts { enabled: false, hosts: [], port: 22, ssh_user: "user", remote_root: "/home/user/builds", health_ttl_sec: 30, connect_timeout_sec: 6, max_remote_jobs: 6, ssh_probe_timeout_sec: 15, ssh_exec_timeout_sec: 1800, rsync_io_timeout_sec: 300, grace_window_sec: 600, reconnect_interval_sec: 10, local_only: [], local_fallback: true, remote_wait_sec: 3600, local_requeue_sec: 900, rsync_only: [], push_excludes: ["node_modules", ".pnpm-store", ".turbo", ".astro", ".cache", "dist", "build", ".next", "coverage", "playwright-report", "test-results", "target", ".tmpjail-work", ".rb-lockhash", ".rb-epoch"], pull_excludes: ["node_modules", ".git", ".pnpm-store", ".cache", "target", ".tmpjail-work", ".rb-lockhash", ".rb-epoch", ".rb-overlay-manifest"], ship_ignored: [".dev.vars", ".env", ".env.local"], } ``` Validated raw values override these defaults except `hosts`, which follows pinned derivation above; current optional `identity_file` and legacy `host` pass through unchanged. Invariant: normalized `hosts` is authoritative and drives live enablement exactly: config is disabled iff `normalized.enabled === false || normalized.hosts.length === 0`; legacy `host` never independently enables config. Normalized output is superset-compatible with every config field consumed by live `remote-build.mjs`; omitted `push_excludes`/`pull_excludes` therefore remain present before unconditional execution dereferences. **Scope:** T2 closes validation + normalization only; it does NOT claim R1 closure. O3 owns preserve-invalid-file, refuse-local, and typed incident emission. - `report.ts`: `reportJob(event: JobReport, opts?: {baseUrl?,tokenPath?,timeoutMs?}): Promise` posts T1 exact job body with mandatory mirror basename token and no caller `id`; `reportConfig(event: ConfigReport, opts?)` posts T1 exact config union, including required `override` on valid/invalid and required `disabled` on valid. Both read `${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/token`, default controller `http://127.0.0.1:8787`, timeout 500ms, and swallow network/HTTP failure (observation fail-open). `OVERDECK_SPINE_REPORT` unset → zero network calls. - `admission.ts`: `hostEligible(host: string, command: string, opts?): Promise<{eligible:boolean,reason:string}>` reads authenticated T1 `/admission/eligible` cached ≤5s. Verdict is advisory and creates no reservation. Another command remains eligible when one capability class is quarantined. Flag unset/controller unreachable/timeout → `{eligible:true,reason:"controller-absent"}`; live per-box `buildslot.sh` remains capacity owner. - `cli.ts` is THE integration surface for Node-run O3 hooks. Pin exact invocations: - `FT_FROM_HOOK=1 ft bun /spine/src/cli.ts parse-config --json ` - `FT_FROM_HOOK=1 ft bun /spine/src/cli.ts report --json` with one exact T1 job/config report object on stdin; select endpoint from strict `stage` union - `FT_FROM_HOOK=1 ft bun /spine/src/cli.ts host-eligible --json ` - CLI contracts: success writes exactly one compact JSON line to stdout and nothing to stderr. `parse-config` success exits 0 with `{ok:true,config}` where `config` is exact normalized parser output; invalid JSON/schema exits 65 with `{ok:false,error:"config-invalid-json"|"config-invalid-shape",detail}`. `report` valid local input exits 0 with `{ok:true}` even when controller delivery fails; invalid/missing stdin exits 64 with `{ok:false,error:"invalid-args",detail}` and performs no request. `host-eligible` valid args exits 0 with exact verdict; invalid args exit 64 using same error shape. No command accepts inline report JSON, preventing argv leakage. **Behavior:** no controller-internal imports; HTTP only. Report/admission clients never write files and never throw network failures. Config parser intentionally fails closed; O3 preserves invalid bytes before refusing local work. `spine/src/index.ts` is never executed/imported by live Node hooks. **Acceptance (one executable check):** - Run: `cd spine && bun test && bun run typecheck` - Expected: PASS — strict parser rejects unknown key; normalized parse output includes every exact pinned default when omitted, preserves validated overrides, and returns exact default `push_excludes`/`pull_excludes`; `{enabled:true,host:"debian1"}` normalizes to `hosts:["debian1"]` and remains enabled; `{enabled:true,host:"debian1",hosts:[]}` normalizes to `hosts:[]` and is disabled because array-valued `hosts` wins; `{enabled:true,hosts:["debian2","debian1","debian2","debian1"]}` normalizes order-preservingly to `hosts:["debian2","debian1"]`; report schema requires mirror basename syntax, excludes id, requires valid/invalid `override`, and requires valid `disabled`; CLI stdout/exit contracts + stdin-only reports are exact; reporters swallow controller failure and honor flag-off; advisory admission propagates denials, scopes quarantine by command, creates no reservation, and fails open when controller observation is absent; spine's own `tsc --noEmit` passes. - [ ] Failing tests first → implement → commit --- ### Task T3: Deck gateway offload verbs **Wave:** 2 **Blocks:** T4, T7 **Blocked by:** T1 (controller routes exist for integration-shaped tests) **Files:** - Modify: `collector/src/actions.ts` — extend the action gateway allowlist + proxy - Test: `collector/src/actions.test.ts` - Modify: `collector/src/index.ts` — construct gateway with resolved controller URL, shared token, and fetcher - Modify: `collector/src/adapters/index.ts` — expose one offload connection resolver taking collector's already-loaded token for adapter + gateway - Modify: `collector/src/adapters/offload.ts`, `collector/src/adapters/offload.test.ts` — correct advertised `job-retry` target key - Modify: `collector/test/adapters-wiring.test.ts` — transitional token-bearing `buildAdapters(config, token)` update required for wave-2 typecheck; T4 later hardens this same test file **Contract (pin EXACTLY):** - Extend existing allowlist with EXACTLY 8 spec verbs: `box-drain`, `box-restore`, `host-quarantine`, `host-unquarantine`, `admission-reconcile`, `job-retry`, `ci-reconcile`, `recall-spill`. - Extend `ActionGatewayDeps` with `controllerUrl`, non-empty `controllerToken`, and injectable `fetcher`; `collector/src/index.ts` calls `loadOrCreateToken()` first, then supplies that returned token to one resolver shared with offload adapter config. Resolver MUST NOT create or re-read token. Creator's default path is `${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/token`; no URL/token re-hardcode in `actions.ts`. - Update every existing `collector/test/adapters-wiring.test.ts` normal-construction call to supply one required non-empty token during T3. This transitional change MUST compile in wave 2; T4 retains ownership of later empty-token assertions and consumer-hardening changes in the same file. - Validate string-only deck args against exact per-verb schemas. Extract required `expectedRevision`, coerce to safe integer, remove it from `args`, then POST `{expectedRevision:number,idempotencyKey:crypto.randomUUID(),args:}` to `/transition/` with Bearer on every attempt. - Gateway ALWAYS mints one UUIDv4 per user invocation. Caller `idempotencyKey` is not accepted or forwarded; controller alone persists replay records. - Before proxy: host targets MUST exist in current `fleet` panel; `jobId` MUST exist in current `remote-jobs` panel. Refuse stale/missing targets with 400. `job-retry` advertisement MUST send `{jobId:job.id,expectedRevision}`, never `{job:...}`. - Audit every allowed attempt/refusal using exact existing `ActionJournalEntry`: `{ts,verb,args,requestedBy,result,rc}`. Do not add `idempotencyKey` or `outcome` fields to collector journal; controller transition journal owns key/result replay. - Controller unreachable → typed 502 `{error:"controller-unreachable"}` (surface honestly; no retry loop in the gateway). **Behavior:** existing `reap|ci-rerun|steer|snooze|decision|abandon|restore` remain regression-green; unknown verb stays 404. **Acceptance (one executable check):** - Run: `cd collector && bun test actions.test.ts adapters/offload.test.ts test/adapters-wiring.test.ts && bun run typecheck` - Expected: PASS — all 8 verbs proxy with gateway-only UUIDs; caller key rejected; revision coerced; current-panel host/job validation enforced; `job-retry` sends `jobId`; controller down → 502; legacy verbs stay green; wiring test uses token-bearing `buildAdapters` flow; collector `tsc --noEmit`, including `test/`, passes in wave 2. - [ ] Failing tests first → implement → commit --- ### Task T4: Offload adapter consumer-hardening **Wave:** 3 **Blocks:** — **Blocked by:** T3 (shared collector test harness churn) **Files:** - Modify: `collector/src/adapters/offload.ts` - Modify: `collector/src/adapters/index.ts` — required-token validation + shared connection resolver; no token file I/O - Create: `collector/src/adapters/index.test.ts` - Test: `collector/src/adapters/offload.test.ts` - Modify: `collector/test/adapters-wiring.test.ts` — supply required token or assert exact empty-token startup error **Contract (pin EXACTLY — the spec's three consumer-hardening rows):** 1. **zod-validate, never cast:** `/status` parses against closed `ControllerStatusSchema` matching `controller/src/status.ts`; every `/api/v1/query` response parses against strict Prometheus vector schema. Reachable wrong shape/non-2xx/malformed JSON MUST throw so collector retains last-good snapshot. NEVER model shape mismatch as an item or `controller-down`. 2. **Token fail-fast:** offload adapter receives collector's already-loaded token as a required string after `collector/src/index.ts:loadOrCreateToken()` succeeds. It MUST NOT read/create token files. Absent-token adapter state is unreachable. Empty input is the only adapter-local token fatal: throw `new CollectorFatalError("TOKEN_EMPTY", ...)` before scheduler/server start and preserve exact code through `collector/src/index.ts` fatal handling. NEVER disable adapter silently, send empty/omitted Bearer, or make a controller request after this error. 3. **`disk_free` per-mount:** consume one series per `(host, mountpoint)` — stop collapsing by host/pairing first mount; panel item carries the mountpoint label. **Behavior:** controller transport failure remains modeled `controller-down` + four `{stale:true}` panels. Reachable bad data throws; collector state retains prior good snapshot unchanged. **Acceptance (one executable check):** - Run: `cd collector && bun test && bun run typecheck` - Expected: PASS — full collector suite, including `test/adapters-wiring.test.ts`, proves collector creation precedes adapter construction, supplies required non-empty token where normal adapter construction is intended, and asserts exact `TOKEN_EMPTY` for defensive empty input; no absent-token adapter branch remains; malformed `/status` + metric vectors reject poll and retain last-good; empty token aborts adapter construction with zero requests; two-mount fixture emits two correctly paired items; collector `tsc --noEmit` passes. - [ ] Failing tests first → implement → commit --- ### Task T5: Watchdog deployment artifacts (debian1) **Wave:** 1 **Blocks:** T6 (O4 consumes it) **Blocked by:** T0 **Files:** - Create: `controller/src/watchdog-main.ts` — compilable entry wrapping the EXISTING `controller/src/watchdog.ts` observer (no logic re-implementation) - Create: `controller/src/watchdog-main.test.ts` - Modify: `controller/src/watchdog.ts`, `controller/src/watchdog.test.ts` — accept entrypoint-resolved token/targets; preserve dedupe + conditional restart-once semantics - Modify: `collector/src/server.ts`, `collector/test/server.test.ts` — authenticated collector health route - Create: `packaging/overdeck-watchdog.service` — systemd --user unit template for debian1 - Create: `packaging/overdeck-tunnel.service` — laptop systemd --user unit owning persistent reverse SSH tunnel - Create: `packaging/deploy-watchdog.sh` — compile-on-laptop, fail-closed binary deployer to debian1 **Contract (pin EXACTLY):** - Collector adds `GET /health` inside existing Bearer gate, returning `{ok:true}`. Missing/wrong Bearer remains 401. No unauthenticated exception. - `watchdog-main.ts` loads strict JSON config from `${OVERDECK_WATCHDOG_CONFIG:-$HOME/.config/overdeck/watchdog.json}`: ```ts { controllerHeartbeatUrl: "http://127.0.0.1:18787/heartbeat", collectorHealthUrl: "http://127.0.0.1:18138/health", tokenFile: string, webhookUrl: string, // HTTPS URL; operator-provided in O4 intervalMs?: number, // integer >10000; default 30000 controllerRestartArgv?: string[], collectorRestartArgv?: string[] } ``` Reject unknown keys/non-loopback target URLs/non-HTTPS webhook/`intervalMs <= 10000`. Missing restart fields and `[]` are both legal observe-only values; non-empty arrays require non-empty strings. Token file is a non-creating trust boundary: absent file throws `new WatchdogFatalError("TOKEN_MISSING", ...)`; empty trimmed contents throws `new WatchdogFatalError("TOKEN_EMPTY", ...)`; both abort before probes and surface exact code in `config-validate`'s JSON error line. Unit never embeds token or webhook URL in argv/environment. - `watchdog.ts`: add explicit observe-only restart contract. Each target carries optional/empty `restartArgv`; `OffLaptopWatchdog` calls `WatchdogRestarter.restart(target,restartArgv)` only when array is non-empty. Remove implicit `systemctl --user restart overdeck-` fallback. First failed probe still notifies once; successful probe clears dedupe. Laptop controller/collector targets MUST use absent/empty restart argv because debian1 has no execution path back to laptop. - Probe deadline: every controller/collector health `fetch` passes `signal: AbortSignal.timeout(10_000)`. Abort, including a hung established connection, is a failed probe and enters existing notify/dedupe handling. Fixed 10s deadline is strictly below every valid `intervalMs`; never start a second probe while one is hung. - Tunnel topology: T0's literal `bindHost === "127.0.0.1"` schema guarantees controller loopback-only; collector remains loopback-only on laptop. `packaging/overdeck-tunnel.service` runs on laptop with exact transport `ssh -N -T -p 2222 -i %h/.ssh/id_ed25519_buildbox -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:18787:127.0.0.1:8787 -R 127.0.0.1:18138:127.0.0.1:31338 debian1`. Unit uses `Restart=always`, `RestartSec=5`, starts after `network-online.target`, and is installed/enabled as laptop user. This reuses existing SSH spine channel; no service binds laptop LAN/tailnet. Debian1 watchdog observes controller and collector only at config's `127.0.0.1:18787`/`:18138`. Laptop freeze drops SSH keepalive/tunnel, probe fails, webhook page fires. - Both health requests carry shared Bearer read from debian1 `$HOME/.config/overdeck/token`; O1 provisions it. O4 provisions watchdog config. `packaging/overdeck-watchdog.service` references config path only. - Standalone artifact: laptop runs `bun build --compile --target=bun-linux-x64 controller/src/watchdog-main.ts --outfile /overdeck-watchdog`. `watchdog-main.ts --help` exits 0 without reading config or starting probes. `watchdog-main.ts config-validate` loads `${OVERDECK_WATCHDOG_CONFIG:-$HOME/.config/overdeck/watchdog.json}`, writes exactly `{"ok":true}` + newline and exits 0 when valid; invalid/missing config writes one JSON error line to stderr and exits nonzero. Debian1 requires no Bun, `ft`, source tree, `node_modules`, or install step. - `packaging/overdeck-watchdog.service`: pin `ExecStart=%h/.local/bin/overdeck-watchdog`; reference config path only. No interpreter/runtime command appears in unit. - `deploy-watchdog.sh`: install/enable/start laptop `overdeck-tunnel.service`; compile in a `mktemp -d` directory; run compiled binary `--help`; create remote `%h/.local/bin` + unit directory; rsync ONLY compiled `overdeck-watchdog` binary mode 0755 to `debian1:~/.local/bin/overdeck-watchdog` and rendered `overdeck-watchdog.service` mode 0644 to `debian1:~/.config/systemd/user/overdeck-watchdog.service` via exact `ssh -p 2222 -i "$HOME/.ssh/id_ed25519_buildbox"`. Verify remote token + watchdog config exist and are mode 0600; run remote binary `config-validate`; `daemon-reload`, enable/restart remote watchdog unit, then require local tunnel and remote watchdog `is-active`. It MUST NOT copy token/config, source, lockfiles, or dependencies; O1/O4 own provisioning. Any failure exits non-zero with one JSON status line. - `deploy-watchdog.sh` dry-run: `OVERDECK_WATCHDOG_DEPLOY_DRY_RUN=1` prints one compact JSON line `{"type":"rsync-argv","argv":[...]}` or `{"type":"ssh-argv","argv":[...]}` for every planned rsync/SSH invocation, then prints fully rendered `overdeck-tunnel.service` and `overdeck-watchdog.service` under exact marker lines `unit:overdeck-tunnel.service` and `unit:overdeck-watchdog.service`. Exit 0 before compile, temp/file writes, local `systemctl`, rsync, SSH, or any network access. Dry-run uses literal no-write staging root `/tmp/overdeck-watchdog-dry-run`. These three argv arrays MUST appear byte-for-byte after JSON parse (`` means `process.env.HOME`, not literal text): ```text ["ssh","-N","-T","-p","2222","-i","/.ssh/id_ed25519_buildbox","-o","ExitOnForwardFailure=yes","-o","ServerAliveInterval=15","-o","ServerAliveCountMax=3","-R","127.0.0.1:18787:127.0.0.1:8787","-R","127.0.0.1:18138:127.0.0.1:31338","debian1"] ["rsync","--archive","--chmod=F755","--rsh","ssh -p 2222 -i /.ssh/id_ed25519_buildbox","/tmp/overdeck-watchdog-dry-run/overdeck-watchdog","debian1:~/.local/bin/overdeck-watchdog"] ["rsync","--archive","--chmod=F644","--rsh","ssh -p 2222 -i /.ssh/id_ed25519_buildbox","/tmp/overdeck-watchdog-dry-run/overdeck-watchdog.service","debian1:~/.config/systemd/user/overdeck-watchdog.service"] ``` Rendered tunnel unit MUST contain exact line `ExecStart=/usr/bin/ssh -N -T -p 2222 -i %h/.ssh/id_ed25519_buildbox -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:18787:127.0.0.1:8787 -R 127.0.0.1:18138:127.0.0.1:31338 debian1`; rendered watchdog unit MUST contain exact line `ExecStart=%h/.local/bin/overdeck-watchdog`. **Behavior:** running deployer twice is idempotent; both user units survive reboot (linger already enabled). Tunnel loss is health failure, never authorization for a remote laptop restart. **Acceptance (one executable check):** - Run: `tmp="$(mktemp -d)" && trap 'rm -rf "$tmp"' EXIT && bash -n packaging/deploy-watchdog.sh && OVERDECK_WATCHDOG_DEPLOY_DRY_RUN=1 bash packaging/deploy-watchdog.sh >"$tmp/deploy-plan" && PLAN="$tmp/deploy-plan" bun -e 'const lines=(await Bun.file(process.env.PLAN).text()).split("\n"); const rows=lines.filter((x)=>x.startsWith("{\"type\":")).map(JSON.parse); const home=process.env.HOME; const want=[["ssh","-N","-T","-p","2222","-i",home+"/.ssh/id_ed25519_buildbox","-o","ExitOnForwardFailure=yes","-o","ServerAliveInterval=15","-o","ServerAliveCountMax=3","-R","127.0.0.1:18787:127.0.0.1:8787","-R","127.0.0.1:18138:127.0.0.1:31338","debian1"],["rsync","--archive","--chmod=F755","--rsh","ssh -p 2222 -i "+home+"/.ssh/id_ed25519_buildbox","/tmp/overdeck-watchdog-dry-run/overdeck-watchdog","debian1:~/.local/bin/overdeck-watchdog"],["rsync","--archive","--chmod=F644","--rsh","ssh -p 2222 -i "+home+"/.ssh/id_ed25519_buildbox","/tmp/overdeck-watchdog-dry-run/overdeck-watchdog.service","debian1:~/.config/systemd/user/overdeck-watchdog.service"]]; for(const argv of want) if(!rows.some((r)=>JSON.stringify(r.argv)===JSON.stringify(argv))) throw new Error("missing exact argv: "+JSON.stringify(argv))' && grep -Fxq 'unit:overdeck-tunnel.service' "$tmp/deploy-plan" && grep -Fxq 'unit:overdeck-watchdog.service' "$tmp/deploy-plan" && grep -Fxq 'ExecStart=/usr/bin/ssh -N -T -p 2222 -i %h/.ssh/id_ed25519_buildbox -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:18787:127.0.0.1:8787 -R 127.0.0.1:18138:127.0.0.1:31338 debian1' "$tmp/deploy-plan" && grep -Fxq 'ExecStart=%h/.local/bin/overdeck-watchdog' "$tmp/deploy-plan" && bun build --compile --target=bun-linux-x64 controller/src/watchdog-main.ts --outfile "$tmp/overdeck-watchdog" && "$tmp/overdeck-watchdog" --help >/dev/null && (cd collector && bun test server.test.ts && bun run typecheck) && (cd controller && bun test watchdog.test.ts watchdog-main.test.ts && bun run typecheck)` - Expected: PASS — deploy shell parses; no-network dry-run proves exact reverse-forward, binary rsync, unit rsync argv arrays and exact rendered `ExecStart` lines, so any port/key/forward/path drift fails; Linux x64 standalone compile + local smoke pass; collector health 401/200; strict watchdog config accepts absent/empty restart argv, rejects intervals at/below probe timeout and other invalid values, returns exact `TOKEN_MISSING`/`TOKEN_EMPTY` for absent/empty token files, and starts zero probes for either token fatal; controller/collector targets resolve exact tunnel URLs as observe-only; both 10s abort paths fail hung probes; no remote restart executes; observer dedupe remains green; controller + collector `tsc --noEmit` pass. - [ ] Implement → commit --- ### Task T6: Real notifier sink **Wave:** 2 **Blocks:** — **Blocked by:** T0, T1, T5 **Files:** - Create: `controller/src/notify.ts` — concrete sink + adapters for both existing notifier interfaces - Create: `controller/src/notify.test.ts` - Modify: `controller/src/config.ts` — `[notify]` config section - Modify: `controller/src/config.test.ts` - Modify: `controller/src/index.ts` — construct + inject incident notifier adapter - Modify: `controller/src/index.test.ts` - Modify: `controller/src/server.test.ts` — add required defaulted `notify` value to existing typed `ControllerConfig` literal; T6 makes `ControllerConfig.notify` required - Modify: `controller/src/watchdog-main.ts`, `controller/src/watchdog-main.test.ts` — inject off-laptop webhook adapter **Contract (pin EXACTLY):** - Preserve two distinct existing seams: `IncidentNotifier.notify({severity,incident}): void` from `incidents.ts:15` and `WatchdogNotifier.notify({target,url,error,restartError?}): Promise|void` from `watchdog.ts:23`. `notify.ts` provides one sink plus explicit adapters for each shape; do not redefine or conflate interfaces. - Sink canonical record is EXACTLY `{title:string,detail:string,severity:"info"|"page"|"high",ts:string}`. No `key`, `target`, `url`, or other canonical fields. Source mappings are exact: Incident source mapping (`IncidentNotification {severity,incident}`): | Canonical field | Exact source/derivation | |---|---| | `title` | `incident.key` | | `detail` | `JSON.stringify({affectedJobs: incident.affectedJobs, remediation: incident.remediation, count: incident.count, state: incident.state, autoResolveCondition: incident.autoResolveCondition, cooldownUntil: incident.cooldownUntil})` | | `severity` | notification `severity` unchanged | | `ts` | `incident.lastSeen` | Watchdog source mapping (`WatchdogNotification {target,url,error,restartError?}`): | Canonical field | Exact source/derivation | |---|---| | `title` | `` `watchdog:${notification.target}` `` | | `detail` | `JSON.stringify({url: notification.url, error: notification.error, ...(notification.restartError === undefined ? {} : {restartError: notification.restartError})})` | | `severity` | literal `"page"` | | `ts` | adapter-injected clock: `new Date(now()).toISOString()` | - Laptop incident sink: `journal` writes one canonical JSON line to stdout. `desktop` invokes `notify-send --urgency=critical --app-name=overdeck <detail>` through injectable argv exec, never shell string. Controller config `[notify] mode = "journal"|"desktop"|"both"`, default `"both"`; unknown mode flows through `loadConfigResult()` as degraded + typed config incident, not process abort. `info` remains journal-only; desktop fires only `page|high`. - Debian1 watchdog sink: POST to T5 config's operator-provided HTTPS `webhookUrl` as ntfy-compatible request: UTF-8 body = canonical `detail`; headers `Title: <canonical title>`, `Priority: urgent`, `Tags: warning`; timeout exactly 5s via abort signal. Non-2xx, timeout, DNS, and transport failure write one JSON error line to watchdog stderr containing `type:"watchdog-webhook-failed"`, target, and error; never throw into watchdog loop. No journal/desktop fallback is treated as paging. - Incident adapter MUST NOT alter reducer cooldown/dedupe. Watchdog adapter receives already-deduped first failure from T5 and sends exactly one webhook page; observe-only targets have no restart attempt and therefore no invented `restartError`. - Failures of `notify-send` (missing binary, no DISPLAY) are logged to the journal sink and NEVER throw into the incident path. - CLI seam: `bun controller/src/notify.ts '<json>'` validates one exact canonical record and sends it through laptop configured sinks; watchdog entry uses webhook adapter directly. **Behavior:** laptop journal+desktop remain laptop-tier incident sinks. Debian1 webhook is sole off-laptop page path. No warn-tier noise. **Acceptance (one executable check):** - Run: `cd controller && bun test notify.test.ts config.test.ts index.test.ts server.test.ts watchdog-main.test.ts && bun run typecheck` - Expected: PASS — both mapping tables asserted field-for-field with fixed clock; info never desktop-pages; page/high desktop locally; report-fed RC `126` then `127` path runs reducer and maps its page-tier `IncidentNotification` through real laptop sink; watchdog POST uses exact ntfy-compatible request + 5s timeout; POST failure logs and is swallowed; broken `notify-send` is swallowed+journaled; invalid mode starts degraded with one config incident; controller `tsc --noEmit` passes. - [ ] Failing tests first → implement → commit --- ### Task T7: Deck offload action wiring **Wave:** 6 **Blocks:** O5 **Blocked by:** T3 **Files:** - Modify: `packages/deck-ui/src/MachineDetailModal.tsx`, `packages/deck-ui/src/MachineDetailModal.test.tsx` — replace unconditional placeholder disables with typed action callbacks + pending/error rendering - Modify: `packages/deck-ui/src/OffloadControl.tsx`, `packages/deck-ui/src/OffloadControl.test.tsx` — target selected fleet host; replace unconditional placeholder disables with typed action callbacks + pending/error rendering - Modify: `apps/web/src/components/ci/CiContent.tsx` — own selection, confirmation, request state, gateway submission, toast, and refresh - Create: `apps/web/src/components/ci/CiContent.test.tsx` — Vitest + Testing Library jsdom container test with mocked React Query state and `postCollectorAction` - Modify: `apps/web/src/components/inbox/ActionConfirmDialog.tsx` — accept accurate action-specific confirmation copy; preserve existing reap caller - Modify: `apps/web/src/components/design-system/gallery-registry.tsx` — register enabled, pending, error, and no-selected-host variants for both changed deck-ui components - Modify: `apps/web/package.json`, `pnpm-lock.yaml` — declare Testing Library + jsdom test dependencies used by `CiContent.test.tsx` **Contract (pin EXACTLY):** - Follow existing live action seam, not a new client: deck-ui remains presentational and MUST NOT import `fetch` or collector code; `apps/web/src/components/ci/CiContent.tsx` calls existing `apps/web/src/lib/action-client.ts:postCollectorAction`, matching live `reap|ci-rerun` submission in `InboxTriageRow.tsx:runGatewayAction` and CI train submission in `CiContent.tsx:onTrainAction`. T3 remains sole collector→controller proxy and sole idempotency-key minter. - Add typed component props for `onAction(action: OffloadAction)`, `pendingVerb: string | null`, and `actionError: {verb:string;message:string} | null`; `OffloadControl` also receives current selected `FleetHost | null`. Remove unconditional `disabled` + `title="Actions land in wave 4"`. Disable only when no required current target/input exists or any action is pending. Pending action exposes `aria-busy`; errors render an accessible `role="alert"` adjacent to action group. One pending request blocks duplicate submission from both surfaces. - Machine card click selects current `FleetHost` and opens detail modal. Split selection from modal visibility: closing modal MUST retain selected host so `OffloadControl` targets same explicitly selected panel record; removing that host from fresh fleet panel clears selection. `OffloadControl` displays selected hostname and disables host-scoped actions until selection exists. `MachineDetailModal` always targets its `machine.host`. Never default silently to first host. - At CONFIRM time, derive `expectedRevision = String(control.revision)` from current non-stale `offload-control` panel and host/command from current selected `fleet` panel; never cache revision in `OffloadAction`, fixture data, or component-local state. Build exact T3 string args: `box-drain|box-restore` → `{host,expectedRevision}`; `host-quarantine|host-unquarantine` → `{host,command:machine.capability.missingCommand,expectedRevision}` and disable when command absent; `admission-reconcile` → `{expectedRevision}` with NO host; `ci-reconcile|recall-spill` → `{host,expectedRevision}`. Reject any other component-advertised verb locally without POST. - Every enabled offload button opens existing `ActionConfirmDialog` before POST. Confirmation copy names verb label, selected host when scoped, and current revision; it MUST NOT claim reversible drain/restore actions are irreversible. Confirm calls `postCollectorAction(action.verb,{args,requestedBy:"overdeck-web"})`; cancel performs no request. Success clears error, emits success toast, and refetches `collector-state`; failure preserves selection, renders typed component error + danger toast, and permits retry after pending clears. - `CiContent.test.tsx` MUST use the existing deck-ui Vitest + Testing Library render/event harness against `CiContent`, mock only React Query state and the action client, and prove: revision changed after dialog open is read at confirm; modal close retains selected host; every supported verb sends its exact args + `requestedBy`; cancel and unknown verb send no request; success refetches `collector-state`; failure preserves selection, clears pending, renders alert + danger toast, and succeeds on retry without reselection. - Reuse existing deck-ui exports and token classes. Add NO primitive, raw color, fabricated host/revision, or parallel action client. Update gallery variants because changed components gain meaningful interactive states. **Behavior:** `MachineDetailModal` and `OffloadControl` perform real T3-backed transitions. Missing/stale panel data disables honestly; selected host, current revision, confirmation, pending lock, response error, and refresh are visible and deterministic. **Acceptance (one executable check):** - Run: `pnpm --filter @overdeck/deck-ui test && pnpm --filter @overdeck/deck-ui typecheck && pnpm --filter web exec vitest run --environment jsdom src/components/ci/CiContent.test.tsx && pnpm --filter web build && pnpm --filter web typecheck` - Expected: PASS — deck-ui component tests prove enabled click→typed callback, no-target/required-command disable, cross-surface pending lock inputs, `aria-busy`, accessible error, and removal of placeholder tooltip; targeted web container test proves confirm-time revision, retained selection, exact args for every supported verb, cancel/unknown no-request, success refresh, and recoverable failure with alert + danger toast; gallery covers enabled/pending/error/no-target states; no new primitive or action client exists. - [ ] Failing tests first → implement → commit --- ## Post-land operations (NEVER before land; run in order; each verified before the next) **O1 — shared Bearer provisioning:** laptop source of truth is `$HOME/.config/overdeck/token`, load-or-created by collector/controller and consumed under the ownership table above. Verify non-empty + mode 0600. Explicitly copy once to debian1 `$HOME/.config/overdeck/token` over fixed port 2222/key, install mode 0600, delete transfer temp, then compare SHA-256 locally/remotely. No service starts before hashes match; never print token content. **O2 — controller service live (laptop):** update deploy clone externally BEFORE invoking its script, in exact order: `git -C $HOME/.local/share/overdeck/deploy fetch --quiet origin`; `git -C $HOME/.local/share/overdeck/deploy checkout --quiet --detach origin/main`; `bash $HOME/.local/share/overdeck/deploy/packaging/deploy-local.sh`. Reason: running Bash retains old script text after its own `git checkout`; invoking pre-T0 script first would never execute T0's controller-unit installer. External checkout guarantees invoked script is T0 version; its internal fetch/checkout is an idempotent no-op. T0 then installs controller unit before restarting service array. Run `systemctl --user enable overdeck-controller.service`. Verify authenticated health: `curl -sS -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(<"$HOME/.config/overdeck/token")" http://127.0.0.1:8787/health` → `200`; same request without header → `401`; deck offload panels leave `controller-down` within one poll cycle. **O3 — live spine hooks (`~/.claude`, hand-applied):** live Node files execute ONLY deploy-clone CLI `$HOME/.local/share/overdeck/deploy/spine/src/cli.ts` through argv-based `spawnSync` (`shell:false`) with executable `ft`, args `bun <cli> ...`, and child env `FT_FROM_HOOK=1`; NEVER load `.ts` into Node and NEVER use dev-root WIP. Pin spawn timeouts: `parse-config` 2000ms; `report` 1500ms; `host-eligible` 1500ms. **Mandatory pre-hook backup:** BEFORE changing live spine bytes, set `epoch="$(date +%s)"`, set `backup="$HOME/.claude/lib/remote-build.mjs.pre-phase2-$epoch"`, require `test ! -e "$backup"`, copy exact current bytes with `cp -- "$HOME/.claude/lib/remote-build.mjs" "$backup"`, and record `backup` plus `sha256sum "$backup"` in the operational run record. Require source and backup SHA-256 to match before applying hooks. Current live file has uncommitted production changes; NEVER derive this backup from Git and NEVER use `git checkout` for O3 rollback. **Hand-applied file manifest (exact):** - Modify ONLY `~/.claude/lib/remote-build.mjs`: config-load section (`REMOTE_CONFIG_PATH`, `loadRemoteConfig`), report helper section (argv-only CLI runner + strict one-line envelope validation), dispatch section (`chooseHost` eligibility filtering + `tryRemoteBuild` started/finished emission), and helper child-env block (`{...process.env,FT_FROM_HOOK:"1"}`). - Touch NO `~/.claude/bin/local-gate`, `~/.claude/build-remote.json`, service unit, shell profile, or persistent env file. Existing `local-gate` behavior is preserved through `loadRemoteConfig()`'s return/throw contract. `OVERDECK_SPINE_REPORT` and `OVERDECK_SPINE_OBEY` remain operator-supplied environment flags, not persisted by O3. Config hook replaces current `loadRemoteConfig()` invalid-JSON early return at live `remote-build.mjs:22–24`, while preserving every current valid-config behavior at lines 25–32: T2's normalized envelope preserves lines 25–26 host derivation/default/dedup semantics; O3 preserves lines 27–32 override and enabled/empty-host return semantics. Stat `configPath` FIRST. `ENOENT` → best-effort report `{source:"remote-build",stage:"missing",configPath,observedAt}`, perform no preservation, open no incident, return `null` (spine disabled), and allow local execution. Any non-`ENOENT` stat failure is parser-unavailable and MUST throw named `RemoteConfigUnavailableError`, causing `local-gate` to refuse execution. Present file → read raw bytes, invoke `parse-config --json <configPath>`, and accept only exit 0 + one valid success JSON line. Exit 65 preserves exact bytes once as `<configPath>.invalid-<epoch>`, computes `override = process.env.BUILD_REMOTE_LOCAL_FALLBACK === "1"`, then best-effort reports `{source:"remote-build",stage:"invalid",kind:<CLI error>,detail:<CLI detail>,configPath,preservedPath,observedAt,override}`. `override:false` MUST throw named `RemoteConfigInvalidError`, refusing implicit local execution. `override:true` MUST return `null`, executing locally without treating invalid bytes/default config as valid; incident remains open and event records authorized fallback. Spawn error/timeout/non-contract stdout/nonzero other than 65 is parser-unavailable and MUST throw `RemoteConfigUnavailableError`. Valid parse uses envelope's exact normalized `config`; NEVER rebuild from raw input or drop parser defaults. Compute `override = process.env.BUILD_REMOTE_LOCAL_FALLBACK === "1"`; when true, set normalized `config.local_fallback = true` before dispatch semantics. Compute `disabled = config.enabled === false || config.hosts.length === 0`. Best-effort report `{source:"remote-build",stage:"valid",configPath,observedAt,disabled,override}` for EVERY valid load. `disabled:true` returns `null`, preserving current spine-off/local-allowed behavior for `enabled:false` OR empty hosts. `disabled:false` returns normalized config. Valid `override:true` therefore remains honored even when source config says `local_fallback:false`, and its valid lifecycle event records override. Any valid report auto-resolves matching open incident. This O3 hook—not T2 parser—closes R1. Job report fields derive EXACTLY from live `remote-build.mjs` values: | Body field | Exact O3 derivation | |---|---| | `source` | literal `"remote-build"` | | `host` | selected `cfg.host` | | `key` | `tryRemoteBuild({key})` argument | | `mirror` | EXACT local `mirror = mirrorName(projectRoot(cwd))` from live `remote-build.mjs:36/438`; NEVER `mirrorPath` or `meta.json.mirror` (both are remote paths) | | `repo` | same canonical local `mirror` value; deterministic basename + 12-hex project-root hash | | `snapshot` | `String(epoch)`: new jobs use exact `epoch = syncPush(...)` return at line 456; attached `RUNNING` jobs use exact `epoch = JSON.parse(meta.stdout).epoch` at lines 453–454 | | `argv` | `[...argv]` from original `tryRemoteBuild` argument; capture before `remoteArgv(argv, root, mirrorPath)` normalization | | `attempt` | producer seed `1` for each live-spine report; T1 ignores this seed for terminal→started and persists/emits `previous attempt + 1`, making controller ledger authoritative across re-runs | | `stage` | literal `"started"` after new `startJob` succeeds or attached state is confirmed `RUNNING`; literal `"finished"` after `watchJob` returns numeric `status` | | `rc` | omit for `started`; `status` from `watchJob` for `finished` | | `startedAt` | canonical source for fresh-start and reattach paths is exact `started_at` persisted by `startJob` in remote job `meta.json` (`remote-build.mjs:398–401`); after fresh start succeeds or attached `RUNNING` is confirmed, O3 reads `meta.json` (`remote-build.mjs:453`) and uses that same value for started and finished reports; NEVER use hook clock for `startedAt`; this preserves T1 byte-exact duplicate equality across reconnects | | `finishedAt` | omit for `started`; one `new Date().toISOString()` captured immediately when `watchJob` returns for `finished` | | `timeoutSec` | `remoteJobTimeoutSec(cfg)`, same value used by `RuntimeMaxSec` at line 401 | Hook never supplies `id` or `infraFailure`. Job start/finish best-effort invoke `report --json` with these exact T1 bodies. Admission/breaker command identity is `basename(String(argv[0]))` from original `tryRemoteBuild` argv, before remote normalization; use same non-empty string for every `host-eligible --json <host> <command>` candidate check and later `CapabilityBreakerRecord(host,command)` scope. Host selection invokes eligibility before placement. Report spawn/non-contract failure is ignored; eligibility spawn/non-contract failure becomes `{eligible:true,reason:"hook-client-unavailable"}`. `OVERDECK_SPINE_REPORT=1` may be enabled first. MUST keep `OVERDECK_SPINE_OBEY` unset until one explicit finished enrollment report for EACH of `debian1` and `debian2` makes both visible but ineligible, then one global `admission-reconcile` probes and promotes the fleet. Verify each host through its `/status.hosts` entry: `state:"available"`, `enrolling` absent/false, and capability probe green before setting `OVERDECK_SPINE_OBEY=1`. **Executable O3 verification (operator runs post-land, after hand edit):** All `events.jsonl` assertions poll for up to 5s because O3 validates the live projection boundary. ```bash set -euo pipefail token_file="${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/token" auth="Authorization: Bearer $(<"$token_file")" tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT events_file="${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/controller/events.jsonl" wait_for_events() { local minimum="$1" local filter="$2" shift 2 local attempt count for attempt in $(seq 0 20); do if count="$(jq -c "$@" "$filter" "$events_file" 2>/dev/null | wc -l)"; then test "$count" -ge "$minimum" && return 0 fi test "$attempt" -eq 20 && return 1 sleep 0.25 done } export OVERDECK_SPINE_REPORT=1 unset OVERDECK_SPINE_OBEY BUILD_REMOTE_LOCAL_FALLBACK # invalid: refuse local, preserve byte-identical file, open incident printf '{"enabled":true,"hosts":["debian1"],BROKEN' >"$tmp/invalid.json" cp "$tmp/invalid.json" "$tmp/original.invalid" if BUILD_REMOTE_CONFIG="$tmp/invalid.json" ~/.claude/bin/local-gate --key o3-invalid -- /usr/bin/true; then exit 1; fi preserved="$(find "$tmp" -maxdepth 1 -name 'invalid.json.invalid-*' -print -quit)" test -n "$preserved" && cmp "$tmp/original.invalid" "$preserved" INCIDENT_DB="${OVERDECK_CONFIG_DIR:-$HOME/.config/overdeck}/controller/state.sqlite" INCIDENT_KEY="spine-config:$tmp/invalid.json" bun -e 'import {Database} from "bun:sqlite"; const db=new Database(process.env.INCIDENT_DB,{readonly:true}); const row=db.query("SELECT state FROM incidents WHERE key = ?").get(process.env.INCIDENT_KEY); if(row?.state!=="open") process.exit(1)' # invalid override: local allowed, bytes still invalid, override event recorded printf '{"enabled":true,"hosts":["debian1"],BROKEN' >"$tmp/invalid.json" BUILD_REMOTE_CONFIG="$tmp/invalid.json" BUILD_REMOTE_LOCAL_FALLBACK=1 ~/.claude/bin/local-gate --key o3-invalid-override -- /usr/bin/true wait_for_events 1 'select(.stage=="config-invalid" and (.reason|startswith("authorized-local-fallback:")))' # missing: local allowed, no preservation, no incident mutation missing="$tmp/missing.json" BUILD_REMOTE_CONFIG="$missing" ~/.claude/bin/local-gate --key o3-missing -- /usr/bin/true test ! -e "$missing" && ! find "$tmp" -maxdepth 1 -name 'missing.json.invalid-*' -print -quit | grep -q . wait_for_events 1 'select(.stage=="config-missing" and .reason=="config-absent")' # valid disabled: enabled:false OR empty hosts each keeps spine off/local allowed printf '{"enabled":false,"hosts":["debian1"],"local_fallback":false}' >"$tmp/disabled-flag.json" BUILD_REMOTE_CONFIG="$tmp/disabled-flag.json" ~/.claude/bin/local-gate --key o3-disabled-flag -- /usr/bin/true printf '{"enabled":true,"hosts":[],"local_fallback":false}' >"$tmp/disabled-hosts.json" BUILD_REMOTE_CONFIG="$tmp/disabled-hosts.json" ~/.claude/bin/local-gate --key o3-disabled-hosts -- /usr/bin/true wait_for_events 2 'select(.stage=="config-valid" and .reason=="config-valid:disabled:default")' # valid enabled override: local_fallback:false becomes true and override is recorded printf '{"enabled":true,"hosts":["127.0.0.1"],"port":1,"connect_timeout_sec":1,"local_fallback":false}' >"$tmp/valid-override.json" BUILD_REMOTE_CONFIG="$tmp/valid-override.json" BUILD_REMOTE_LOCAL_FALLBACK=1 ~/.claude/bin/local-gate --key o3-valid-override -- /usr/bin/true wait_for_events 1 'select(.stage=="config-valid" and .reason=="config-valid:enabled:override")' # lifecycle: observe running, then terminal, with started/finished ledger events lifecycle_key="o3-lifecycle" lifecycle_root="$(git rev-parse --show-toplevel 2>/dev/null || pwd -P)" lifecycle_mirror="$(ROOT="$lifecycle_root" bun -e 'import {createHash} from "node:crypto"; import {basename} from "node:path"; const root=process.env.ROOT; process.stdout.write(`${basename(root)}-${createHash("sha256").update(root).digest("hex").slice(0,12)}`)')" job="$(KEY="$lifecycle_key" MIRROR="$lifecycle_mirror" bun -e 'import {createHash} from "node:crypto"; process.stdout.write(createHash("sha256").update(process.env.KEY+process.env.MIRROR,"utf8").digest("hex"))')" BUILD_REMOTE_CONFIG="$HOME/.claude/build-remote.json" ~/.claude/bin/local-gate --key "$lifecycle_key" -- /bin/bash -lc 'sleep 10; true' & gate_pid=$! running_seen=0 for _ in $(seq 1 20); do if curl -fsS -H "$auth" http://127.0.0.1:8787/status | jq -e --arg id "$job" '.jobs[] | select(.id==$id and .stage=="running")' >/dev/null; then running_seen=1; break; fi sleep 1 done test "$running_seen" = 1; wait "$gate_pid" curl -fsS -H "$auth" http://127.0.0.1:8787/status | jq -e --arg id "$job" '.jobs[] | select(.id==$id and (.stage=="succeeded" or .stage=="failed"))' wait_for_events 1 'select(.job==$id and .stage=="started")' --arg id "$job" wait_for_events 1 'select(.job==$id and .stage=="finished")' --arg id "$job" # deterministic fresh-fleet enrollment: one report per host, independent of build placement spine_cli="$HOME/.local/share/overdeck/deploy/spine/src/cli.ts" for host in debian1 debian2; do ts="$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)" jq -nc --arg host "$host" --arg ts "$ts" '{source:"remote-build",host:$host,key:("o3-enroll-"+$host),mirror:"o3-enrollment",repo:"o3-enrollment",snapshot:"0",argv:["true"],attempt:1,stage:"finished",rc:0,startedAt:$ts,finishedAt:$ts,timeoutSec:60}' | FT_FROM_HOOK=1 ft bun "$spine_cli" report --json | jq -e '.ok==true' done # admission: each report-enrolled host denied until one global probe/reconcile pass for host in debian1 debian2; do curl -fsS -H "$auth" "http://127.0.0.1:8787/admission/eligible?host=$host&command=bun" | jq -e '.eligible==false' done rev="$(curl -fsS -H "$auth" http://127.0.0.1:8787/status | jq -r .revision)" curl -fsS -H "$auth" -H 'content-type: application/json' -X POST "http://127.0.0.1:8787/transition/admission-reconcile" --data "{\"expectedRevision\":$rev,\"idempotencyKey\":\"o3-admit-fleet-$rev\",\"args\":{\"reason\":\"post-report enrollment\"}}" | jq -e '.result.reconciled==true' for host in debian1 debian2; do curl -fsS -H "$auth" http://127.0.0.1:8787/status | jq -e --arg h "$host" '.hosts[$h] | .state=="available" and (.enrolling // false)==false and (.capability.probes[] | select(.name=="capability") | .ok==true)' done export OVERDECK_SPINE_OBEY=1 for host in debian1 debian2; do curl -fsS -H "$auth" "http://127.0.0.1:8787/admission/eligible?host=$host&command=bun" | jq -e '.eligible==true'; done ``` Global `admission-reconcile` above is pinned fleet probe trigger; concrete prober concurrently runs exact `ssh -p 2222 -i ~/.ssh/id_ed25519_buildbox <host> true` with 5s per-host timeout and ≤8s whole-reconcile deadline once for each enrolling-maintenance host. Per-host `/status.hosts[$host]` assertions intentionally fail unless each report enrollment, probe state write, and promotion occurred. Run fault scenarios SEQUENTIALLY on live `/home/user/.claude/build-remote.json` fleet, exactly `debian1` + `debian2`; NEVER require or invent third builder: 1. **Breaker-blocked host:** first require both hosts eligible for chosen non-empty command. Feed `debian1` two distinct accepted finished reports for that exact command with RC `126`, then `127`; require breaker `open` and `hostEligible(debian1,command)` false. Keep `debian2` eligible, run one real keyed build for same command with `OVERDECK_SPINE_OBEY=1`, and require resulting job record names `host:"debian2"` and reaches terminal state. Reset BEFORE next scenario: POST pinned `host-unquarantine` with `host:"debian1"`, exact command, fresh `expectedRevision`, and unique idempotency key; require breaker `half-open` and advisory eligibility true. Feed one new accepted finished RC `0` report for `debian1` + command; require breaker `closed`, exact `capability-restored` event, and eligibility true. An RC `0` reported while still open MUST have left it open. 2. **Unavailable host:** re-require both hosts eligible. POST pinned `box-drain` for `debian1` with fresh `expectedRevision` + unique idempotency key; require `debian1` unavailable/ineligible while `debian2` remains eligible. Run one new real keyed build and require resulting job record names `host:"debian2"` and reaches terminal state. Reset with pinned `box-restore` for `debian1` using fresh revision/key; require state `available` and eligibility true. After each scenario require `host_slot_reservations` count remains zero and no state from first scenario leaks into second. Stop controller once; reporting/admission MUST fail open and per-box `buildslot.sh` build MUST continue. Then restart it explicitly (`systemctl --user start overdeck-controller.service` — T0's graceful exit 0 means `Restart=on-failure` does NOT auto-restart a stopped unit) and gate on authenticated health before proceeding: `curl -fsS -H "$auth" -o /dev/null http://127.0.0.1:8787/health`. Missing config MUST fail open locally; present invalid config MUST fail closed except recorded override. **Rollback:** `unset OVERDECK_SPINE_REPORT OVERDECK_SPINE_OBEY`; copy recorded `remote-build.mjs.pre-phase2-<epoch>` backup over `$HOME/.claude/lib/remote-build.mjs`, then require restored file SHA-256 to equal recorded backup SHA-256. NEVER use `git checkout`: live pre-hook file contains uncommitted production changes. Retain backup until hash verification passes. Restart no service: hook is loaded by each new `local-gate` process. **O4 — watchdog on debian1:** operator supplies one HTTPS ntfy topic URL as `OVERDECK_WATCHDOG_WEBHOOK_URL` (never committed/hardcoded). Validate scheme, then provision debian1 `$HOME/.config/overdeck/watchdog.json` mode 0600 with T5 exact tunnel URLs, token path, `webhookUrl`, and absent/empty restart argv; transfer JSON over SSH stdin, never URL argv/log output. Run `bash $HOME/.local/share/overdeck/deploy/packaging/deploy-watchdog.sh` (deploy clone copy, post-O2 so clone carries it) after O1. Verify laptop tunnel + debian1 watchdog units `active`; authenticated controller `/heartbeat` through debian1 `127.0.0.1:18787` and collector `/health` through `127.0.0.1:18138` succeed. Stop tunnel briefly → zero restart execution + exactly one received ntfy page per target (controller + collector); restore tunnel → both dedupe keys clear. A logged webhook failure is acceptance failure, not a page. **O5 — end-to-end acceptance (blocked by T7):** deck `/ci` + offload panels show live cluster queue + remote jobs. At run time, operator selects any `<host>` key whose current authenticated `/status.hosts[<host>].state === "available"`; substitute that exact value in following actions. Use T7-enabled deck controls only: confirm `box-drain` on `<host>` from deck, observe typed pending/success state, and require spine stops dispatching there because advisory `hostEligible` is false; then confirm `box-restore` from deck and require host returns available/eligible. Record selected host + both from-deck outcomes in plan doc. ## Documented deviations - Reality-ingest routes vs transition-only mutation rule (`docs/specs/2026-07-18-offload-control-plane-spec.md:33-48`, `spec:33-48`): authenticated `POST /jobs/report` and `POST /spine/config/report` are REALITY-INGEST endpoints recording observed external job, breaker, and config state. They are not operator mutations. Spec transition-only rule governs operator-initiated desired-state changes; those still MUST use typed `/transition/:verb` API. - Staged-CAS publication required by `docs/specs/2026-07-18-offload-control-plane-spec.md:73` (`spec:73`) is an explicit Phase-2 deviation. Phase-2 leaves live publication path byte-identical to today: O3's manifest deliberately excludes `syncPull` and its publication call, so this phase introduces no new publication-path risk. Residual risk remains plain: live `remote-build.mjs:348/470` rsyncs remote tree over live checkout on pull. Follow-up: publication-CAS wiring rides with dispatch-migration phase and consumes already-landed controller workspace-CAS seam in `controller/src/workspace.ts`. - Full per-repo/command MANIFEST verification at admission required by `docs/specs/2026-07-18-offload-control-plane-spec.md:79` (`spec:79`) and `docs/plans/2026-07-18-offload-robustness.md:140` (`robustness:140`) is an explicit Phase-2 deviation. Deferred with controller-owned dispatch for same reason as atomic reservation: live spine owns placement; controller lacks an authoritative dispatch request carrying selected repo/command manifest. Phase-2 depth is connectivity enrollment probe plus report-fed per-host/command breakers from real RC `126`/`127` outcomes. `ssh ... true` MUST NOT be represented as full MANIFEST verification. ## Out of scope (explicit) - Migrating `local-gate`/`remote-build.mjs`/`buildslot.sh` INTO the overdeck repo (full spine migration) — deliberate future phase; Phase-2 only hooks them. - Controller-owned dispatch (scheduler replacing the live spine's own admission) — the spine still dispatches; the controller observes + gates eligibility only. - Spec atomic reservation on placement (`tryReserveHostSlot`) applies to controller-owned placement and is deferred with dispatch migration. Phase-2 `hostEligible` is advisory + fail-open and creates no reservation; oversubscription is prevented by each box's machine-global FIFO `buildslot.sh` admission queue. - Full mobile apps, provider SDKs, routing/escalation, and multi-channel integrations — out of scope. Single debian1 HTTPS ntfy webhook POST required by R7 is in scope. - dynwf / mega-plan-harness follow-ups. ## Decision-enumeration - No user-gated decisions: landing/deploy facts are frozen (merge-to-main, ship.sh, landed checkout); O-steps are operational and fail-open/fail-closed as specified. Irreversible ops: none (worst case = disable flags + stop services, fully reversible).