# Build/Offload Control-Plane Spec

Audience: AI coding agents first. This is the **source of truth** for the laptop→cluster build-offload control plane and the wire contract Overdeck consumes. Implementation plans (`docs/plans/2026-07-18-offload-robustness.md` (co-located in this repo), tasks R1–R7) **derive from this spec**; when a plan and this spec disagree, **this spec wins**. Optimize edits for activation, not prose.

## Purpose

**One desired-state controller owns all offload state. Every mutation goes through its typed API. Nothing else edits config, flock files, GitHub labels, or runner services.** No policy in operator memory. This kills the failure classes from the 2026-07-18 incident and the `gpt-5.6-sol` review (`/home/user/codex-build-ci-offload-review.md`): scattered state with no single owner.

## Boundary (in / out)

- **IN** (owned here, lives in `~/.claude/{bin,lib}`): the controller, its state store, the transition API, the status+metrics endpoint, the event stream, admission + scheduler policy, capability admission, per-job workspace + artifact-publication rules.
- **OUT** (owned elsewhere): Overdeck UI/panels (consumer — `~/Projects/overdeck`); build-execution mechanics inside `lib/remote-build.mjs`; host metric collection (node_exporter — reused, not owned).
- **Overdeck NEVER mutates offload state directly** and **NEVER** reads `local-gate.log`, flock files, or `build-remote.json` as authoritative. It reads `GET /status` + metrics and acts via the transition API only.

## Domain model

- **Host state** (per build host): `available | draining | maintenance | restoring | degraded`.
- **Cluster desired mode**: the operator-declared intent (e.g. remote-only) the reconcile loop drives toward.
- **Revision**: monotonic integer, bumped on every committed transition. Every mutating call carries the caller's `expectedRevision`.
- **Fallback lease**: `{host?, granted, expires}` — permission for laptop-local fallback. **ALWAYS expiring; there is no permanent local-fallback.**
- **Capacity reservation**: per-host slot count, atomically reserved at dispatch.
- **CI intent**: per-runner `{labelled, service_state}` (the controller owns runner label + service state).
- **Job** (ledger): `{id = sha256(key+mirror), repo, host, snapshot, attempt, stage, workspace, publication}`. Its `/status` wire projection is `RemoteJob` (see Status endpoint); the wire field names `snapshot` (submitted-tree digest) and `publication` (staging→promote lifecycle) are **normative for the consumer** — **NOT** `snapshot_digest` / `output_manifest`.

## State store

Transactional store (SQLite WAL) holding desired + observed state, `revision`, fallback lease, capacity reservations, CI intent, and the job ledger.

- **Crash-resumable**: on any restart (including laptop reboot) the reconcile loop re-derives actions from desired-vs-observed.
- Ledger writes are **atomic** — no unlocked read-modify-write.
- **Ticket/job identity = PID+starttime / systemd unit. NEVER `lsof` or PID-alone** for correctness (PID reuse and fd-inheritance both misattribute; the unproven "flock held with no fd owner" ghost model is sidestepped by starttime identity).

## Transition API — the ONLY way state changes

Verbs: `box-drain`, `box-restore`, `host-quarantine`, `host-unquarantine`, `admission-reconcile`, `job-retry` (typed infra failure only), `ci-reconcile`, `recall-spill`.

**Wire contract (pin EXACTLY — the consumer proxies to this):**

- Endpoint: `POST /transition/:verb` on the controller (127.0.0.1, `Authorization: Bearer <token>`).
- Request body (JSON): `{expectedRevision: number, idempotencyKey: string, args: {...verb-specific}}`.
- Response: **200** `{revision: number, result: {...}}` on commit · **409** `{error:"stale-revision", currentRevision: number}` on stale · **422** `{error:"invalid-args", detail}` on schema/guard failure · **403** `{error:"audit-write-failed"}` when the pre-execution journal write fails.
- **Precedence (fixed order):** idempotency replay is checked **BEFORE** revision. A replayed `idempotencyKey` returns its prior committed result **even if `revision` has since advanced** — replay is never re-classified as stale.

Guards per call:

- Stale `expectedRevision` → **409, no-op** (never a partial apply).
- Replayed `idempotencyKey` → return the prior result, **no re-execute**.
- **Fail-closed audit**: intent is journaled **before** execution; audit-write failure → transition **REFUSED** (403 above).

**Consumer-arg constraint (normative):** the Overdeck action gateway carries `args` as `Record<string,string>` (string values only — an offload item advertises e.g. `{host:"debian1", command:"playwright browsers", expectedRevision:"4192"}`). Therefore `expectedRevision` crosses the deck→controller hop as a **string** and the proxy layer MUST coerce it to `number` before calling `/transition/:verb`. The controller endpoint itself takes numeric `expectedRevision`.

**`idempotencyKey` ownership (normative):** the offload adapter advertises action `args` WITHOUT an idempotency key (it carries only `expectedRevision` + verb targets — see the adapter today). The **deck action gateway MINTS the `idempotencyKey` (UUID v4) at invocation time**, once per user-initiated action, and puts it in the `/transition/:verb` POST body. The controller PERSISTS `(idempotencyKey → committed result)` in its SQLite store for a bounded **24 h** retention window; a replay inside the window returns the stored result, a key past retention is treated as new. The adapter/UI MUST NOT reuse a key across distinct user actions.

**Per-verb `args` schema (values arrive as strings at the deck seam; the controller coerces numerics).** `result` is the verb-specific commit payload echoed in the 200 body.

| Verb | `args` | Effect |
|------|--------|--------|
| `box-drain` | `{host, expectedRevision}` | host `available → draining` |
| `box-restore` | `{host, expectedRevision}` | host `draining\|maintenance\|degraded → restoring → available` (guards below) |
| `host-quarantine` | `{host, command, expectedRevision}` | open circuit for `command` on `host`; quarantine that capability class |
| `host-unquarantine` | `{host, command, expectedRevision}` | half-open re-probe of `command` on `host` |
| `admission-reconcile` | `{expectedRevision, reason?}` | re-run global admission/dispatch pass (clears wedge) |
| `job-retry` | `{jobId, expectedRevision}` | retry ONE job on typed infra failure only |
| `ci-reconcile` | `{expectedRevision, host?}` | drive runner label + service state to intent |
| `recall-spill` | `{expectedRevision, host?}` | drain stale laptop spill lease |

**Invariants (MUST hold):**

- `draining → maintenance` **ONLY** when that host's remote jobs **and** CI jobs are idle.
- `restoring → available` requires capability probe green **and** storage **and** runner **and** offload health all green.
- Fallback lease is **always expiring**.
- Config parse/schema failure → `degraded` + one typed incident; **NEVER** silent local fail-open. A valid `enabled:false` is a legitimate disabled state; corruption is not. Explicit `BUILD_REMOTE_LOCAL_FALLBACK=1` operator override is honored but recorded as an event.
- **Artifact publication (highest-risk defect):** a remote job writes **ONLY** a declared output manifest into a job staging dir. Before atomic promotion into the live checkout, verify (a) the submitted-snapshot digest still matches **and** (b) local-checkout generation is unchanged (snapshot CAS). Mismatch → do **NOT** promote, preserve staging, emit `artifact-publication-blocked` (severity act). **NEVER rsync a remote working tree over live source.**

## Scheduler / admission

- **ONE global FIFO queue** across all build hosts. Dispatch each build to the **least-loaded eligible** host (`available ∧ capability-passed ∧ not-quarantined ∧ capacity-reserved`). Atomic capacity reservation on placement.
- **Spill rule (EXACT):** the laptop runs a build **ONLY** when ALL builders are overloaded **AND** queue length > builder count. Spill uses an expiring lease; when a builder frees, in-flight spill is not force-killed but no new spill is admitted; `recall-spill` drains stale spill.
- **Capability admission:** per repo/command toolchain manifest (declared, versioned). At admission verify the target host satisfies it — command presence + version + writable paths + disk + systemd capability — **not** just SSH + runner-hash. Repeated `126/127` for a command+host → circuit-breaker `open` → quarantine that class + emit `capability-missing`; half-open re-probe restores. Manual package installs are drift, a first-class incident.
- Host join: `debian2..N` enroll via the controller (capability probe → eligible). **Fleet size is dynamic — no hardcoded host list anywhere.**

## Status + metrics endpoint — the seam Overdeck reads

Bind **127.0.0.1 only**. Auth: **`Authorization: Bearer <token>`** on every request (the Overdeck collector token convention). Unix-socket is a **non-goal for v1** (earlier ambiguity resolved: Bearer).

**Token provisioning is fail-fast (normative):** the controller MUST require Bearer and return **401** on a missing/empty token. The adapter MUST be configured with a non-empty token and MUST send the header on **every** request. Today the adapter sends the header only when the token is truthy and the collector's default loader yields `""` when the file is absent (silent no-auth) — the build MUST make an absent/empty offload token a **startup error**, not a silently unauthenticated poll. Consumer gate.

### `GET /status` → `ControllerStatus` (complete snapshot; field names are normative — the adapter deserializes these exact keys)

```
HostState = "available" | "draining" | "maintenance" | "restoring" | "degraded"
hostname  = string   // the `hosts` map key; also the type of RemoteJob.host

ControllerStatus = {
  desired: HostState, observed: HostState, revision: number,
  lease:      { active: boolean, expired?: boolean, host?: string|null, expiresAt?: string|null, reason?: string|null },
  capacity:   { builders: number, idleSlots: number, overloaded?: boolean },
  reconciler: { healthy: boolean, lastAt?: string },
  dispatch:   { state: "healthy"|"wedged"|"paused", detail?: string, host?: string },
  hosts:      Record<hostname, HostStatus>,   // a MAP keyed by host, NOT an array
  queue:      QueueStatus,
  jobs:       RemoteJob[],
}
HostStatus  = { state: HostState, role: "builder"|"workstation", primary?: boolean, enrolling?: boolean,
                capability?: { probes: CapabilityProbe[], circuitOpen?: boolean, missingCommand?: string } }
QueueStatus = { depth: number, oldestAgeSeconds: number, p95AgeSeconds: number, sloSeconds: number,
                dispatchTarget?: string, spill?: { active: boolean, stale?: boolean },
                tickets: QueueTicket[], wedge?: { host: string, detail: string } }
QueueTicket = { position: number, key: string, repo: string,
                owner: { pid: number, starttime: number, label?: string },   // PID+starttime identity, matches State-store rule
                enqueueAgeSeconds: number, state: string, dispatchTarget?: string }
RemoteJob   = { id: string, repo: string, snapshot: string, stage: string, host: hostname, rc?: number|null,
                pullBytes?: number|null, pullDurationSeconds?: number|null, publication?: PublicationState }
PublicationState = { state: "none"|"staged"|"promoted"|"blocked"|"discarded", reason?: string }
```

**Runtime validation (normative, consumer-hardening):** the adapter MUST **schema-validate** the `/status` body against `ControllerStatus` (zod parse), NOT cast it. A **reachable** `/status` that is valid JSON but the **wrong shape** MUST **throw** (poll rejects → last-good retained) — it MUST NOT leak partial/undefined fields into panels. Today the adapter casts (`requestJson<T>` returns `res.json() as T`, no parse), so only non-JSON throws; closing this is a consumer gate.

### Metrics — Prometheus **query API**, NOT an exporter scrape

The adapter calls **`GET {metricsUrl}/api/v1/query?query=<promql>`** (default `metricsUrl` = `http://127.0.0.1:8787`, same origin as `/status`) and expects the standard Prometheus vector JSON (`{status, data:{resultType:"vector", result:[{metric,value:[ts,val]}]}}`). It does **NOT** scrape a `/metrics` text endpoint. The controller MUST expose (or front with a Prometheus that exposes) these **exact** series:

- Cluster: `build_offload_queue_depth`, `build_offload_queue_oldest_age_seconds`, `build_offload_queue_age_p95_seconds`, `build_offload_fallback_lease_expired`, `build_offload_artifact_cas_mismatch_total`.
- Capability: `build_offload_exit_total{host!="",command!="",code=~"126|127"}` (labels `host`, `command`, `code`).
- Per-host, label `host` only: `build_offload_host_running_jobs`, `build_offload_host_slots_free`, `build_offload_host_slots_total`, `build_offload_host_dispatch_accept`, `build_offload_host_load`, `build_offload_host_cores`, `build_offload_host_builds_24h`, `build_offload_host_mem_used_bytes`, `build_offload_host_mem_total_bytes`, `build_offload_host_swap_used_bytes`, `build_offload_host_swap_total_bytes`, `build_offload_host_net_rx_bytes_per_second`, `build_offload_host_net_tx_bytes_per_second`.
- Per-host, **extra label required** (the consumer keys on it — omitting it drops the sample):
  - `build_offload_host_disk_free_bytes{host,mountpoint}` and `build_offload_host_disk_size_bytes{host,mountpoint}` — **one series per (host, mountpoint)**; `mountpoint` is MANDATORY.
  - `build_offload_host_core_load_percent{host,core}` — one series per (host, core); `core` is the integer core index (`"0"`, `"1"`, …).
  - `build_offload_host_cpu_temp_celsius{host,sensor}` — `sensor` ∈ `"pkg" | "max" | "crit"`.

Host machine series (load/mem/swap/net/disk/temp) originate in **node_exporter**, but the adapter queries them under the `build_offload_host_*` namespace — the controller (or a recording-rule / relabel layer it owns) MUST republish them under those exact names **with the labels above**. The consumer **never** scrapes node_exporter directly.

**Consumer disk-free multi-mount defect (gate):** the adapter's `disk_free` path collapses samples to one value per host before re-attaching the *first* mount name, so on a host with >1 mount the free-bytes/size pairing is wrong. The producer MUST still emit one series per (host, mountpoint); the **consumer** fix (iterate raw per-mount samples for `disk_free`, as it already does for `disk_size`) is an Overdeck-side gate.

## Event stream (producer-internal — NOT a deck wire contract)

Every state transition + job-lifecycle step emits a typed event. **The Overdeck deck does NOT consume this stream** — it reads `/status` + metrics only; the event stream is the controller's own monitoring/audit source of truth. **Events are the monitoring source of truth; `local-gate.log` is diagnostic payload only, never authoritative.**

- Transport: **append-only newline-delimited JSON** to the controller's event log (the SQLite store's event table is authoritative; the JSONL file is its projection). No SSE/WebSocket obligation in v1 (add only if a consumer needs live push).
- Event shape (field types pinned): `{ts: string(ISO), job: string, repo: string, host: hostname, snapshot: string, attempt: number, stage: string, reason: string, rc: number|null, durationSeconds: number}`.
- Replay/ordering: events are strictly append-only and ordered by `revision`; consumers (if any are added later) replay from a `revision` cursor.

## Overdeck consumption contract (normative for the consumer)

- The offload adapter polls `GET /status` + the metric queries; `poll()` returns a **complete snapshot**.
- Controller **unreachable** (transport error) → the adapter **MODELS controller-down**: one `offload:controller-down` build item (`severity:"act"`, `kind:"build"`, `actions: []`) + all four panels emitted with data **exactly `{stale: true}`** — no last-good payload is retained; the healthy poll always overwrites with fresh projections. **This is NOT a thrown error.** Detect the transport reject **structurally** (typed error tag) — **never** match error-message substrings: the collector runs under Bun, whose connect error (`"Unable to connect…"`, code `ConnectionRefused`) differs from Node's (`ECONNREFUSED`, `fetch failed`).
- Controller **reachable but** non-2xx/malformed status, **or any metric query failing** → **throw** (poll rejects; the collector keeps the prior good snapshot). Do **not** model as controller-down. (A parse-failed `/status` and a transport-failed `/api/v1/query` both reject — distinct from the unreachable-controller path above.)
- **Panels (exactly 4):** `offload-control`, `cluster-queue`, `remote-jobs`, `fleet`. **Fleet membership = the keys of `ControllerStatus.hosts`** (the `/status` map); per-host metrics are joined in by `host` label from the series. Membership is NOT derived from the Prometheus series. Still **no hardcoded hosts** — the set is whatever `status.hosts` reports. Offload Items are `kind:"build"`.
- **Actions:** the design intent is that Overdeck `POST /actions/:verb` proxies the typed transition verbs to `POST /transition/:verb` with `expectedRevision` + idempotency + fail-closed audit; args are validated per verb (host/job/pid must exist in the current panels); argv is an array, **never** shell-concatenated; allowlist only. **NOT YET WIRED** — the collector gateway allowlist today is `reap | ci-rerun | steer | snooze` (offload verbs 404). See Implementation gates.

## Implementation gates — built vs not (as of 2026-07-18)

This spec describes the **target** control plane. What actually exists today is a much smaller SSH-based offloader; the Overdeck consumer (adapter A8) was landed **ahead of** this producer. Every row below is a consumer-ahead-of-producer gap the build MUST close — the deck already reads/advertises these, nothing serves them.

| Seam | Consumer expects | Built? | Closes in |
|------|------------------|--------|-----------|
| `GET /status` → `ControllerStatus` on `127.0.0.1:8787` | full snapshot above | **NO** — nothing binds 8787; adapter shows persistent `controller-down` | R2 (controller) |
| `GET /api/v1/query` (`build_offload_*` series) | Prometheus vector JSON | **NO** — no query endpoint, series unpublished | R2 + R6 |
| `POST /transition/:verb` (8 verbs) | typed transition API, 409/replay | **NO** — no controller to receive it | R2 |
| Deck action gateway → transition verbs | `box-drain … recall-spill` proxied | **NO** — collector allowlist is `reap\|ci-rerun\|steer\|snooze` only; offload verbs 404 | R2 + Overdeck X1 |
| Structured events / metrics source of truth | typed event stream | **NO** — only `local-gate.log` diagnostics exist | R6 |
| Per-job workspace + artifact CAS promotion | staging→verify→atomic promote | **NO** — `syncPull` rsyncs the remote tree over the live checkout (highest-risk defect) | R3 |
| `idempotencyKey` minted by deck gateway + 24 h controller replay store | key in POST body; stored result | **NO** — adapter advertises no key; gateway does not mint one | R2 + Overdeck X1 |
| Adapter zod-validates `/status` + metric vectors (throw on wrong shape) | schema parse, not cast | **NO** — `requestJson` casts; only non-JSON throws | Overdeck consumer-hardening |
| Offload token fail-fast (absent/empty → startup error) | non-empty Bearer every request | **NO** — header skipped when token `""`; default loader returns `""` | Overdeck consumer-hardening |
| `disk_free` per-mount consumption | one series per (host, mountpoint) | **producer OK / consumer buggy** — adapter collapses free-bytes by host then pairs first mount | Overdeck consumer-hardening |

**Rule:** bring R2 up **additively behind a flag** — do not break the live SSH build spine while the controller is introduced. The deck already degrades honestly (`controller-down` + stale panels) until 8787 answers.

## Non-goals

Not a general CI system. Not a metrics store (reuses Prometheus/node_exporter). Does not own build-execution mechanics. Does not render UI.

## References

- Implementation plan (derives from this): `docs/plans/2026-07-18-offload-robustness.md` (co-located in this repo) (R1–R7).
- Origin review: `/home/user/codex-build-ci-offload-review.md` (`gpt-5.6-sol`).
- Consumer spec: `~/Projects/overdeck/docs/specs/2026-07-18-overdeck-app-spec.md` (references this contract).
- Live-system notes: memory `project_remote_build_offload`, `project_overdeck`.