# Replacing the two hand-rolled schedulers with k3s

Audience: AI coding agents first. Status: DESIGN. Task: overdeck #96.

## Why

Laptop pinned ~90-100% CPU while all three buildboxes idle at load 0.6
(`~/.claude/buildbox-hosts.json` all three `state: reachable`). Offload is
structurally not working: two independent placement engines exist, they do
not share state, and one of them (`controller/src/capability.ts`) has been
rejecting hosts for weeks. This spec covers PHASE 1 only — design. No k3s
install happens in this change.

**Checked, not assumed: offload is not simply switched off.**
`~/.claude/build-remote.json` has `"enabled": true` (read directly), so the
laptop-pinned symptom is not explained by a trivially-flipped config flag —
the two-scheduler duplication/disagreement below is the real proximate cause,
not a one-line fix the owner is missing.

## The two schedulers today

### Scheduler A — cpu-guard / local-gate / remote-build.mjs (per-invocation, SSH-driven)

Entry point: every heavy build/test command on the laptop is routed through
`modules/workstation/claude/lib/cpu-guard.sh`. Its heavy-command classifier
(`_cg_heavy`, cpu-guard.sh:165-236) decides whether a command is a build; if
`~/.claude/build-remote.json` has `"enabled": true` it execs
`~/.claude/bin/local-gate` (cpu-guard.sh:243-249), whose logic lives in
`modules/workstation/claude/lib/remote-build.mjs`.

Responsibilities, with citations:

- **Host list**: `registryBuildHosts()` (remote-build.mjs:100-106) reads
  `~/.claude/buildbox-hosts.json` via `resolveHosts(loadRegistry(), {order:
  "build"})` — the same registry this design reads for control-plane
  placement.
- **Eligibility check**: `hostEligibleCli()` (remote-build.mjs:145-160) shells
  out to `~/.local/share/overdeck/deploy/spine/src/cli.ts host-eligible`,
  which calls `GET http://127.0.0.1:8787/admission/eligible` on the overdeck
  controller (`spine/src/admission.ts:5`, `DEFAULT_BASE_URL =
  "http://127.0.0.1:8787"`). **Any failure of this call — timeout, non-zero
  exit, unparseable envelope — is swallowed and returned as `{eligible: true,
  reason: "hook-client-unavailable"}`** (remote-build.mjs:150,154,158). The
  controller's capability verdict is advisory only; a controller that is
  down or wrongly rejecting a host cannot block local-gate.
- **Placement/scoring**: `selectHosts()` (remote-build.mjs:300-321) — its own
  load+thermal scorer, independent of the controller's. Excludes hosts at
  critical package temperature, then sorts by
  `(load + CLAIM_WEIGHT*claims) / cores` plus a thermal-warm penalty, with a
  sticky-preferred tiebreak (`stickyFile`, remote-build.mjs:870) and original
  registry order as final tiebreak. `load`/`claims` come from a local
  telemetry cache built from files under `~/.claude/run/remote-build/`, not
  from the controller's `HostRecord.slotsUsed`.
- **Dispatch**: `chooseHost()` (remote-build.mjs:871-924) — picks the top of
  `selectHosts()`, then `startJob()`/`watchSsh()` (remote-build.mjs:953,975)
  rsync the repo mirror and open an SSH exec session directly to the chosen
  box on port 2222 with the buildbox SSH key.
- **Reporting**: `bestEffortSpineReport()` (remote-build.mjs:133-143) POSTs a
  `report` to the same spine CLI, which forwards to the controller's
  `POST /jobs/report` (`controller/src/server.ts:241`) — again best-effort;
  a failure is logged, never retried, never blocks the job.
- **Local concurrency floor**: if no remote host is chosen (registry
  unreadable, `local-gate` absent, feature disabled, or the job runs on the
  laptop itself for another reason), the job still passes through
  `modules/workstation/claude/lib/buildslot.sh` — a machine-global,
  `/proc/stat`-adaptive admission queue (FIFO ticket file per waiter under
  `/run/user/1000/buildslot`, `slots_max` derived from live interactive CPU
  demand, buildslot.sh:9-14, 220-238). This is NOT a remote placement
  engine — it caps *local* concurrency only, on whichever machine runs it
  (laptop for local fallback jobs, or a buildbox for remote-executed jobs
  under its own local cpu-guard wrapper).

### Scheduler B — overdeck controller (`controller/src/{scheduler,admission-loop,store,capability}.ts`)

Runs as `overdeck-controller.service`
(`packaging/overdeck-controller.service`), a `bun run src/index.ts` process.
**Correction to the task brief: it listens on `127.0.0.1:8787`
(`controller/src/config.ts:7`, `DEFAULT_CONTROLLER_PORT = 8787`), confirmed
live (`ss -tlnp` shows `127.0.0.1:8787` held by the running
`overdeck-controller.service`, PID matches `systemctl --user status`).
Port `31340` belongs to a different service, `botmaster-proxy`
(`packaging/botmaster-proxy.ts:6`, `PORT = ... ?? 31340`) — an unrelated D1
metrics proxy the collector also polls
(`collector/src/adapters/index.ts:40`). Do not conflate the two; the design
below targets 8787.**

Responsibilities, with citations:

- **Ticket queue**: `ClusterScheduler.enqueue()` (`scheduler.ts:47-58`)
  appends a `QueueTicketRecord` (owner pid+starttime, repo, command) to the
  store.
- **Placement**: `ClusterScheduler.reconcile()` (`scheduler.ts:60-88`) —
  reclaims tickets whose owning process died (`reclaimDeadTickets`,
  `isProcessOwnerAlive` checks `/proc/PID/stat` start-time match,
  scheduler.ts:105-112, 166-178), then for each queued ticket in FIFO order
  calls `reserveLeastLoaded()` (scheduler.ts:125-138): filters builders by
  `isEligible()` (state=available, `capabilityOk`, not dispatch-paused, not
  quarantined for that command, not overloaded — scheduler.ts:140-148),
  sorts by `load()` = `(slotsUsed + pendingReservations)/slotsTotal`
  (scheduler.ts:157-163), and reserves a slot on the first host that accepts
  the reservation transactionally (`store.tryReserveHostSlot`). If every
  builder is overloaded and the queue is deeper than the builder count, it
  places a time-boxed spill ticket onto `"laptop"`
  (scheduler.ts:71-84, `tryPlaceSpill`, 5-minute default lease).
- **Capacity/health state**: `HostRecord.state` is one of `available |
  draining | maintenance | restoring | degraded`
  (`controller/src/status.ts:5-10`, `HostStateSchema`). Host rows carry
  `slotsUsed`, `slotsTotal`, `dispatchPaused`, `quarantinedCommands`,
  `capabilityOk`, `capabilityReason` (`store.ts:78-...`, `HostRecord`).
- **Capability admission / circuit breaker**: `CapabilityService.admit()`
  (`capability.ts:210-246`) SSH-probes a host (`ssh -p 2222 -i
  ~/.ssh/id_ed25519_buildbox <host> true`, `createSshCapabilityProber`,
  capability.ts:114-174, 5s timeout) against a `ToolchainManifest`; on
  failure it opens a per-host-per-command circuit breaker after
  `FAILURE_THRESHOLD = 2` (capability.ts:58, 292-303) and sets
  `host.capabilityOk = false`, which `isEligible()` in the scheduler reads
  directly — this is the mechanism the task brief says "has been rejecting
  hosts for weeks." A probe that cannot run (transport failure) is treated
  as `unconfirmed`, not a failure (capability.ts:176-180) — the breaker only
  opens on a real remote-command exit 126/127, not on an unreachable host,
  so a *stuck-open* breaker is evidence of a real repeated probe failure,
  not transient flakiness. (Root-causing that specific stuck-open state is
  out of scope for this design; it is one motivating example, not something
  this migration is required to fix independently — k3s replaces this
  mechanism outright, see below.)
- **Enrollment loop**: `startAdmissionReconcileLoop()`
  (`admission-loop.ts:41-102`) polls every 60s
  (`ADMISSION_LOOP_INTERVAL_MS`) for hosts in `enrolling && state ===
  "maintenance"` and drives an `admission-reconcile` transition so a newly
  added host does not sit unusable until an operator clicks Reconcile
  (comment, admission-loop.ts:36-39, citing a debian3 day-long outage).
- **HTTP surface**: `/health`, `/heartbeat`, `/status`, `/metrics`,
  `/api/v1/query`, `/admission/eligible` (GET), `/jobs/report` (POST),
  `/spine/config/report` (POST), `/workspace/create`, `/workspace/gc`
  (`controller/src/server.ts:166-289`). No HTTP endpoint dispatches a job.
  **Confirmed, not inferred: `ClusterScheduler.enqueue()`
  (`scheduler.ts:47-58`) has zero callers outside `scheduler.test.ts` and
  `incidents.test.ts`** (`grep -rln "scheduler.enqueue" controller/src/*.ts`
  returns only those two test files). `reconcile()` itself *is* called live —
  once at startup (`server.ts:110`/`133`) and after every
  `admission-reconcile` transition (`transitions.ts:524`, driven by
  `admission-loop.ts`'s 60s poll) — but with `queuedTickets()` permanently
  empty in production, `reconcile()`'s only live effect is
  `reconcileTerminalHostReservations()`/`reclaimDeadTickets()` bookkeeping;
  its FIFO/least-loaded/spill-to-laptop placement logic never runs against a
  real build request today.

### Where they disagree

Documented already in
`docs/specs/2026-08-06-cluster-source-of-truth-design.md:35`: *"two health
models; `unreachable-or-full` conflates down with full"*. Concretely:

1. Scheduler A's `selectHosts()` load score and Scheduler B's
   `reserveLeastLoaded()` load score are computed from **disjoint state** —
   A reads its own telemetry cache files, B reads `HostRecord.slotsUsed` and
   its own reservation table. Neither sees the other's in-flight jobs, so
   both can independently pick the same "least loaded" host and
   over-subscribe it.
2. Scheduler A treats controller unavailability/rejection as advisory
   (fail-open to eligible, remote-build.mjs:150/154/158); Scheduler B's
   breaker is the sole source of truth for its own placement and fails
   closed (a quarantined host is filtered out of every future
   `reconcile()`, scheduler.ts:140-148) — with no external channel to
   un-stick it besides `CapabilityService.halfOpen()`, which is not called
   by anything discovered in this pass.
3. **Confirmed (not inferred): only Scheduler A's path drives the reported
   symptom.** `cpu-guard.sh` wires Scheduler A into every real build
   invocation today. Scheduler B's ticket-queue/placement code
   (`enqueue()`/`reserveLeastLoaded()`/spill-to-laptop) has **zero
   production callers** — confirmed by grep, see HTTP surface note above —
   so it is dead weight relative to actual build traffic, exercised only by
   its own test suite. This changes the framing from "two schedulers
   disagree over the same jobs" to: **Scheduler A places every real job
   today; Scheduler B's only live function is host state/health tracking
   (`HostRecord.state`, `capabilityOk`) that Scheduler A partially,
   advisorially, and fail-open-ly consults** via `hostEligibleCli()`. The
   capability breaker (point 2 above) is the one part of B that does affect
   A's traffic, indirectly, through that advisory call — and only when it
   returns non-2xx/malformed, which A also swallows fail-open. Net effect:
   B's placement/queue code (`scheduler.ts`'s `enqueue`/`reconcile`
   ticket-handling, the spill-to-laptop path) can be deleted in Phase 5
   without a live-traffic trace — this is no longer an open item (see below).

## What k3s replaces natively vs. what survives as overdeck code

| Responsibility | Today | k3s native? | Disposition |
|---|---|---|---|
| Track which hosts exist, are healthy, have capacity | `HostRecord` rows, `status.ts` HostState, A's registry read | **Yes** — Node objects, `kubectl get nodes`, node conditions (`Ready`, `MemoryPressure`, `DiskPressure`) | Replace. Kubelet + node-problem-detector report condition; no code to maintain. |
| Least-loaded placement among eligible hosts | `reserveLeastLoaded()` (B), `selectHosts()` (A) | **Yes** — kube-scheduler's default `NodeResourcesFit`/`NodeResourcesBalancedAllocation` scoring | Replace. Delete both. |
| FIFO admission queue when all builders are full | `enqueue()`/`reconcile()` ticket queue (B), buildslot.sh ticket queue (A/local) | **Partially** — the scheduler queues unschedulable Pods natively (`PodScheduled=False`, reason `Unschedulable`); FIFO-with-priority and the specific fairness guarantee buildslot.sh gives (no waiter starved indefinitely under sustained load, `buildslot.sh:4-6`) is not a default k3s behavior. | Kubernetes-native equivalent = **Kueue** (a k8s-native queueing add-on) or plain `PriorityClass` + `ResourceQuota` per namespace. Evaluate Kueue in Phase 2+; do not hand-roll a queue on top of k8s. |
| Capacity ceiling per host (`slotsTotal`) | `slotsTotal` field, `isOverloaded()` | **Yes** — node-level allocatable CPU/memory + Pod resource `requests`/`limits` | Replace. `slotsTotal` maps to `--max-pods` or a ResourceQuota; no bespoke counting. |
| Adaptive local concurrency reacting to *interactive, non-build* CPU demand on a shared desktop/build box (buildslot.sh's `/proc/stat` non-nice-time sampling) | buildslot.sh `effective_slots()` (buildslot.sh:220-238) | **No** — k8s has no concept of "shrink admission because someone is using this box interactively right now"; node conditions react to pressure, not to nice-vs-non-nice CPU split, and have no sub-second adaptive ceiling | **Must survive as overdeck code.** This is the single most load-bearing piece of custom logic in either scheduler and has no k8s equivalent. Options: (a) keep buildslot.sh as-is on any box that is *also* used interactively (the laptop, if it ever hosts a fallback node) and never run it on dedicated buildboxes where there is no competing interactive load; (b) write a small node-exporter-style side-car that publishes a **k8s-native custom node taint/label** (`interactive-load=high`) derived from the same `/proc/stat` non-nice-time calculation, and use it in Pod `nodeAffinity`/`tolerations` so the *k8s* scheduler avoids the loaded node — this is the correct long-term shape, deferred to Phase 2+ since the target buildboxes are dedicated (roles never include "workstation"). |
| Capability admission — does this host actually have the toolchain a job needs (command present, right version, writable paths, disk, systemd)? | `CapabilityService.admit()`, `missingCapabilities()` (capability.ts:387-401) | **Partially** — k8s node *labels/taints* can express "has toolchain X vN", set once at join time or by a labeling DaemonSet, and Pod `nodeSelector`/`nodeAffinity` enforces it at schedule time (no per-job SSH probe). Live per-job re-verification (disk free right now, path writable right now) is not native. | Replace the **admission gate** with node labels set at join/periodic-relabel time (an overdeck-owned DaemonSet or cron Job replaces the SSH-based `createSshCapabilityProber`). The **circuit breaker** (open after 2 consecutive exit 126/127, half-open recovery) has no k8s equivalent and is worth keeping in spirit — but as a small controller that *relabels a Node* (e.g. removes a `capability=X-ok` label) rather than as a separate admission path a Pod scheduler bypasses. |
| Eviction / rescheduling a failed or stuck job | Not really implemented today — `complete()` marks terminal state; no requeue-and-retry logic found in `scheduler.ts` | **Yes** — Job/Pod backoff, `restartPolicy`, `activeDeadlineSeconds`, `backoffLimit` | Replace/gain. This is a capability the hand-rolled schedulers never had. |
| Reclaiming work whose *submitting process* died (not the worker — the client that requested the job) | `isProcessOwnerAlive()` (scheduler.ts:166-178, pid+`/proc/PID/stat` starttime match) deletes tickets whose owner process exited | **No** — a k8s Job outlives its submitter by design; nothing tracks "did the client that created this Job exit" | **Must survive as overdeck code.** Without it, a Job submitted by an agent session that then dies (crash, session end) will run to completion and burn a node for no one. The overdeck-owned watcher (see Job execution model below) needs to own this: track submitter liveness the same way (pid+starttime) and delete/cancel the Job via the k8s API if the submitter is gone before the Job starts. |
| Reentrant/nested build calling another build on the same host (`buildslot.sh`'s `ancestor_holds_slot`) | buildslot.sh ancestor-pid walk lets a sub-build proceed without deadlocking on its own parent's slot | **Yes, for free** — a sub-build launched inside a Job's Pod is contained by that Pod; it is not a second top-level admission request racing the k8s scheduler | Subsumed. No equivalent code needed on k3s; call this out explicitly so Phase 5 does not try to port `ancestor_holds_slot` — it would be solving a problem Pod containment already solves. |
| Sticky host preference (repo mirror already warm on host X) | `stickyFile` (remote-build.mjs:870) | **No native primitive**, but expressible | Keep as a **Pod nodeAffinity `preferredDuringScheduling`** hint set per-repo by overdeck when dispatching, informed by the same "which node last built this repo" state overdeck already owns. |
| Enrollment reconcile loop (host stuck in "enrolling") | `admission-loop.ts` | **Yes** — a new node `kubectl join`s and is `Ready` deterministically; no polling loop needed | Replace. Delete. |
| Repo mirror sync + credential delivery to the execution host | `remote-build.mjs` rsync + git push-to-bare + SSH key file | **No** — k8s does not sync your repo for you | **Must survive as overdeck code**, reshaped into: overdeck's existing rsync/bare-repo mirror mechanism, and a **read-only host-mount** of a credentials directory into the Pod (owner's explicit decision, see below) instead of an SSH key shipped by the dispatcher. |
| Logs / exit code back to the collector | `emitJobReport()` → spine CLI → `POST /jobs/report` | **Partially** — k8s natively gives you Pod `phase`, container `exitCode`, and `kubectl logs`/the container runtime's log files | Replace the transport (SSH exec + polling) with the k8s API (watch Pod status, tail logs via the API server) but **keep the same `/jobs/report` shape** the collector already ingests — a small adapter reads Pod events and re-emits the existing report format, so the collector needs no schema change. |

## Control-plane placement

Read `~/.claude/buildbox-hosts.json` (not hardcoded). All three hosts are
`"state": "reachable"`. Roles:

- `debian1`: `["builder", "agent-seat", "e2e", "dangerlab", "agent-sandbox"]`
  — additionally the tunnel endpoint, GitHub Actions runner, and KVM host
  (per its `notes` field). Busiest box by role count.
- `debian2`: `["builder", "agent-seat", "e2e", "agent-sandbox"]`.
- `debian3`: `["builder", "agent-seat", "e2e", "agent-sandbox"]`.

`orders.build` and `orders.e2e` are both `["debian1", "debian2",
"debian3"]` — this is the existing **fill order**: `debian1` absorbs
placement first today, `debian3` last, so `debian3` carries the least
routine load under current behavior.

**Measured, not just inferred from fill-order**: live telemetry pulled
directly from each host (`~/.local/state/overdeck/buildbox-telemetry.json`
over `ssh -F /dev/null -p 2222 debian{1,2,3}`) at design time confirms the
fill-order inference:

| host | load avg (1m/8 cores) | `uptime` load avg | build.slice pids |
|---|---|---|---|
| debian1 | 7.50 | 7.27, 10.24, 7.02 | 314 |
| debian2 | 2.30 | 1.94, 3.87, 3.54 | 699 |
| debian3 | 0.56 | 0.47, 1.05, 1.03 | 0 |

`debian3` is genuinely the most idle box right now, not just least-loaded by
convention — the earlier caveat ("verify in Phase 2 with live metrics") is
resolved; treat the choice below as final, not provisional.

**Decision: k3s server (control plane + etcd) on `debian3`; `debian1` and
`debian2` join as agent nodes.**

Rationale:

- `debian3` has the plainest role set (no dangerlab, no CI-runner, no KVM,
  no tunnel-endpoint duties) — the control plane should not compete with
  the box the fleet already leans on hardest (`debian1`) or share a fault
  domain with the danger-lab VM work that must stay isolated
  (`debian1`'s `dangerlab` role; the task's hard constraint against
  destructive-test contamination is easier to keep watertight if the k3s
  API server is not co-resident with the danger-lab host).
- A single-server k3s control plane (etcd embedded, no HA) is legitimate
  here: this is a 3-node internal build fleet, not a customer-facing
  cluster, and `debian3` being the fill-order tail means it is also the
  most idle historically — the right home for a low-churn, low-CPU control
  plane process.
- Control-plane-on-`debian3` is a **single point of failure for
  scheduling**, not for build execution: Phase-1 design keeps the
  offload-degrades-to-local path (below) so a `debian3` outage stops new
  k8s-scheduled jobs, not the fleet.
- This is a placement call the agent owns per the project's "reversible
  mechanics are never a user decision" rule — not an owner junction.
  Swapping the server role to another box later, if load patterns shift, is
  a `k3s` config change, not a redesign.

**API-server reachability (`:6443`) — checked, not assumed a blocker.**
All three hosts have `access.lan: null` (different router than the
workstation — tailscale is the only path per the registry's own notes) but
`tailscale status` shows all three `active` with direct tailnet connections
already established, and each box's `iptables -L INPUT` shows default
`ACCEPT` policy with only a tailscale-managed `ts-input` chain — no
box-level firewall rule blocking arbitrary ports was found on any of the
three. This means opening `:6443` for the k3s API server is not expected to
require touching buildbox firewall/network config (the hard constraint
against touching sshd/network/firewall config should not be triggered by
Phase 2's k3s install) — but this is inferred from the *absence* of a
blocking rule, not from an actual `:6443` connection test, since k3s is not
installed anywhere yet. **Phase 2's first verification step must be an
actual cross-host `:6443` connectivity check** (e.g. `curl -k
https://debian3.<tailnet>:6443/healthz` from debian1/debian2) before
assuming the control-plane join will work; if it does not, that is a
genuine blocker to report, not something to route around by weakening a
firewall rule without the owner's involvement.

## Job execution model: Job, not bare Pod

Each build/test invocation is a **`batch/v1` Job with `backoffLimit: 0` and
a single Pod**, not a bare Pod and not a long-lived Deployment:

- `backoffLimit: 0` because the hand-rolled schedulers never silently
  retried a build (a retried build with side effects — partial artifacts,
  half-run migrations — is worse than a visible failure); overdeck's own
  dispatcher decides whether to re-submit, matching today's behavior where
  `ClusterScheduler.complete()` just records a terminal state.
- `activeDeadlineSeconds` set per job from the same timeout the current
  `local-gate`/`remote-build.mjs` step timeouts already use
  (`ssh_exec_timeout_sec`, `dispatch_deadline_ms`) — one job cannot wedge a
  node forever.
- `ttlSecondsAfterFinished` short (minutes) so finished Jobs are
  garbage-collected automatically instead of accumulating (today's
  equivalent: `remote-build.mjs`'s own state-dir cleanup,
  `remoteCleanupPaths`).
- One Pod per Job, not a `parallelism`/`completions` fan-out — each overdeck
  build request maps 1:1 to one execution, matching the current
  one-ticket-one-host model.

**Repo mirror**: overdeck already owns a push-to-bare-then-pull mirror
mechanism (`syncPushGit`/`syncPull`, remote-build.mjs:732,809). Keep it
unchanged as the transport, but land the mirror on a **host-path volume on
the target node** (`/var/lib/buildbox`, the scratch disk every buildbox
already mounts there per the registry's `notes` field) mounted into the Pod,
rather than rsync-into-a-freshly-cloned-container-filesystem per job — this
preserves the incremental-mirror speed advantage the current design already
has and avoids re-syncing a large repo on every Job.

**Credentials**: **owner-decided constraint — read-only host mount.** The
buildbox SSH key and any registry/npm credentials already live on each
buildbox's filesystem (used today by cpu-guard's own build steps). Mount
that existing credentials directory into the Pod as a `hostPath` volume with
`readOnly: true`. Do **not** introduce a `Secret`-based credential path in
Phase 1 — that is a legitimate later hardening step (k8s Secrets, or better,
an external-secrets operator), but the task scope is placement, not
credential architecture, and the owner has already fixed this specific
decision.

**Logs and exit codes to the collector**: the collector already ingests
`/jobs/report` in the shape `emitJobReport()` produces
(`remote-build.mjs:162-183`, forwarded through the spine CLI's `report`
command to the controller). Build a small **overdeck-owned watcher**
(not a k8s-native component) that watches Job/Pod status via the k8s API
(`client-go`-equivalent for whatever the controller's stack is — the
controller is already a Bun/TypeScript service, so the k8s JS client or
plain REST calls to the k3s API server's `:6443` work without adding a new
language) and on Pod completion: reads the exit code from
`status.containerStatuses[0].state.terminated.exitCode`, pulls the Pod's
logs via the k8s API's `/log` subresource, and re-emits the **existing**
`emitJobReport()` JSON shape to `/jobs/report`. The collector's ingestion
code does not change.

## Migration path

Reversible, staged, no production cutover in this task:

1. **Phase 1 (this change)**: design only, landed to `main` as a doc.
2. **Phase 2 (next task)**: install k3s server on `debian3`, join
   `debian1`/`debian2` as agents (`k3s agent` with the server's join
   token). No `cpu-guard.sh`/`local-gate` traffic touches k3s yet. Prove one
   real Job schedules and runs end-to-end (a trivial `true`/echo command is
   sufficient — do not run a resource-exhaustion or destructive workload
   anywhere but the danger-lab VM per hard constraint). This is the target
   of Phase 2 below.
3. **Phase 3**: build the overdeck-owned Job-status watcher (logs/exit-code
   bridge to `/jobs/report`) and the repo-mirror hostPath wiring. Validate
   against the *existing* collector ingestion with no collector-side
   changes.
4. **Phase 4 — parallel run, no cutover**: add a **feature-flagged** third
   path in `cpu-guard.sh`/`local-gate` (a new `"k3s_enabled": true` key
   alongside today's `"enabled"` in `~/.claude/build-remote.json`) that,
   when on, submits the Job to k3s instead of the direct-SSH path, for a
   deliberately narrow allowlist of low-risk commands first (e.g. `pnpm
   typecheck` before `pnpm build`). Both paths coexist; nothing is deleted.
   Compare wall-clock and success rate against the SSH path over a real
   week of traffic before trusting it.
5. **Phase 5 — cutover**: flip the default for all heavy commands to the k3s
   path once Phase 4's comparison is clean. Scheduler A's `selectHosts()`
   and Scheduler B's `reserveLeastLoaded()`/`reconcile()` placement code
   paths become dead once nothing calls them; **delete** them explicitly
   (not just stop calling) — `admission-loop.ts`,
   `capability.ts`'s SSH-based prober and breaker, and `scheduler.ts`'s
   placement logic — while **keeping**: the ticket/report data model to the
   extent the collector's schema depends on it, and `buildslot.sh`'s
   adaptive-concurrency mechanism (no k8s equivalent, see table above) on
   any box that still mixes interactive and build load.
6. **Rollback at any phase**: flip `k3s_enabled` back to `false` in
   `~/.claude/build-remote.json` (a config edit, not a deploy) — Phase 4/5
   traffic falls straight back to the untouched direct-SSH path, since it
   is never deleted until step 5 confirms clean comparison. Before step 5,
   rollback is a config flag. After step 5, rollback means re-adding the
   deleted placement code from git history (`git checkout <pre-cutover-sha>
   -- controller/src/scheduler.ts ...` per this repo's revert convention) —
   flagged explicitly so step 5 is not taken lightly.

## Degradation when k3s is down

**Never silently drop a job.** Concretely, at every phase from 4 onward:

- The Job submission call to the k3s API server (`:6443` on `debian3`) is
  wrapped with the same fail-open-to-existing-path pattern
  `hostEligibleCli()` already uses today (remote-build.mjs:150/154/158): on
  timeout, connection refused, or a non-2xx response, `local-gate` falls
  through to **today's direct-SSH `chooseHost()`/`selectHosts()` path**,
  unchanged, not to the laptop. This is why Phase 5 keeps the SSH path
  physically present in the repo even after it stops being the default —
  it is the k3s fallback, not dead weight.
- Only if *both* k3s and every direct-SSH buildbox candidate are
  unavailable does the existing `buildslot.sh` local-concurrency admission
  on the laptop apply, exactly as it does today — this design does not
  change the laptop-fallback trigger condition, which is already governed
  by `docs/specs/2026-08-06-cluster-source-of-truth-design.md`'s workstation
  execution-eligibility rule (execution-ineligible while any builder is
  available).
- The overdeck controller's own `/admission/eligible` and `/jobs/report`
  advisory calls keep their existing fail-open/best-effort semantics
  unchanged — a down k3s control plane must not make those calls block
  either.
- A `debian3` outage (the k3s server) does not take down `debian1`/`debian2`
  as SSH-reachable direct build targets — they simply stop being schedulable
  as k3s agents until `debian3` recovers, and the fail-open above routes
  their traffic through the direct-SSH path in the interim.

## Open items this design does not resolve

- Whether Kueue or plain `PriorityClass`/`ResourceQuota` is the right
  FIFO-fairness replacement for `buildslot.sh`'s ticket queue on
  k3s-scheduled (not locally-adaptive) traffic — deferred to Phase 2+ design,
  noted as a gap, not a decision made here.

Resolved during this design pass (no longer open): Scheduler B's ticket
queue is confirmed dead code relative to live traffic (zero non-test
`enqueue()` callers); `debian3` as control-plane host is confirmed by live
telemetry, not just fill-order inference. See "Where they disagree" and
"Control-plane placement" above.

## Phase 2 status (this run)

Phase 2 (server install + one agent join + prove one Job) was executed in
this same task, immediately after this design was reviewed and landed. No
production traffic was migrated — `cpu-guard.sh`/`local-gate` are untouched.

- **Server**: k3s v1.36.3+k3s1 installed on `debian3` via the official
  `get.k3s.io` installer, `sudo -n`, no laptop involvement. Reconfigured
  (idempotent re-run of the installer) to advertise the tailscale address
  (`--node-ip=100.101.104.41 --node-external-ip=100.101.104.41
  --flannel-iface=tailscale0`) instead of the default LAN-interface IP,
  because every buildbox's `access.lan` is `null` in the registry — the
  boxes sit behind different routers and tailscale is the only path
  (confirmed live: `debian3`'s `INTERNAL-IP`/`EXTERNAL-IP` now read
  `100.101.104.41` in `kubectl get nodes -o wide`, not the LAN address).
  This is a correction to this doc's earlier "Phase 2's first verification
  step must be an actual cross-host `:6443` connectivity check" — that check
  was run (`curl -sSk https://100.101.104.41:6443/healthz` from `debian2`
  returned `HTTP:401`, i.e. TLS/TCP reachable, auth rejected as expected)
  and passed with no firewall change needed, confirming the earlier
  iptables/tailscale-ACL read.
- **Agent**: `debian2` joined via `K3S_URL=https://100.101.104.41:6443` +
  the server's node-token, `INSTALL_K3S_EXEC="agent --node-ip=100.79.69.43
  --node-external-ip=100.79.69.43 --flannel-iface=tailscale0"`. `debian1`
  was deliberately not joined in this run (spec's own rationale: keep the
  danger-lab/CI-runner/tunnel-endpoint box out of the new cluster until the
  design is further validated) — the task only required one agent.
  `kubectl get nodes` shows both `debian3` (`control-plane`, `Ready`) and
  `debian2` (`Ready`).
- **Proof job**: a `batch/v1` Job (`overdeck-k3s-proof`, `backoffLimit: 0`,
  `activeDeadlineSeconds: 60`, `ttlSecondsAfterFinished: 300`,
  `nodeSelector: kubernetes.io/hostname: debian2`) running
  `busybox:1.36` (`echo ...; date; exit 0`) was applied from `debian3`'s
  control plane. Result: `Completed`, `succeeded: 1`, Pod
  `overdeck-k3s-proof-h59rm` scheduled onto `debian2` (cross-node — proves
  the control plane on `debian3` actually dispatches work to a different
  physical box, not just to itself), logs read back via `kubectl logs`
  (`overdeck-k3s-proof-96 running on overdeck-k3s-proof-h59rm`), exit code
  `0` read via `status.containerStatuses[0].state.terminated.exitCode` —
  the exact field this design's Job-status watcher (Job execution model
  section, above) is specified to read in Phase 3.
- **Not done in this run** (correctly out of scope per the task): the
  overdeck-owned Job-status watcher/`/jobs/report` bridge (Phase 3), the
  `k3s_enabled` feature flag and parallel run (Phase 4), any change to
  `cpu-guard.sh`/`local-gate`/`controller/src`, and joining `debian1`.

## Implementation status — Phase 2/4 submission slice (2026-08-08)

Implemented, off by default:

- `modules/workstation/claude/lib/k3s-remote-build.mjs` — `buildJobManifest`
  (backoffLimit 0, ttlSecondsAfterFinished, env forwarding, resource
  requests/limits) + `tryK3sBuild` (kubectl apply → poll Job conditions for
  Complete/Failed True with deadline → logs → pod exit code). Pre-acceptance
  failure (config, API unreachable, submission rejected) returns `null` so
  the caller falls through to the unchanged SSH path; post-acceptance it
  never falls back and returns exit code or reserved status 125
  (`k3s-status-unknown`).
- Hook in `tryRemoteBuild` (`remote-build.mjs`) before the legacy path,
  gated behind the flag and the existing local-only exemptions.
- **Flag surface (canonical, per Phase 4 above): `"k3s_enabled": true` in
  `~/.claude/build-remote.json`.** Optional config keys: `k3s_image`,
  `k3s_namespace`, `k3s_kubeconfig`. Env vars are tunables/overrides only:
  `BUILD_REMOTE_K3S` (1/0 force-override for one-shot testing),
  `BUILD_REMOTE_K3S_IMAGE`, `_NAMESPACE`, `_KUBECONFIG`, `_TTL_SECONDS`,
  `_JOB_TIMEOUT_SEC`, `_POLL_MS`, `_CPU/MEMORY_REQUEST/LIMIT`,
  `_SUBMIT_TIMEOUT_MS`, `_IMAGE_PULL_POLICY`.
- Tests: `tests/k3s-remote-build.test.mjs` (unit, injected kubectl),
  `tests/k3s-remote-build-integration.sh` (real cluster via
  `~/.kube/config-buildboxes`, SKIPs when unreachable).

Implemented in Phase 4 (2026-08-09): the existing git snapshot transport now
fans materialization out to reachable registry builders before submission,
restricts the Job to nodes where that exact fan-out succeeded, mounts the
populated `/var/lib/buildbox/builds/<mirror>` hostPath checkout as the working
directory, and passes its epoch into a read-only init-container guard. Missing,
empty, or superseded workspaces fail before the build container starts; an
invalid materialization contract is rejected before `kubectl apply`. The exact
`overdeck.dev/build-key` annotation remains on both Job and Pod template.

Still not implemented: the allowlist + parallel-run comparison traffic policy
and cutover/deletion (Phase 5). `k3s_enabled` remains off by default.