# buildbox

Audience: AI coding agents first.

Converges debian1/debian2/debian3 to declared state. One source, N hosts. Boxes are never synced from each other.

```
buildbox audit [host...]          # check every declared item, exit 1 on drift
buildbox bootstrap [host...]      # converge user-level items (idempotent)
buildbox harden <host>            # apply host-config/ as root
buildbox claude-parity [host...]  # one line per host: is ~/.claude identical and live
```

No host argument means every host the registry marks `reachable`. A box added to
`~/.claude/buildbox-hosts.json` is picked up with no code change here.

## ~/.claude convergence

`bootstrap` makes `~/.claude` on every box byte-identical to the workstation's, for the
paths `modules/workstation/manifest/claude.json` declares — the same manifest `deckctl
sync` installs the workstation from. That manifest is the only list; this module does not
keep a second one.

| | |
|---|---|
| **Fleet-identical** | every `entries[]` path in `manifest/claude.json`: `hooks`, `skills`, `agents`, `commands`, `output-styles`, `lib`, `bin`, `workflows/lib`, `systemd`, `scripts`, `tests`, `docs`, `command-assets`, `security-gate`, the statusline scripts, `tools.json`, `settings.json`, `CLAUDE.md`, `mcp.json`, the `*.json` configs |
| **Machine-local, never shipped** | everything the manifest does not name: `projects/`, `sessions/`, `shell-snapshots/`, `run/`, `receipts/`, `jobs/`, `daemon/`, `file-history/`, `handoffs/`, `debug/`, `cache/`, `history.jsonl`, `plugins/` |
| **Credentials** | not distributed by convergence, deliberately — see below |

- Entries declared `source: deploy` (`hooks`, `lib`, `bin`) ship from the workstation's
  landed deploy clone, not the working tree, because that is what the workstation itself
  executes. Shipping a working tree would put code on three boxes that no one is running.
- The workstation holds those entries as symlinks into the deploy clone; a box has no
  deploy clone, so it gets real files. Symlinks *inside* an entry are preserved as
  symlinks — `~/.claude/bin/bun` is a relative link onto `_cpu-guard-shim.sh`, and a box
  that materialized it as a plain copy would run builds unguarded while every
  byte-comparison of regular files passed. The digest covers link targets for that reason.
- `plugins/` is Claude Code's own state: 605 MB of caches and `installPath` values that
  are absolute paths into a per-account home. Copying it fleet-wide would ship one
  machine's absolute paths to three boxes. Plugin *declarations* are a separate problem
  from config convergence and are not solved here.

### Credentials and blast radius

Convergence ships **config only**. `bootstrap` pushes exactly two secrets, both by the
pre-existing push-only-if-absent path: the npm `_authToken` and a `gh` token. Claude's own
`~/.claude/.credentials.json` is **not** distributed — on the workstation it is a symlink
into a live account home, and a copy on three boxes turns any single box compromise into
an account compromise across the whole fleet plus the workstation. A box that needs to run
a seat gets its credential from the seat stack, per seat, revocable per seat.

### Verifying a box

`claude-parity` is the one command, and it asserts behavior, not file counts:

- **content** — mode, path, content and symlink target of every declared entry, hashed
  order-independently, compared against the same digest computed over the exact tree the
  push writes.
- **hook** — `hooks/deny-gate.mjs` is run with the payload Claude Code hands it, and must
  deny `ccr start` (a rule that lives in the shipped `tools.json`) *and* allow an ordinary
  command. That verdict depends on two declared entries agreeing and on the hook's
  interpreter resolving on the box; it depends on nothing machine-local.
- **wrapper** — `~/.claude/bin/bun` must still be a symlink onto the cpu-guard shim and
  must execute. A wrapper that resolved to the real binary would answer `--version`
  happily and offload nothing. The version is printed but never asserted: `devtools.json`
  pins `BUN_MAJOR` only, so a patch skew is a passing box by the toolchain's own contract
  and must not flap a gate the seat stack blocks launches on. Version parity is
  `item_bun`'s job.

Probes must not turn on machine-local state. The sudo-password gate looks like the
strongest hook to probe and is the wrong one: it keys on a file that exists only on the
workstation, so it reports a false failure on every box forever.

## Rules

- NEVER hand-apply root config on a box. Edit `host-config/`, run `buildbox harden <host>`. Hand-applied state is invisible to `audit` and dies at reboot.
- `harden` takes **exactly one host per invocation**, enforced in `bin/buildbox`, not merely documented. It arms a hardware watchdog *and* it partitions and formats the scratch disk; either one wrong fleet-wide has no undo and nobody is there to stop it. One host, verify, reboot, then the next. `audit` and `bootstrap` fan out in parallel and buffer per-host output — only `harden` is serial.
- NEVER add buildbox config under `modules/buildbox/systemd/system`, `.../systemd/user-root`, `.../system-bin`, `.../system-sbin`. `lib/deckctl/system-units.sh` globs those names and copies them onto the **workstation**. `host-config/` is outside every one of those globs — keep it that way. Verify: `lib/deckctl/system-units.sh --dry-run` must print zero lines naming buildbox.
- `buildbox-checks.sh` NEVER mutates root state. Root items are audit-only there.
- Tools with their own repo (`ft`, `slopgate`, the quietcontext plugin) are declared in `devtools.json` as a repo plus a pinned sha, never vendored here and never hand-copied onto a box. Bump `ref` to roll the fleet; `item_devtools` clones, checks out, builds, and links the declared bins. A ref that is not pushed to its origin fails the box, which is the intended verdict — an unpushed local commit is not fleet state.
- OpenSSH listens on **2222, tailnet addresses only**. `host-config/apply.sh` generates `/etc/ssh/sshd_config.d/10-tailscale.conf` per node from `tailscale ip`, drops superseded hand-applied drop-ins, and reverts itself if the box does not come back up on 2222. Audited by `sshd-port` / `sshd-port22`. Port 22 is tailscaled's; an OpenSSH listener there is drift.

## Scratch disk

Every box has a second disk, formatted ext4 with filesystem label `buildbox-scratch` and mounted at `/var/lib/buildbox`. Regenerable build state is bind-mounted from there onto the home directory. Declared in `host-config/apply.sh`, audited by `item_scratch_disk`, designed in `docs/specs/2026-08-06-buildbox-second-disk-design.md`.

| `/var/lib/buildbox/…` | bound onto |
|---|---|
| `builds` | `~/builds` |
| `pnpm-store` | `~/.local/share/pnpm` |
| `npm` | `~/.npm` |
| `cargo-registry` | `~/.cargo/registry` |
| `dev-tools` | `~/.dev-tools` |

- **Blast-radius rule — only regenerable state goes on this disk.** A build rebuilds it; nothing there needs a restore. `actions-runner-*` trees hold GitHub registration credentials and stay on root. `~/.cargo/bin` is on every launcher's resolution path and stays on root; only `registry` moves. NEVER add a path here that cannot be recreated by re-running a build.
- **NEVER put a device node in declared state.** `/dev/sda` reorders across boots and differs per box. fstab keys on `LABEL=buildbox-scratch`, which is byte-identical on all three boxes; the device is discovered at `harden` time.
- **Discovery is fail-closed.** `apply.sh --discover-scratch` prints `CANDIDATE <dev> used=<bytes>` or `REFUSE <why>` and changes nothing. Zero candidates, two candidates, a filesystem holding more than 512 MB, and a filesystem that will not mount read-only are all refusals. Refusal rules are tested in `test/second-disk.test.sh` — run it before touching them.
- **Every fstab line carries `nofail`.** A dead scratch disk degrades the box to "builds are slower and root fills up", never to "the box does not boot". `x-systemd.requires=/var/lib/buildbox` orders each bind after its filesystem, so a bind can never shadow an unmounted parent with an empty root-backed directory.

### Recovery

Dead or wiped scratch disk: `buildbox harden <host>`. Discovery finds the unlabelled disk, reformats, re-converges fstab and re-migrates. There is no restore step and no backup, by design.

Interrupted migration leaves `<target>.migrated` beside the bind target — duplicate data, never missing data. Re-run `buildbox harden <host>`; it resumes from the stash and removes it only after the copy verifies.

## Unattended failsafe

Requirement: nobody can press reset on these boxes. A runaway seat must be shed, not wedge the machine. Five layers, each a fallback for the one above.

| # | Layer | Declared in | Fires when |
|---|---|---|---|
| 1 | Per-slice ceilings — `agent.slice` 40%/52%, `build.slice` 26%/33%, swap 4G/0 | `host-config/systemd-user-root/*.slice.d/90-ceiling.conf` | a workload **placed in those slices** exceeds its budget → cgroup OOM kill inside the slice |
| 2 | `systemd-oomd` on `user@<uid>.service` — pressure kill 50%/20s, swap kill 90% | `host-config/systemd-system/user@.service.d/50-oomd-swap-kill.conf` + distro `10-oomd-user-service-defaults.conf` | sustained memory pressure or swap exhaustion **anywhere under the user manager** → swap kill picks the descendant cgroup with the largest swap use; pressure kill picks by pressure + reclaim activity |
| 3 | Global kernel OOM killer, with the rescue path protected | `host-config/systemd-system/{system.slice.d,ssh.service.d,user.slice.d,user-.slice.d,session-.scope.d}/50-rescue-path.conf` | free memory is gone → kills the workload, never sshd |
| 4 | `kernel.panic_on_oops=1` + `kernel.panic=20` | `host-config/sysctl.d/99-buildbox-failsafe.conf` | kernel oops → reboot in 20s instead of a dead console |
| 5 | Hardware watchdog `iTCO_wdt`, `RuntimeWatchdogSec=60s` | `host-config/system-conf.d/50-buildbox-watchdog.conf` | PID1 unschedulable for 60s (total livelock) → hardware resets the box |

Layer 5 is the actual answer to "never needs a reset button": it is the only layer that survives a livelock in which every userspace guard is itself unschedulable.

**Placement decides whether layers 1–2 bind.** Layer 1 binds only a workload started with `Slice=agent.slice` / `build.slice`; a user service with no `Slice=` lands in `app.slice` and escapes it. Layer 2 binds anything under `user@<uid>.service` (all user services and slices, rootless podman scopes included). A bare `ssh debian1 'claude …'` runs in `session-N.scope` — a *sibling* of the user manager — so **neither** layer 1 nor layer 2 binds it. Layers 3–5 are unconditional and bind every placement.

### Deliberately not enabled

- `kernel.hung_task_panic` — covers "PID1 healthy, tasks stuck in D state". On local NVMe with no NFS that is rare, while a spurious 120s D-state under legitimate heavy build IO reboots the box mid-run. Layer 5 already covers the failure mode that matters.
- `vm.panic_on_oom` — pinned to `0`. The OOM killer must keep shedding load; panicking would turn every capped-cgroup kill into a reboot.
- `kernel.sysrq=1` is set but is a **manual** rescue tool (needs a live shell), not a failsafe. Never count it as a layer.

### Known limit

Layer 1 is advisory against an uncooperative agent: `90-ceiling.conf` outranks `~/.config/systemd/user.control/` drop-ins only by lexicographic basename order, so a user-written `95-*.conf` would beat it — and an agent can also simply launch outside those slices. Layers 3–5 are root-owned and hold regardless. Do not re-architect layer 1 to fix this without a separate task.

Layer-3 session-scope reserve is live on every session scope on both boxes (32M, verified); `daemon-reload` applies it to already-running transient scopes, no reconnect needed.

Not tested, by design: no reboot and no induced OOM. Reboot-persistence of `sysctl.d` / `system.conf.d` and watchdog firing are verified by inspection only — never induce a failure to prove a guard fires.