# Buildbox second disk — design

Audience: AI coding agents first.

Slug: `buildbox-second-disk`

## Problem

Every buildbox carries a second, unconfigured disk while its root filesystem runs out of space.
Measured on debian1 (the only box reachable during design; debian2 and debian3 were mid-upgrade):

| | debian1 |
|---|---|
| Root | `nvme0n1p2`, 110 G, **83 G used (80%)** |
| Second disk | `sda`, 111.8 G, non-rotational (WD Green SATA SSD), one NTFS partition, **108 MB used** — `System Volume Information` only |
| fstab today | three entries: `/`, `/boot/efi`, swap. No second-disk entry |

What fills root:

| Path | Size |
|---|---|
| `~/builds` | 33 G |
| `~/.local` (of which `share/pnpm` 7.1 G) | 12 G |
| six `actions-runner-*` trees | ~22 G |
| `~/.cache` | 5.7 G |
| `~/.npm` | 2.3 G |
| `~/.cargo` (of which `registry` 308 M) | 329 M |
| `~/.dev-tools` | 293 M |

The boxes are headless and unattended with nobody to press a reset button, so every decision below
is constrained by the module's existing doctrine: a box must never fail to boot, and a box must never
be converged by hand.

## Decision

One filesystem on the second disk, mounted at `/var/lib/buildbox`, with the **regenerable** build
state bind-mounted onto it from the user's home. Root keeps the OS, home configuration, seat state
and the runner registrations.

The line this draws is blast radius, not size. Everything moved to the second disk can be rebuilt by
re-running a build; nothing moved is irreplaceable. A failed SATA SSD therefore costs a cache rebuild
and never a restore — and the recovery procedure for a dead second disk is "wipe and re-converge",
which is the same procedure the module already uses for everything else.

`/var/lib` is the FHS location for variable state owned by an application, and `buildbox` names the
module that owns these machines. `/srv` was rejected: FHS scopes it to data the host *serves* to
others, which build scratch is not.

### What moves

| Bind source (second disk) | Bind target (home) | Measured |
|---|---|---|
| `/var/lib/buildbox/builds` | `/home/user/builds` | 33 G |
| `/var/lib/buildbox/pnpm-store` | `/home/user/.local/share/pnpm` | 7.1 G |
| `/var/lib/buildbox/npm` | `/home/user/.npm` | 2.3 G |
| `/var/lib/buildbox/cargo-registry` | `/home/user/.cargo/registry` | 308 M |
| `/var/lib/buildbox/dev-tools` | `/home/user/.dev-tools` | 293 M |

≈43 G of 112 G, leaving ~69 G of headroom, and reclaiming ≈43 G of the 83 G on root.

### What deliberately does not move

- **`actions-runner-*`** (~22 G) — a runner tree holds its registration credential and its own
  service unit. It is not regenerable without re-registering against GitHub, so it fails the
  blast-radius test that defines this disk.
- **`~/.cache`** (5.7 G) — mixed ownership. It holds build caches but also `ci-disk-alarm.stamp`
  and other small state whose couplings are not enumerated. Excluded until something needs it.
- **`~/.cargo`** as a whole — `~/.cargo/bin` is on the resolution path for every launcher. Only
  `registry` moves.

## Architecture

Five stages, each fail-closed. Stages 1–3 are root work and run from `host-config/apply.sh`
(`buildbox harden`); stage 5 is the audit and runs in `lib/buildbox-checks.sh` on every
`buildbox audit`.

```dot
digraph second_disk {
  rankdir=LR;
  discover [shape=diamond,label="1. discover\nexactly one candidate?"];
  refuse   [shape=box,label="report and stop\n(never guess a device)"];
  prepare  [shape=box,label="2. partition + format\nGPT, one part, ext4,\nLABEL=buildbox-scratch"];
  mount    [shape=box,label="3. fstab + mount\nLABEL=, nofail"];
  migrate  [shape=box,label="4. one-time migration\nrsync home dir -> disk"];
  audit    [shape=box,label="5. audit item\nreports drift forever after"];
  discover -> refuse  [label="0 or >1"];
  discover -> prepare [label="exactly 1"];
  prepare -> mount -> migrate -> audit;
}
```

### 1. Discovery — never hardcode a device node

Device names reorder across boots and differ per box, so `/dev/sda` must not appear in any declared
state. The candidate is derived: a whole-disk block device that is **not** the disk carrying `/`,
carries no mounted filesystem, and is not itself already the scratch disk.

Fail closed. Zero candidates or more than one candidate is a report, never a guess — on a box where
the operator cannot be asked, picking the wrong disk is unrecoverable.

Idempotence is keyed on the filesystem label: a disk already carrying `buildbox-scratch` is never a
candidate for formatting, so re-running `harden` converges the fstab entry and stops.

### 2. Preparation — refuse to destroy data

The disk is wiped, so the refusal rules are the load-bearing part of this design:

- refuse if the candidate holds any filesystem whose used bytes exceed a small threshold
  (debian1's NTFS holds 108 MB of Windows volume metadata and nothing else; the threshold exists to
  pass that case and fail a real one)
- refuse if the candidate is the root disk, by comparing the parent device of `/`
- refuse if `lsblk` reports any mountpoint anywhere on the candidate

Layout: GPT, one partition spanning the disk, ext4 with the reserved-block percentage set to zero —
this is scratch, and the default 5 % reserve is ~5.5 G of pure loss — and filesystem label
`buildbox-scratch`.

### 3. Mount — identical on every box, and never blocks a boot

fstab keys on `LABEL=`, not `UUID=`. A UUID is unique per box, which would make the declared fstab
line differ per host and break "one source, N hosts". The label is written at format time and is
byte-identical on all three boxes. There is exactly one such disk per box, so collision is not
possible.

The literal entries, which are declared state and must appear verbatim:

```
LABEL=buildbox-scratch  /var/lib/buildbox  ext4  defaults,noatime,nofail,x-systemd.device-timeout=10s  0  2
/var/lib/buildbox/builds        /home/user/builds                  none  bind,nofail,x-systemd.requires=/var/lib/buildbox  0 0
/var/lib/buildbox/pnpm-store    /home/user/.local/share/pnpm       none  bind,nofail,x-systemd.requires=/var/lib/buildbox  0 0
/var/lib/buildbox/npm           /home/user/.npm                    none  bind,nofail,x-systemd.requires=/var/lib/buildbox  0 0
/var/lib/buildbox/cargo-registry /home/user/.cargo/registry        none  bind,nofail,x-systemd.requires=/var/lib/buildbox  0 0
/var/lib/buildbox/dev-tools     /home/user/.dev-tools              none  bind,nofail,x-systemd.requires=/var/lib/buildbox  0 0
```

`nofail` on every line is the same doctrine as the rest of the module's failsafe: a dead second disk
degrades the box to "builds are slower and root fills up", never to "the box does not boot and
nobody is there to fix it". `x-systemd.requires` orders each bind after the filesystem it lives on,
so a bind can never shadow an unmounted parent with an empty root-backed directory.

Directory ownership: `/var/lib/buildbox` is root-owned `0755`; every subdirectory that backs a bind
is `user:user`, created before first mount.

### 4. One-time migration

Per pair, and only when the target on the second disk is empty and the home directory is not:
quiesce anything writing (runner services, seats), copy with `rsync -aHAX --delete`, verify the copy,
then mount the bind and remove the now-shadowed original content from root.

The order matters and is the one irreversible step: content is deleted from root only *after* the
bind is mounted and verified, so an interrupted migration leaves a box with duplicate data — wasteful
but correct — never with missing data.

### 5. Audit

A new `item_scratch_disk` in `lib/buildbox-checks.sh`, registered alongside the other items and
following their established shape (`ok` / `bad`, `bad` sets drift, root-level facts are report-only).

It reports, as one line each:

- the scratch filesystem: present, mounted at `/var/lib/buildbox`, its size and free bytes, and
  whether the backing device is rotational — a spinning second disk is not a failure but changes
  what belongs on it, and the fleet must not silently assume SSD
- each declared bind: mounted, and pointing at the second disk rather than at root
- drift when the disk is absent, when a bind is missing, or when a bind target resolves to the root
  filesystem (the silent-degradation case that otherwise looks healthy)

### Disk-space alarm

`user-config/bin/ci-scratch-prune.sh` already alarms on floors for `/` and `/tmp`. It gains a floor
for `/var/lib/buildbox`, because moving 43 G of churn onto a new filesystem without a floor
reintroduces exactly the ENOSPC failure this work exists to remove. Same syslog + cooldown-stamp
mechanism it already uses; no new alerting path.

## Distribution across the fleet

The distribution system already exists and must not be reinvented: `modules/buildbox/bin/buildbox`
is one source and N hosts. It reads the host list, port, ssh user and identity from
`~/.claude/build-remote.json`, ships `lib/buildbox-checks.sh` over ssh, and with no host argument
runs against every declared host. Nothing here is run three times by hand today.

Two things are genuinely missing, and only one of them applies to this work.

### Fan-out is serial

`audit` and `bootstrap` walk the hosts in a `for` loop, so a three-box audit costs three round trips
end to end. These verbs are read-mostly and idempotent and have no cross-host coupling, so they can
run concurrently: one background ssh per host, per-host output captured to its own buffer and
printed grouped in declared host order once all have finished, exit status the OR of all hosts.

Buffering is not cosmetic — interleaved `OK`/`DRIFT` lines from three boxes are unreadable and,
worse, unattributable, which is how a drift line gets pinned on the wrong box.

### `harden` must stay serial — and should be enforced, not merely documented

This is the important half. The README already says never to run `harden` on two boxes at once,
because it arms a hardware watchdog and a bad watchdog config applied fleet-wide reboot-loops every
box with nobody there to stop it. This design adds a second reason of the same kind: `harden` is now
the verb that **partitions and formats a disk**. A fan-out bug, a wrong candidate, or a refusal rule
that is too permissive would hit all three boxes simultaneously and destroy the fleet in one command.

The rule is currently a sentence in a README that the code does not enforce — `harden` loops over
however many hosts it is given, and with no argument it defaults to *all* of them. That default is
the sharp edge. `harden` gains a fail-closed guard: exactly one host per invocation, no host-list
default, an explicit error naming the correct form otherwise.

So: parallel fan-out for `audit` and `bootstrap`, an enforced one-host-at-a-time for `harden`, and
this disk work rolls out one box at a time on purpose — verify, reboot, then the next.

## Files

| File | Change |
|---|---|
| `modules/buildbox/host-config/apply.sh` | discovery, refusal rules, partition/format, fstab convergence, directory ownership |
| `modules/buildbox/bin/buildbox` | parallel fan-out for `audit`/`bootstrap`; fail-closed one-host guard on `harden` |
| `modules/buildbox/lib/buildbox-checks.sh` | `item_scratch_disk`, registered before `item_system` |
| `modules/buildbox/user-config/bin/ci-scratch-prune.sh` | floor + alarm for `/var/lib/buildbox` |
| `modules/buildbox/test/second-disk.test.sh` | new; discovery and refusal logic against fixtures |
| `modules/buildbox/README.md` | the disk's role, the blast-radius rule, and the recovery procedure |

Migration (stage 4) is an operator step run once per box under `harden`, not a persistent code path.

## Error handling

| Condition | Behaviour |
|---|---|
| No candidate disk | report, exit non-zero, change nothing |
| More than one candidate | report both, exit non-zero, change nothing |
| Candidate holds data above the threshold | report used bytes and the filesystem type, change nothing |
| Disk present but unmountable at boot | `nofail` — box boots, binds do not mount, audit reports drift |
| Bind target resolves to root filesystem | audit drift; builds keep working on root |
| Migration interrupted | duplicate data on both filesystems; re-runnable; never data loss |

## Testing

- `modules/buildbox/test/second-disk.test.sh` — the refusal rules are the part that can destroy a
  box, so they are tested directly against synthetic `lsblk`/`findmnt` fixtures: zero candidates,
  two candidates, candidate-is-root-disk, candidate-holds-data, candidate-already-labelled.
- `buildbox audit debian1 debian2 debian3` is clean on all three, and reports the same scratch line
  on each — the fleet-identical acceptance criterion.
- Reboot verification on one box before the next: `harden` is already documented as one host at a
  time, and this change adds an fstab entry, which is the classic way to make a headless box
  unbootable. `nofail` is what makes that safe, and the reboot is what proves it.

## Architecture Decisions

Three modules proposed (discovery/preparation in `apply.sh`, audit in `buildbox-checks.sh`, alarm in
`ci-scratch-prune.sh`), which is below the size gate for a full depth pass. Recorded decisions:

- **Bind mounts, not symlinks.** A symlink is visible to every tool and some refuse to follow one
  across a filesystem boundary; a bind is invisible and behaves as the real path. It is also
  reversible by unmounting, where a symlink must be un-made.
- **`LABEL=` over `UUID=`.** Chosen so the declared fstab text is identical on all three boxes.
  Accepts a theoretical label collision that cannot occur with one scratch disk per box.
- **No LVM, no pooling of the two disks.** Pooling root and scratch into one volume would remove the
  path question entirely, but root is a plain ext4 partition; converting it in place on a headless
  box with no console is a one-way door with no recovery path. Rejected on reversibility.
- **No `/home` relocation.** Considered and rejected with the same blast-radius argument that
  defines what moves: it would put seat state, credentials and runner registrations on the cheap
  disk.
- **Runner trees stay on root.** They fail the regenerable test.