# debian3 enrollment runbook

Audience: AI coding agents first.

Enrolls debian3 (tailnet `100.101.104.41`, MagicDNS `debian3`) into the buildbox fleet.
Source of truth for fleet state: `modules/buildbox/` — never hand-apply root config on a box.

**Run the steps in order.** `build-remote.json` has `local_fallback: false`: a host in `hosts[]`
that is not reachable hard-fails every job routed to it. Step 9 (adding `debian3` to `hosts[]`)
is LAST and only after step 7 is clean.

## State at time of writing

- debian3 answers OpenSSH on **22 only**; port 2222 refused. Tailscale SSH is OFF.
- No key we hold is authorized on debian3. Step 1 is the only manual step and it BLOCKS all others.
- Fleet reaches every box with `ssh -F /dev/null` on `port` from `~/.claude/build-remote.json` (2222).
  `~/.ssh/config` is NOT consulted by `bin/buildbox` or remote-build; it serves interactive/rsync use.
- debian1 is already at declared state (2222, tailnet-bound). debian2 carries hand-applied
  `/etc/ssh/sshd_config.d/60-buildbox.conf` (`Port 22` + `Port 2222`, bound `0.0.0.0`).

## 1. Owner: authorize the key and passwordless sudo (manual, on debian3's console)

The key is NOT negotiable: `bin/buildbox` and remote-build pass `identity_file` from
`build-remote.json` = `~/.ssh/id_ed25519_buildbox`.

Print it on the workstation:

```bash
cat ~/.ssh/id_ed25519_buildbox.pub
```

On debian3's console, as `user`:

```bash
mkdir -p -m 700 ~/.ssh
printf '%s\n' 'ssh-ed25519 AAAA... buildbox' >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
sudo -n true && echo "sudo -n OK"   # buildbox harden runs `sudo -n bash apply.sh`; must print OK
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub   # record for step 4
```

Verify from the workstation before continuing:

```bash
ssh -F /dev/null -p 22 -i ~/.ssh/id_ed25519_buildbox -o BatchMode=yes -o StrictHostKeyChecking=accept-new user@100.101.104.41 true && echo "reachable"
```

## 2. Flip sshd to 2222 via `harden` over port 22

Do NOT hand-edit sshd config on the box — `harden` owns it (`host-config/apply.sh` generates
`/etc/ssh/sshd_config.d/10-tailscale.conf` from `tailscale ip`, removes superseded drop-ins,
validates with `sshd -T`, and reverts itself if the box does not come back on 2222).
The port is not reachable yet, so run this one `harden` through a temporary config on port 22:

```bash
cd /home/user/Projects/overdeck
CFG=$(mktemp /tmp/build-remote-debian3.XXXXXX.json)
python3 -c "import json,os; c=json.load(open(os.path.expanduser('~/.claude/build-remote.json'))); c['hosts']=['debian3']; c['port']=22; json.dump(c, open('$CFG','w'), indent=2)"
BUILDBOX_CONFIG=$CFG modules/buildbox/bin/buildbox harden debian3
rm -f "$CFG"
```

One host only. NEVER `harden` fleet-wide: it arms a hardware watchdog, and a bad watchdog
config applied to every box at once reboot-loops the fleet with nobody there to stop it.

## 3. Verify the flip

```bash
ssh -F /dev/null -p 2222 -i ~/.ssh/id_ed25519_buildbox -o BatchMode=yes -o StrictHostKeyChecking=accept-new user@100.101.104.41 \
  'sudo -n ss -lntp | sed -n "/:2222 /p;/:22 /p"'
```

Expect listeners on `100.101.104.41:2222` and the tailnet IPv6 address, and NOTHING on 22.
Port 22 is dead after the flip — Tailscale SSH is off on debian3, so the console is the only
fallback. Do not proceed until 2222 answers.

## 4. Record host key under the alias, then add the `~/.ssh/config` entry

Our probes recorded debian3 under the raw IP only. `HostKeyAlias debian3` looks the key up
under the literal string `debian3`, so insert it under that name and drop the stale raw entries:

```bash
ssh-keygen -R 100.101.104.41
ssh-keygen -R '[100.101.104.41]:2222'
ssh-keyscan -p 2222 -t ed25519 100.101.104.41 2>/dev/null | sed 's/^\[[^]]*\]:2222/debian3/' >> ~/.ssh/known_hosts
```

`ssh-keyscan` is trust-on-first-use: compare its fingerprint against the one printed on the
console in step 1 before trusting it.

```bash
ssh-keygen -lf <(ssh-keyscan -p 2222 -t ed25519 100.101.104.41 2>/dev/null)
```

Append to `~/.ssh/config`, modeled on the `debian1` entry (`IdentityFile` is the buildbox key
because that is the one authorized in step 1):

```
Host debian3
	HostName 100.101.104.41
	Port 2222
	User user
	IdentityFile ~/.ssh/id_ed25519_buildbox
	IdentitiesOnly yes
	BatchMode yes
	ConnectTimeout 10
	ServerAliveInterval 30
	ServerAliveCountMax 3
	HostKeyAlias debian3
	StrictHostKeyChecking yes
```

```bash
ssh debian3 true && echo "alias OK"   # must succeed with StrictHostKeyChecking yes
```

## 5. Bootstrap user-level state

```bash
cd /home/user/Projects/overdeck
modules/buildbox/bin/buildbox bootstrap debian3
```

Installs mise/node/bun/rust/gitleaks/corepack shims, the GC timer, agent-confine pair, npm token,
git identity. Re-run until it reports no `DRIFT`.

## 6. Re-run `harden` on the real config (idempotency proof)

```bash
modules/buildbox/bin/buildbox harden debian3
```

Second run must be a no-op for sshd and must reach the box on 2222. This proves the normal
harden path works before debian3 carries traffic.

### Layer-5 watchdog verification — NEVER assume it armed

`host-config/system-conf.d/50-buildbox-watchdog.conf` sets `RuntimeWatchdogSec`, but the kernel
needs a watchdog device. The fleet's device comes from `iTCO_wdt` (Intel PCH). If debian3's
chipset differs, the module does not load, `harden` still reports success, and layer 5 silently
does not arm.

```bash
ssh debian3 'sudo -n modprobe iTCO_wdt; ls -l /dev/watchdog*; cat /sys/class/watchdog/watchdog0/{state,timeout,identity}'
```

`apply.sh`'s readback already prints `watchdog0.state` / `watchdog0.timeout` — that is the
primary signal; the `modprobe` above is the diagnosis when it prints `none`. If `iTCO_wdt` is
absent, find the right driver before declaring debian3 enrolled:

```bash
ssh debian3 'ls /lib/modules/$(uname -r)/kernel/drivers/watchdog/'
```

## 7. Audit must be clean

```bash
modules/buildbox/bin/buildbox audit debian3
```

Exit 0, no `DRIFT` lines. `sshd-port` must read `2222 on 100.101.104.41:2222 …` and
`sshd-port22` must read `unused`.

## 8. Converge debian2 to the same declared sshd state

`buildbox audit debian2` reports `DRIFT sshd-port22` today: debian2 carries hand-applied
`60-buildbox.conf` and an OpenSSH listener on `0.0.0.0:22`. Fleet audit cannot be green until
this is converged, so do it here — one host, after debian3 is proven.

```bash
modules/buildbox/bin/buildbox harden debian2
```

This removes `60-buildbox.conf`, drops the port-22 listener, and rebinds 2222 to debian2's
tailnet addresses only. Then flip its `~/.ssh/config` entry, which still says `Port 22`:

```bash
sed -i '/^Host debian2$/,/^$/ s/^\tPort 22$/\tPort 2222/' ~/.ssh/config
ssh debian2 true && echo "alias OK"
```

debian2 already answers on 2222 today, so the `~/.ssh/config` edit is safe before or after the
harden. Verify: `modules/buildbox/bin/buildbox audit debian2` exits 0.

## 9. Only now: add debian3 to `hosts[]`

Edit BOTH copies — they must stay identical:

- `~/.claude/build-remote.json` (live)
- `modules/workstation/claude/build-remote.json` (tracked)

```bash
diff <(python3 -m json.tool ~/.claude/build-remote.json) \
     <(python3 -m json.tool /home/user/Projects/overdeck/modules/workstation/claude/build-remote.json)
```

The tracked copy has uncommitted modifications in the main checkout as of 2026-08-06 — reconcile
that diff BEFORE editing `hosts[]`, or the commit ships an unrelated change.

Add `"debian3"` to `hosts[]` in both, then confirm routing:

```bash
modules/buildbox/bin/buildbox audit          # all three hosts, exit 0
```

## Cache policy (decision recorded, do NOT implement here)

- `~/builds` and `~/Projects` on a box are **derived mirrors**. remote-build rsyncs them on demand
  and the GC sweep reaps them. NEVER copy them box-to-box — module invariant: boxes are never
  synced from each other. They populate naturally from the workstation on the first job.
- `~/.local/share/pnpm` (7.1G on debian1) and `~/.cargo` (329M) are genuine warm caches. A cold
  debian3 is a slow debian3. Warming them is worth doing as a separate task; it is not part of
  enrollment.

## Known pre-existing issue (out of scope here)

`modules/workstation/claude/lib/buildbox-checks.sh` is a stale fork of
`modules/buildbox/lib/buildbox-checks.sh` (missing `item_node_system` and every item added since).
Nothing in the repo references it. Do not edit it as part of enrollment; delete or reconcile it
as its own task.