--- name: od-cluster description: Cluster/offload incidents — buildbox unreachable, k3s/offload failover, remote build spill, e2e-remote routing, SSH to debian hosts failing. Triggers on buildbox, debian1/2, remote build, spill, failover, unreachable host, e2e-remote. Invoke BEFORE hardcoding a hostname or SSHing blindly. --- # Cluster / offload — incident playbook audience: AI coding agents first. BLUF: read registry → contact ONLY reachable hosts → follow spill order. ## Covers **Absorbs:** `tool-reliability-fatigue` (buildbox/k3s/offload subset — ids C084, C166 per taxonomy `_meta`). **Keywords:** buildbox, k3s, offload, debian1, debian2, remote build, failover, ssh, e2e-remote, spill, vm, cluster, unreachable host. **Registry coverage:** 2 of 217 (2026-08-08). Low count, high blast radius — wrong host contact or hardcoded hostname breaks failover. ## Doctrine Read `~/.claude/buildbox-hosts.json` first. NEVER contact host registry does not mark reachable. ### Reboot prohibition NEVER reboot or consider rebooting any cluster/buildbox machine without explicit owner permission for that specific reboot. General task approval, prior reboot approval, recovery urgency, existing automation, or a command's default behavior NEVER grants reboot permission. Planning, scripts, automation, recovery, hardening, migration, and verification MUST choose non-reboot paths by default. Rebooting physical Debian1/2/3 to test is forbidden. Simulate routine persistence with `systemd-nspawn`. Use QEMU/KVM snapshots only for boot loaders, initramfs, disk discovery, encryption, or early-boot networking. Only explicit owner permission for one named machine reboot now can authorize a real reboot. ```text DO NOT: bind-mount change → reuse broad harden command → automatic reboot TARGET: stop writers → copy → update+verify fstab → mount live → verify → restart writers ``` ## First steps — registry is source of truth 1. **Read registry:** `~/.claude/buildbox-hosts.json` — host identity, `reachable` flag, spill order, access doors. NEVER hardcode `debian1`/`debian2`. 2. **Reachability:** for each candidate host, use registry state only. Unreachable → next in spill order; do not retry SSH hammering. 3. **Classify workload:** remote build (`build-remote`), seat dispatch (`seat-remote`), e2e (`e2e-remote`), harness on box. Each wrapper reads same registry. 4. **SSH/door failure:** capture `ssh -v` one attempt; check door name from registry; verify laptop network — not credential plane (see `od-auth` if 401/token). 5. **Offload path:** `e2e-remote --help` for browser+dev-server pairs; server binds `127.0.0.1`; artifacts inside repo tree for pull-back. 6. **Failover:** if primary box wedged, spill to next reachable. NEVER reboot a box without explicit owner permission for that specific reboot (`docs/specs/2026-08-06-execution-plane-insulation.md`). ### Collector release packaging - `collector/src/buildbox-registry.ts` resolves `../../modules/workstation/claude/lib/buildbox-registry.mjs` inside the immutable release root. Collector release staging MUST archive and copy that reader alongside the collector. - A healthy fleet can otherwise appear completely unavailable: the collector fails closed before querying any host, so Cluster shows no machines or host actions even though `~/.claude/buildbox-hosts.json` is valid and the buildboxes are reachable. - Regression proof belongs at the release layout seam: prove the isolated release imports and reads the registry with the reader present, and still fails closed when it is absent. ### Local registry-reader runtime - Workstation control and observability scripts that read `buildbox-registry.mjs` MUST invoke a pinned local runtime (`/usr/bin/node`, with an explicit test override where needed), never PATH-wrapped `node` or `ft node`. - The CPU guard classifies semantic argv such as `--order build` as heavy work. Sending a local registry read through that shim can therefore attempt remote-only dispatch and falsely report the whole fleet unavailable. - Regression proof MUST put a failing `node` shim first on PATH and show that registry resolution still succeeds through the pinned local runtime without any remote-dispatch attempt. ## Never-touch — applies here From `modules/workstation/claude/incidents/never-touch.md`: - NEVER edit sshd config on buildboxes. - NEVER bypass admission queue. - NEVER `export OD_LOCAL_DISPATCH_OK=1` or `E2E_REMOTE_OK=1` to silence guards — route correctly instead. ## Placement map — read these paths `modules/workstation/claude/incidents/placement-map.md`: - `~/.claude/buildbox-hosts.json` — single source of host identity/state/spill order. - `~/.local/share/overdeck/deploy` — deployed collector reads registry same as agents. - `~/.local/state/overdeck` — fleet journals, dispatch state. ## Resolve — exact CLI (never guess syntax) ``` od-incidents list [--type cluster-offload] [--state S] od-incidents show od-incidents search od-incidents resolve --artifact [--summary ] ``` `resolve` REFUSES without `--artifact` (exit 2). Artifact = registry readout showing reachable host + successful remote command output. ## Self-update — MANDATORY This doc is the fleet's memory. You changed the architecture, wiring, commands, or doctrine this skill describes — or a live incident just proved a rule here wrong or missing → EDIT THIS FILE in the same landing as the change. Adopted AND rejected decisions both go in, with the measured why. An owner reminder to record a lesson is a failure of this rule, not the trigger for it.