# Cluster degraded — oversized mirror trips the whole fleet

status: ACTIVE
owner request (2026-08-11): "The cluster is degraded (local-gate rsync timed out against
debian3 and debian1) — find the fastest way to put it back online."

## Outcome

A single non-git project tree can no longer take the buildbox fleet out of service, and the
fleet runs at its full three-host capacity.

## Findings (measured, 2026-08-11)

The fleet was never down. During the reported window (`~/.claude/local-gate.log`,
08:07–08:13Z) dozens of jobs finished `status=0` on both debian1 and debian3.

- Transport healthy: 20 MB rsync push measured 4.4 MB/s (debian1) and 3.1 MB/s (debian3);
  `local-gate --remote-doctor` reports `PASS remote rsync` on both.
- The failures all belong to ONE job key, `shim-npx-1722002-1786435586`, whose mirror is
  `Press.zone-29d8296fa4cb` (state files under `~/.claude/run/local-gate/`).
- `~/Projects/Press.zone` has no `.git`, so `projectRoot()` returns the directory itself and
  `syncPushGit()` takes the `no-git` full-tree rsync path. After `push_excludes` that tree is
  **15.3 GB / 352,160 files** (`wordpress` 15G, `ThemeFactory` 6.5G).
- The push deadline is the ~120 s dispatch budget (`dispatch_timeout_sec: 120`), not
  `rsync_timeout_sec: 900` — observed `rc=124 ... timed out after 113s/115s/117s`. At the
  measured throughput the budget covers ~400 MB, so this mirror can never push.
- Each timeout is classified transport-class and calls `markHostUnhealthy` — so a client-side
  size problem trips `host tripped host=debian3 reason=push-failed`, then debian1, latching
  `cached-unhealthy` and making unrelated jobs see `host skipped ... reason=not-ready`.

Secondary drift found while verifying:

- Registry marks debian2 `unreachable` and omits it from `orders`, but debian2 answers SSH,
  is idle (load 0.08), and holds exact-pinned parity (node 24.18.0 / pnpm 11.5.2 / bun 1.3.14).
- debian3 mise pins floating majors (`node = "24"`, no `pnpm` key) so the doctor reports
  `node parity local=v24.18.0 remote=v24.19.0` and `pnpm parity local=11.5.2 remote=11.20.0`.
  Parity is diagnostic-only — it never removed debian3 from the pool — but pnpm 11.20 vs 11.5
  risks lockfile divergence on remote builds.
- debian1 `seat audit failed (unknown)` sits next to `step=doctor-host timed out after 30s`
  while the box ran two jobs; unverified, re-check when quiesced.

## Acceptance criteria

1. A mirror whose push times out on two distinct hosts stops tripping host health and is
   refused fast with its own reason instead of burning the dispatch budget per host.
2. A single wedged host still trips and still spills — the existing failover is unchanged.
3. `modules/workstation/claude/tests/remote-build.test.mjs` green, plus coverage for both
   branches of the new rule.
4. debian2 restored to `reachable` and present in the `build` and `e2e` spill orders.
5. debian3 mise pins exact node/pnpm/bun matching the workstation.

## Execution steps

- [x] Diagnose from `~/.claude/local-gate.log` + `~/.claude/run/local-gate/` state files.
- [ ] `remote-build.mjs`: per-mirror push-timeout ledger; second distinct host ⇒ refuse
      `push-oversize` without `markHostUnhealthy`; clear on successful push.
- [ ] Tests for both branches.
- [ ] Registry: debian2 `reachable` + in `orders.build` / `orders.e2e`.
- [ ] debian3 mise pins (host-side, surgical keys only — keep `npm:@openai/codex`).
- [ ] Land, `packaging/deploy-local.sh`, re-run `local-gate --remote-doctor`.

## Deferred — flagged, not built

`modules/buildbox/lib/buildbox-checks.sh` bootstraps with `mise use -g node@24` / `bun@1`
(floating majors) while the doctor demands exact equality with the workstation. Every upstream
24.x release re-breaks parity. Pinning exact versions from whatever the workstation happens to
run inverts the problem — one local bump then breaks the fleet. The real fix is one shared
pinned toolchain manifest; larger than this task and out of its scope.

`~/Projects/Press.zone` itself is the owner's project, not a dev-tool: not restructured here.
With this change its jobs fail fast and locally instead of taking the fleet down.

## Current receipt

Landed `b86ee4380`, deployed (`deploy-local` reported `sha b86ee438`). Installed runtime
`~/.claude/lib/remote-build.mjs` carries `pushOversize`; `~/.claude/buildbox-hosts.json` shows
three `reachable` hosts in both spill orders. `remote-build.test.mjs` green (6/6 blocks,
including the new ledger block) on the merged tree. `local-gate --remote-doctor` after deploy:
all three hosts PASS ssh, node/pnpm/bun parity and remote rsync.

Production repro against the real mirror (`cd ~/Projects/Press.zone; local-gate --key
oversize-proof-1 -- node -e ...`): debian1 push timed out at 111 s and tripped, the run spilled,
debian3 timed out at 115 s and the ledger fired — `remote skip key=oversize-proof-1
mirror=Press.zone-29d8296fa4cb reason=push-oversize hosts=2`, no second trip. Retries
short-circuited on the early-skip without contacting a host, and the gate printed
`refusing local run`, so no laptop-CPU fallback was created. A normal job through the same gate
immediately afterwards returned `FLEET_OK`.

`ship.sh land` could not be used: the land queue returned `land-queue-failed — conducted the
queue without a verdict`, and it reset the worktree commit (files survived, re-committed).
Landed through the project's DIRECT LAND escape hatch from a bundle-cloned throwaway repo,
because the shared checkout cannot fetch at all (see below). Own verification: the touched
module's full test file.

## Open — not fixed here

- **RESOLVED without intervention — the shared checkout's dangling ref.** For part of this
  session `refs/heads/wt/factory-gpt-engine` and `worktrees/factory-gpt-engine/HEAD` pointed at
  `73d14eb08…`, an object present in no local DB, buildbox mirror, or GitHub, and every
  `git fetch` from the shared checkout died `fatal: bad object` — the likely reason the
  land-queue conductor returned no verdict. The owning lane has since advanced that branch to
  `b52a55303`, a commit that exists; `fetch` now exits 0 and the checkout sees the landed tip
  `b5c85ea48`. Nothing was deleted or repaired by hand. The transient shape is worth knowing:
  a lane mid-write can leave the shared object DB referencing an object it has not yet written,
  which breaks fetch for every other session until that lane advances.
- **Seat claude-version drift.** The doctor's seat leg prints `local git origin unresolved for
  github.com probe` + `step=doctor-host timed out after 30s` on all three hosts, but calling
  `auditHostSeat` directly resolves the origin fine and reports the real finding:
  `seat audit failed (claude-version:2.1.226)` against local 2.1.227. The origin-probe line is
  an artifact of the doctor's own invocation, not a broken origin. Nothing under `~/.claude/bin`
  calls `auditSeatHosts`, so seat dispatch is not gated by this leg. `buildbox claude-parity`
  reports config drift on all three hosts; `buildbox-parity.sh` is timer-driven and converges it.
- `deploy-local` printed `shim-drift-check: HAZARD` for uncommitted shim edits in the
  `muxrecover` and `plans-archive` worktrees — other sessions' WIP, untouched.

## Next executable action

None for this plan. The two open items above belong to their own lanes.