# local-gate remote dispatch timeout

audience: AI coding agents first.

- Status: ACTIVE
- Outcome: `local-gate` dispatches to a buildbox without phantom sub-second timeouts; a genuinely exhausted dispatch budget aborts with a named cause instead of a fake 1ms per-step timeout. Local fallback stays refused.
- Source request (owner, 2026-08-12): "Fix local-gate remote dispatch. All buildboxes accept SSH/rsync initially, but dispatch later times out during git-overlay-list, materialize (reported as a 1 ms timeout), or git-push. Investigate why materialize receives a 1 ms budget and why Git push hangs after successful connectivity checks. Do not enable local fallback."
- Reproduction receipt: `/home/user/.cache/agent-tmp/claude-1000/-home-user-Projects-overdeck/ad9bb539-c189-4030-be04-8f153ba7b79f/tasks/bxwzed9ad.output`
- Worktree: `.worktrees/localgate-dispatch` (branch `wt/localgate-dispatch`)
- Subject: `modules/workstation/claude/lib/remote-build.mjs`

## Observed failure

```
step=git-overlay-list host=debian2 after=5s
step=materialize      host=debian2 after=1s   -> Error: timed out after 1ms
step=git-overlay-list host=debian3 after=14s
step=materialize      host=debian3 after=1s   -> Error: timed out after 1ms
step=git-push         host=debian1 after=97s  -> spawnSync git ETIMEDOUT
remote-only dispatch failed step=git-push host=debian1; refusing local run
```

## Measured findings

1. `git-overlay-list` is a **local** command — `overlayFiles()` (`remote-build.mjs:851`) runs
   `git ls-files -o -i --exclude-standard` in the workstation checkout. It is reported with a
   `host=` label, which misdirects diagnosis to the buildboxes. Measured in the main checkout:
   **12.9s wall** (`user 0.4s sys 1.1s` — I/O bound; the ignored-file walk traverses
   `.claude/worktrees/**`, hundreds of trees). Requested budget is
   `local_step_timeout_sec`/dispatch remainder; debian3 got 14s and debian2 got 5s.
2. Every step timeout is clamped by one fleet-wide wall-clock budget:
   `stepTimeoutMs()` (`:448`) returns `Math.max(1, min(requested, dispatch_deadline_ms - now))`,
   with `dispatch_deadline_ms = start + dispatch_timeout_sec (120s)` set once at
   `runRemote` (`:1264`) and never refreshed per failover host.
3. The `Math.max(1, …)` floor is why `materialize` reports **1ms**: the budget was already
   spent by `git-overlay-list`, but the code proceeds with a 1ms timeout instead of aborting.
   `reportTimeout()` (`:456`) then renders 1ms as `after=1s`, hiding it further.
4. `git-push` never "hangs after connectivity checks": the connectivity probe is `echo ok`
   and never touches the mirror. The push is clamped the same way — `sh()` (`:799`) takes
   `min(requested, local_step_timeout_sec-or-dispatch-remainder)`, so the explicitly requested
   `execMs(cfg)` (`ssh_exec_timeout_sec` = 1800s) was cut to the 97s of remaining fleet budget.
5. Installed copy `~/.claude/lib/remote-build.mjs` is HEAD **plus an unlanded `--remote-doctor`
   `seatSpawn` change**; do not clobber it on install.

Root cause: a 120s control-plane budget is shared across host selection, repeated local
overlay enumeration, data transfer, and every failover host — and exhaustion degrades into
1ms phantom timeouts instead of an explicit abort.

## Acceptance criteria

- `git-overlay-list` runs at most once per dispatch, host-independent, and is not labelled with a host.
- Data-transfer steps (`git-push`, `overlay-scp`, `materialize`, rsync) use their configured
  transfer budgets, not the control-plane dispatch remainder.
- Budget exhaustion aborts with `reason=dispatch-budget-exhausted step=<consumer>`; no step
  is ever issued a sub-second phantom timeout.
- No local fallback introduced; remote-only refusal preserved.
- `modules/workstation/claude/tests/remote-build.test.mjs` green, plus a new regression covering
  both branches (budget available → real timeout; budget exhausted → named abort).
- Installed before landing: candidate written to `~/.claude/lib/remote-build.mjs` (preserving
  the unlanded doctor change) and proven through the real `local-gate` entrypoint.

## Execution steps

1. [done] Reproduce and measure; identify the four defects above.
2. [active] `local-gate --remote-doctor` baseline (running).
3. Implement: hoist overlay enumeration out of the per-host loop; exempt transfer steps from
   the control-plane deadline; replace the 1ms floor with an explicit exhaustion abort.
4. Tests: extend `remote-build.test.mjs`; run the workstation test suite.
5. Install to `~/.claude/lib/`, run a real `local-gate` dispatch as proof.
6. Land + deploy.

## Doctor baseline (measured 2026-08-12)

All three hosts PASS ssh, tool parity, rsync and seat-GC checks, then FAIL late with
`ssh unreachable` + `step=doctor-host timed out after 30s` at elapsed **52.8s / 53.8s / 80.2s**.
The hosts are healthy; `doctor_host_timeout_sec: 30` is below the audit's real cost, and
exhaustion surfaced as a phantom "unreachable". Same defect class as the dispatch budget.
Host selection (`chooseHost`) probes every host for readiness (cold cache ⇒ runner+supervisor
deploy), job count and telemetry — that sweep is what consumed the 120s dispatch budget before
`git-overlay-list` ever ran.

## Changes made

- `stepTimeoutMs` floor `1ms → MIN_STEP_BUDGET_MS (1000ms)`; `ssh()`/`sh()` abort with
  `dispatch budget exhausted before step=<step>` instead of issuing a phantom sub-second timeout.
- `sh()` honours an explicitly requested transfer budget (only the dispatch deadline may
  shorten it) instead of capping it at `local_step_timeout_sec`.
- `runRemote` runs mirror sync (`syncPush`) on a config with no `dispatch_deadline_ms` —
  data plane on rsync/`ssh_exec` budgets, control plane on the dispatch deadline.
- `dispatch_timeout_sec 120 → 300`, `doctor_host_timeout_sec 30 → 180` (measured 80.2s worst
  host) in `DEFAULT_REMOTE_CONFIG` and both `build-remote.json` copies.
- Doctor clamp test fixture `123456 → 86_400_000` so it asserts the clamp invariant
  independently of the configured host budget.
- New regressions in `remote-build.test.mjs`: budget-exhaustion abort (both branches) and
  mirror-sync deadline exemption.

## Current receipt

2026-08-12: candidate complete in `.worktrees/localgate-dispatch`; `remote-build.test.mjs` green
including two new regressions. Related suites running. `~/.claude/build-remote.json` already
carries the raised budgets, and a subsequent `node --test` was observed dispatching successfully
to a buildbox mirror (`/home/user/builds/localgate-dispatch-*`) — remote dispatch is working
again. Library not yet installed.

## Next executable action

On green: back up and install `lib/remote-build.mjs` to `~/.claude/lib/`, prove through the real
`local-gate` entrypoint, then land + deploy.