# Remote build: delete the scheduler, keep the wrapper

audience: AI coding agents first.

- Status: REJECTED — owner declined 2026-08-15. NEVER launch, NEVER re-propose this outcome; kept as the record of what was decided against.
- Outcome: remote dispatch stops being a hand-rolled distributed scheduler. All three
  buildboxes stay in use, the agent-facing wrapper is unchanged, and the subsystems that
  produce the recurring daily fire are removed rather than repaired.
- Source request (owner, 2026-08-12): "I need speed of going back online, and never having to
  deal with remote build problems again. I am tired from this recurring daily fire." Follow-up:
  "Why can't we use an existing product for this? Which product would you recommend? I need FOSS
  and stable." Owner corrections: three buildboxes must all stay in use; a wrapper is mandatory
  because agents do not type commands correctly.
- Related: [local-gate remote dispatch timeout](2026-08-12-local-gate-dispatch-timeout.md) — the
  fix that ended today's outage. This plan removes the class of defect, not one instance.

## Why no existing product is a drop-in

Requirement is transparent interception: an agent types `pnpm test` in a shell and it executes
on a buildbox, streamed back. FOSS CI (Woodpecker, Drone, Concourse, Jenkins, self-hosted
Actions runners) is branch/webhook-triggered and does not do this — adopting one keeps the
interception layer and adds a daemon, a queue and a pipeline DSL. Buildkite's scheduler is
proprietary SaaS, so it fails the FOSS constraint regardless.

The two FOSS systems that genuinely do remote execution — Bazel/Buck2 + BuildBarn or NativeLink
(REAPI), and Nix remote builders — both require hermetic actions with declared inputs. A
pnpm/turbo monorepo is not that. Weeks of migration; a product decision, not a fix.

Conclusion: the fix is deletion, not adoption. Two mature FOSS tools replace the invented parts.

## What is invented and must go

`modules/workstation/claude/lib/remote-build.mjs` (~1500 lines) became a distributed scheduler
coordinating through JSON files in a state dir across concurrent processes, with no consensus:

- host selection by scoring (readiness probe + runner deploy + job count + load telemetry, per
  host, serially, on every dispatch) — this sweep is the 20–100s of dead time before any work
  starts, and it consumed the dispatch budget that caused today's outage;
- health cache with TTL: a slow host is recorded `ok:false` and then SKIPPED by later
  dispatches, so one slow moment removes a working box from the fleet. This is why
  `--remote-doctor` printed `PASS ssh` and `FAIL ssh unreachable` for the same host in one run;
- sticky-host affinity files, claim files, push-oversize ledger — all state coordinating the
  above.

## Target shape

- **Static assignment, not selection.** Stable hash of repo/worktree name → box. Computed
  locally, no probing. All three boxes carry work; each box keeps the same repo's mirror and
  `node_modules` warm, which today's host-bouncing throws away.
- **Fixed fallback order, one attempt each.** Assigned box does not answer → next box in a
  static list → fail and report. No scoring, no health cache.
- **Continuous file sync** (Mutagen, MIT — or Unison, GPL) replaces per-dispatch staging: the
  tree is already present when the command runs, so there is no push step to time out.
- **Plain `ssh <box> -- cd <dir> && <cmd>`** under SSH's own timeouts.
- **Wrapper unchanged.** Agents keep calling `local-gate`; the interception hook stays exactly
  as it is. Only the machinery behind it shrinks.

Keep: the interception hook, and the git-push-to-bare-mirror staging (sound; may be retired once
sync is continuous).

## Acceptance criteria

- All three buildboxes receive work under the static assignment; measured warm-cache hit rate
  improves over host-bouncing.
- Dispatch overhead before first byte of real work is measured before and after.
- No health cache, no scoring, no claim/affinity/ledger state files remain.
- Agent-facing surface (`local-gate` invocation) is byte-identical for callers.
- Landed behind a flag with the current path intact, so a single switch reverts.
- Owner runs a day on the new path before the old code is deleted.

## Execution steps

1. Measure current dispatch overhead and per-host cache warmth as the baseline.
2. Implement static assignment + fixed fallback behind a flag; keep scoring path selectable.
3. Introduce Mutagen sync per box; prove a dispatch with no staging step.
4. Run both paths side by side for a day; compare overhead and failure count.
5. Delete the scheduler subsystems and their state files; land + deploy.

## Current receipt

2026-08-12: proposed after the dispatch-timeout outage. Owner corrected an earlier one-box
sketch — three boxes stay, and a wrapper is mandatory. Both corrections are folded in above.
Nothing implemented.

## Next executable action

None. Owner declined 2026-08-15 — see Rejection below.

## Rejection (2026-08-15)

Owner declined this proposal. Do not launch it and do not re-propose the same outcome
(replacing the scheduler with static assignment + Mutagen sync). This file is preserved
as the record of what was decided against.