# Remote agent seats

audience: AI coding agents first. Imperative, BLUF.

One harness seat = one wrapper invocation = **one rootless podman container on a buildbox**, capped
by the kernel, with the seat's workspace as its only writable host mount. Agent work does not run on
the workstation and cannot rearrange the buildbox.

## Call it the same way you always did

Nothing about `WRAPPER-CONTRACT.md` changes. The orchestrator still invokes the wrapper by path with
the fixed flag set and reads the exit code. The wrapper itself decides to remote, from config:

```
ca.sh|codex.sh|claude.sh  ->  lib/remote-seat.sh (shim)  ->  remote-seat.mjs (laptop)
                    ->  remote-build.mjs transport (ssh, rb-<jobid>.service)
                    ->  seat-run.sh (buildbox)  ->  podman run  ->  seat-entrypoint.sh
                    ->  the SAME wrapper, HARNESS_SEAT_CONTAINER=1, running the engine locally
```

The shim sits **after** arg validation and after `--health` / `--list-models`: bad args still fail
locally with exit 2, and inspection modes never pay an ssh round trip.

Works in **any** repo, not just this one: the prompt and the seat logs ride the transport's
ignored-file overlay, so the launcher appends `/seat-prompt-*.txt` and `.harness-seat/` to the
repo's `.git/info/exclude` (common dir — covers every linked worktree) before writing the prompt.
Nothing is added to the user's tracked `.gitignore`.

## Enable / disable

`seat-remote.json` is the switch — never an env the orchestrator has to set.

- `adapters.<id>: true` → that adapter runs remotely. Remoted today: `codex`, `cursor`, `claude`.
- Remoting is switched off only in `seat-remote.json` / `build-remote.json`; a headless local dispatch is refused with exit 97 by `local-dispatch-guard`.
- `HARNESS_SEAT_CONTAINER=1` → set inside the container; stops infinite re-remoting.

## Exit codes

| situation | rc | why |
|---|---|---|
| wrapper ran in the container | its own rc | pass-through: 0 / 2 / 3 / 75 / 124 |
| host unreachable, push failed, host at its seat limit, image absent, adapter blocked | `3` | engine down — **nothing dispatched**; never a fallback onto the workstation |
| remoting switched off in `build-remote.json` (`enabled:false`, `local_only`) | launcher `70` → wrapper runs locally | a deliberate opt-out, not a failure |
| dispatch started, outcome unknown (ambiguous start, pull failed, epoch mismatch, **buildbox rebooted or crashed mid-run**) | `124` | non-completion — retry once; claiming 3 here would double-dispatch a live seat |

The transport has codes of its own (`254` job-vanished, `143` killed by shutdown, a non-`remote-rc`
classification). None is a WRAPPER-CONTRACT code. `contractRc()` maps every one of them to `124`.
NEVER pass a transport code through to the orchestrator.

## Caps

`agent.slice` on both boxes is MemoryHigh 13.4G / MemoryMax 17.4G. Per seat: 4G memory,
`--memory-swap` equal to it (swap denied), 2 CPUs, 512 pids. `seat-run.sh` refuses a dispatch once
the host already runs `limits.maxConcurrentPerHost` (3) seats, so the aggregate stays at 12G — under
the high watermark. `agent.slice` remains the backstop, but it enforces by SIGKILL, which would
reach the orchestrator as a bogus engine crash; the count gate is what keeps it from firing. The
count and the `podman create` that registers the seat's name are taken under one `flock`, so
simultaneous dispatches cannot both read N-1.

Verified host-side, not from `podman inspect`: a running seat's cgroup is
`…/user@1000.service/agent.slice/libpod-<id>.scope` with `memory.max` set to the seat's limit.

## Surviving a reboot or crash

A seat is a transient systemd unit around a podman container. Neither survives a reboot; a tmux
session would not either. What survives is disk. Recovery = durable seat record + a boot pass that
turns it into a truthful terminal state.

`seat-reconcile.sh` on the box does that pass. Every dispatch re-installs and re-arms it
(`harness-seat-reconcile.service`, `WantedBy=default.target`), so a freshly provisioned box is
covered by its first seat. It runs with no `node` and no `jq` — a systemd user unit has neither.

Per seat from an earlier boot:

1. **Rescue.** Commit the mirror's uncommitted tree to `refs/harness-seat/<runId>`. This is the only
   copy of what the agent had in flight.
2. **Finalize.** Write the seat's rb job `rc=124`, so the laptop's watcher gets a terminal answer
   instead of polling a job that can never finish. Skipped when the job already has an `rc` — a
   graceful shutdown's `143` is a real terminal state, and `contractRc()` maps it to `124` anyway.
3. **Settle.** `phase=rebooted` in the seat record; reap every `harness-seat-*` container.

Plain (non-seat) rb jobs orphaned by the boot get `rc=254`, the runner's own abnormal code.

The laptop then fetches `refs/harness-seat/<runId>` home. It is deliberately NOT merged or checked
out: the run did not finish, and mutating the caller's tree with a half-done agent's edits is worse
than leaving a ref. Recover with `git diff HEAD refs/harness-seat/<runId>`.

### The race the boot unit alone cannot win

The transport destroys a mirror working tree (`git checkout -qf` + `git clean -qffd`) on the next
dispatch, and a dispatch arriving right after boot can beat the unit to it. So the destroyer calls
`seat-reconcile.sh guard <mirrorPath>` from **inside** its own `~/.rb/mat-<mirror>.lock`, and the
guard is fail-closed: no `.reconciled` stamp for this boot ⇒ reconcile now. Rules that keep it
deadlock-free and lossless:

- `guard` MUST NOT take the mirror lock its caller already holds.
- A rescue for any OTHER mirror takes that mirror's lock **non-blocking**. Never wait — a peer may
  hold the two locks in the opposite order.
- A skipped rescue withholds the stamp, so the peer's own guard repeats the pass. NEVER stamp a
  boot whose seats are not all settled.

## Build the image

```
modules/harness/seat/build-image.sh [debian1 debian2] [--force]
```
Builds on the boxes, never locally. Idempotent — skips a host that already has the tag.

## Tests

```
modules/harness/seat/test/seat-contract.test.sh              # local: exit codes, gating, commit replay
modules/harness/seat/test/seat-containment.test.sh debian1   # on a box: kernel caps + mount containment
modules/harness/seat/test/seat-recovery.test.sh debian1      # on a box: reboot recovery, in a sandboxed HOME
```
`seat-recovery.test.sh` fabricates the on-disk shape a reboot leaves behind under a temp `HOME`, so
it proves the reconciler without rebooting. Rebooting a buildbox for real is still the acceptance
test — both a graceful `systemctl reboot` and a hard `sysrq-b` were run against debian2.
The containment test reads cgroup files and `/proc/self/mountinfo` from **inside** the container.
`podman inspect` only replays the flags that were requested and proves nothing.

## What does not move, and why

- **north (`na.sh`)** — requires a local `ccr` proxy daemon (`ccr-up.sh`); `ccr` is not installed on
  either buildbox and the proxy is workstation-local.
- **pi (`pi.sh`)**, **grok (`grok.sh`)** — the `pi` / `grok` binaries are not installed on either
  buildbox; both wrappers exit 3 on their missing-binary precondition.
- **canary (`canary-stub.sh`)** — an offline deterministic fixture; remoting it would add transport
  risk to a test double for no gain.