# Factory pi agent processes run on the buildboxes

audience: AI coding agents first.

## Outcome

Factory pi agent processes (the coding-agent subprocess `agent_pi.py` spawns) run on a
buildbox instead of the workstation, so more factory runs fit off-laptop, without touching
pi's protocol, observability, or gates. Opt-in via config, PI_PATH override still works,
default stays local pi.

## Status

IDLE — code landed to `main` (two commits: `55c6e167d` wrapper+config+devtools entry,
`7439a7845` fixed a bin-path collision the first commit caused). Post-land deploy did
not complete this session: the shared deploy queue was congested by concurrent lanes
(observed `~20` queued deploy-queue requests from other sessions at land time; two
attempts, one 600s and one 1700s, both timed out with zero output, consistent with lock
contention rather than a defect in this change). Per the "proof path broken ⇒ land, name
the unverified step" rule, code is landed and this is reported honestly rather than
fabricated. Live proof against a real pi turn also stays BLOCKED on `buildbox bootstrap`
actually converging the fleet (this session's devtools.json entry only declares the
target state — nobody has run bootstrap against it yet).

## Session receipt (this session's implementation pass)

- Implementation delegated to `cdx exec -m gpt-5.6-terra -c model_reasoning_effort=medium`
  per owner's explicit mid-task directive; reviewed, fixed, and integrated by hand
  afterward (see "review findings" below).
- Files created: `modules/workstation/bin/pi-remote`,
  `modules/workstation/bin/pi_remote_dispatch.py`, `modules/systray/containment.py`,
  `modules/systray/tests/test_pi_remote_dispatch.py`.
- Files changed: `modules/systray/remote_dispatch.py` (+`extra_rel_dir`/`sync_extra_dir`
  helpers), `modules/systray/command_router.py` (containment logic extracted to shared
  `containment.py`, used by both cdx/cld and pi-remote), `modules/harness/factory/adw_modules/{agent_pi.py,agents.py,data_types.py}`
  (`spawn_path` param + `pi_spawn` config key, `PI_PATH` env still wins),
  `modules/harness/factory/sssf.config.yaml` (`pi_spawn: local` default),
  `modules/buildbox/devtools.json` (`pi` vendor entry, npm:@earendil-works/pi-coding-agent
  @0.80.3 — matches the exact version installed on this workstation, verified via
  `pi --version`), `modules/buildbox/bin/buildbox` (`push_pi_auth` alongside
  `push_npmrc_token`, converges `~/.pi/agent/auth.json` from the workstation on
  `bootstrap` — only pushes when the workstation's own auth.json carries a real
  `openai-codex` credential, never overwrites with an unauthenticated file),
  `modules/buildbox/test/vendor.test.sh` (+1 check for the pi vendor entry),
  `modules/harness/factory/tests/{test_config_layering.py,test_agent_gate_retry.py}`
  (+2 tests: config key layers correctly; `pi_spawn: remote` threads `agents.PI_REMOTE`
  into `agent_pi.run`'s `spawn_path`).

### Review findings — 2 real bugs in codex's draft, both fixed and empirically re-verified

1. **containment mode mapping regression**: codex's extracted `containment.append_record`
   only mapped `mode == "container"` to `containment: "podman-rootless"`; the original
   inline logic in `command_router.py` mapped every mode except `aborted`/`info-only`
   to `podman-rootless` (covers `in-container` too). Caught by
   `modules/systray/tests/test_command_router.py::test_every_agent_start_path_is_contained_or_aborts[cdx_exec_in_container]`
   going red — this is a real regression the test suite exists to catch. Fixed to
   restore the original mapping (`aborted`/`info-only` → `none`, `remote` → `ssh-agent-seat`,
   else → `podman-rootless`). Reverified: `test_command_router.py` 131/131 green.
2. **remote-kill mechanism was two separate broken designs stacked**: the draft used
   `ssh -tt` (a pty) plus a remote `trap ... HUP TERM` wrapper. Live-tested both
   independently on debian1 (verified reachable, agent-seat role): (a) `ssh -tt` merges
   the remote command's stdout AND stderr into one pty stream — confirmed by writing to
   each separately and observing both land in the local stdout pipe. `agent_pi.py`'s
   entire protocol depends on stdout(JSON events)/stderr(diagnostics) staying separate;
   this would have silently corrupted every remote pi run. (b) the `-tt` was seemingly
   there to make SIGTERM propagate, but tested standalone without `-tt`: a local SIGTERM
   to the ssh client does NOT kill the remote command (verified: `sleep 60` on the
   remote side outlived the local ssh client by 20+ seconds after SIGTERM). Replaced
   both with a remote pidfile (`echo $$ > <pidfile> && exec pi ...`) plus an explicit
   `ssh host kill -TERM <pid>` issued from the wrapper's own signal handler — no pty,
   stdout/stderr stay separate, and the kill is a positive proof (`ssh ... pgrep`) not
   an assumption. Live-tested end to end on debian1 with a fake `pi` stand-in (real pi
   is not yet on the fleet — see below): confirmed line-by-line stdout streaming
   (2 lines observed 1s apart), confirmed the remote process pid landed in the pidfile,
   confirmed `kill -TERM <remote-pid>` via ssh actually terminated the remote process
   (`pgrep` empty after, non-empty before), confirmed session-dir mirror pull-back
   succeeded with no error.
3. **ssh trailing-argv rejoin bug** (found during the live test above, not present in the
   original draft's "safe" calls but was in the pi-launch line): OpenSSH joins ALL
   trailing command-line arguments with plain spaces before handing them to the remote
   shell — passing `["bash", "-c", command]` as three separate Popen argv elements
   re-splits `command` on ITS OWN internal spaces instead of preserving it as one `-c`
   argument (empirically reproduced: `mkdir -p X && cd Y ...` arrived at the remote
   shell as `mkdir` with `-p` as import argv, splitting mid-command and erroring
   `mkdir: missing operand`). Fixed by wrapping the whole command in a single
   `shlex.quote`d string (`bash -c '<command>'` as ONE list element), matching the
   pattern the codebase already established in `remote_dispatch.git_pointer_command`.
   Re-verified live after the fix: full push→launch→stream→kill→pull cycle passes.
4. **post-land deploy bug (found and fixed in a second commit, `7439a7845`)**: the pi
   devtools.json vendor entry's `bins.pi` declared destination `~/.local/bin/pi`, which
   collides with the workstation's own manifest-owned symlink (`modules/workstation/manifest/bin.json`
   → `modules/workstation/bin/pi`, the account-routing wrapper that resolves the real
   pi via `~/.npm-global/bin/pi` and syncs settings through deckctl). The workstation
   fleet node runs BOTH the `shared-agent-tools` (devtools.json) and `workstation`
   (agent-home manifests) profiles simultaneously (`modules/fleet/fleet.json`), so both
   declarations landed on the identical destination path and `expand.mjs`'s
   `rejectDuplicateDestinations` correctly refused to apply either — first surfaced as
   a real `deckctl sync apply` failure during the first `bash packaging/deploy-local.sh`
   run after landing. Fixed by renaming the fleet-installed binary's PATH name to
   `pi-agent` (`bins: { "pi-agent": "~/.local/share/mise/shims/pi" }`,
   `version_cmd: "pi-agent --version"`) so it never touches `~/.local/bin/pi`; updated
   `pi_remote_dispatch.py`'s remote launch command to `exec pi-agent` accordingly.
   Re-verified: `vendor.test.sh` 14/14 green after the rename.

### Test counts

- `python3 -m pytest modules/harness/factory/tests/ -q`: 189 baseline (this session's
  own re-verified count, not the briefed 186) → **191/191 green** after (+2 new tests).
- `python3 -m pytest modules/systray/tests/ -q`: 1023 passed, 1 skipped, 2 failed — both
  failures pre-existing and unrelated (`test_account_lock.py::test_store_rejects_malformed_or_unsafe_state`
  fails on umask/tmp-dir permissions in this sandbox environment;
  `test_packaging_desktop_entry.py::test_cinnamon_autostart_desktop_entry_is_templated_and_installed`
  fails on fixture bin-directory ordering — confirmed via `git diff --name-only` that
  neither touched file is part of this change, and confirmed the containment-mapping
  regression (the one failure this change DID cause) is fixed and green).
- `bash modules/buildbox/test/run-all.sh`: 19/21 passed, 1 not_run, 1 failed
  (`node-runtime.smoke.sh` — a pre-existing node v22 vs v24 drift on a seat image,
  confirmed unrelated: no node entries in the devtools.json diff). `vendor.test.sh`
  14/14 green including the new pi-entry check.
- `bash modules/buildbox/test/vendor.test.sh` standalone: 14/14.

### Live end-to-end proof performed (with a fake `pi` stand-in, since real pi is not yet
### on the fleet — this is NOT the item-6 live-proof the plan requires, it proves the
### wrapper's OWN mechanics are correct, independent of pi itself)

Ran the actual `push`/`launch`/`stream`/`kill`/`pull` sequence from
`pi_remote_dispatch.py` against debian1 with a disposable non-fleet-managed script
standing in for `pi` (placed outside `~/.local/bin` in a private scratch dir so it never
touched real fleet state, and removed afterward — `~/.local/bin/pi` was NEVER written on
any box). Confirmed: git-tracked-file + working-tree mirror push succeeds; the
`--session-dir` extra-directory mirror push/pull succeeds; the remote process launches
and streams stdout line-by-line with real inter-event latency preserved (not buffered);
a positive remote-pid kill terminates the process (verified via `pgrep`, not assumed);
stdout and stderr remain separate channels throughout (no `-tt` pty). All scratch
artifacts on debian1 were removed after the test; `~/.overdeck/containment.jsonl` /
`~/.local/state/overdeck/containment.jsonl` on the workstation were not touched by the
live test (it called the mirror primitives directly, not the full `pi-remote` CLI
entrypoint, to avoid writing a spurious containment record for a fake binary).

### What is still NOT verified (honest gap, not faked)

- A real `pi` agent turn through `pi-remote` end to end — blocked on `pi` not yet being
  installed+authenticated on any buildbox. This lands with this change
  (`devtools.json` vendor entry + `push_pi_auth`), but a `buildbox bootstrap <host>`
  run (which this session did not trigger — it converges the whole fleet's declared
  state, a broader action than this task's scope) and a subsequent real
  `python3 -m pytest`-free smoke run of `pi-remote -p ... --session-id ...` against an
  authenticated remote pi are the concrete next steps.
- The full `pi-remote` CLI entrypoint's own containment-record writing was exercised
  only by unit tests (mocked dispatch) and code review, not a live run, since a live run
  requires the real pi binary this session confirmed is not yet on any box.

## Source request

Owner (relayed via orchestrator), 2026-08-14: "make factory pi agent processes run on the
buildboxes instead of the workstation, WITHOUT changing the factory's pi protocol,
observability, or gates. fewer Claude orchestrators, more factory runs, all off-laptop."
Full task text preserved in orchestrator transcript; key facts and design constraints below.

## Facts verified this session

- `modules/harness/factory/adw_modules/agent_pi.py::run()` spawns `[PI_PATH, ...]` via
  `subprocess.Popen` locally, streams stdout JSONL line-by-line to the tracer/UI. `PI_PATH`
  env var already exists (default `"pi"`), so *some* remote path is pluggable without
  touching this file.
- `PiRequest.session_dir` and `PiRequest.raw_output_path` are **absolute laptop paths** under
  `data_dir` (`/home/user/.local/state/overdeck/factory/data/sessions/<adw_id>/<agent>/...`),
  NOT inside the git-mirrored repo tree. `remote_dispatch.open_session` mirrors only the git
  checkout (`repo_root`) — it has no notion of `data_dir`. A naive `pi-remote` wrapper that
  just runs `pi` on a buildbox via ssh with `cwd` mirrored will point `--session-dir` at a
  node path that does not exist, and will need `data_dir` (or at least the one agent's
  session subdir + raw_output path) synced both ways per call, since `agents.py::execute`
  reads `run.session_dir`/agent_dir contents (prompts, raw_output.jsonl) after each send.
  Confirmed by live test: pi session continuity is keyed by `--session-id` + `--session-dir`
  contents, NOT by `cwd` — cwd changing between calls does not break continuity, only
  session-dir content does. So the wrapper's real job is: mirror cwd (repo) AND mirror
  session_dir/raw_output_path (data_dir) both ways per pi invocation — remote_dispatch's
  `Session.pull_back()` only pulls the repo tree back, not arbitrary out-of-tree paths.
- pi is **not installed** on debian1 (bare host: `pi: command not found`; `~/.pi/agent` has
  only an empty `{}` auth.json placeholder, no models.json, no binary) and **not installed**
  inside the sandbox container either (`agent-sandbox --host debian1 -- pi --version` →
  "pi: command not found — recorded as a sandbox tool gap"). `modules/sandbox/image/Containerfile`
  bakes in codex via a pinned GitHub release tarball; it has no pi install step. codex IS
  present in the container (`codex-cli 0.146.0`) because it's baked into the image.
  `~/.pi/agent/auth.json` present on debian1 as an empty `{}` is stale/irrelevant — nothing
  populates it; systray's `pi_auth_sync.py` only runs on the workstation today and is not
  wired to sync to boxes.
- Parity mechanism for third-party tools reaching the boxes: `modules/buildbox/devtools.json`
  (git-repo tools pinned by ref, vendor tarballs pinned by version, e.g. codex is a `vendor`
  entry of `kind: mise`) + `modules/buildbox/lib/claude-parity.sh`, converged by
  `buildbox-parity.timer` → `buildbox bootstrap <host>`. Installing pi on the boxes belongs
  here as a new `vendor` entry (pi ships as an npm package `@earendil-works/pi-coding-agent`,
  workstation installs it via `npm-global`) — NOT hand-scp.
- `remote_dispatch.py` module docstring names 3 known mirror limitations: (1) node-side
  commits come home as uncommitted working-tree changes (only tree mirrored, not
  refs/objects — except linked-worktree gitdir mirroring added later, which DOES mirror
  refs/objects, so this limitation is about the *plain-checkout* path, not the worktree
  path); (2) sibling worktree dispatch clobbers shared remote refs; (3) out-of-tree
  `core.hooksPath` doesn't resolve remotely, so hooks don't run there.
  Cross-checked against factory's actual git/commit behavior: `runner.py::commit_all` calls
  `git_helper.commit_all` **laptop-side**, after a phase's agent call returns — the pi
  subprocess itself never commits (`agent_pi.py` has no git invocation). So limitation (1)/(2)
  do not apply: the factory's commits always happen on the workstation's own checkout, never
  inside the remote mirror, so there's nothing to clobber or lose. Limitation (3) (hooks) is
  irrelevant to a pi subprocess running noninteractively via `-p --mode json` with no git
  hooks in its path. **Verdict: none of the 3 known mirror limitations break factory
  semantics for the *pi-subprocess-only* offload described here** (agent_pi spawns pi,
  reads its stdout, no git). The NEW risk this task introduces (session_dir/raw_output_path
  living outside repo_root) is not one of the 3 documented limitations — it's a new one
  specific to the pi protocol's session model, and IS a real footgun for a naive wrapper.
- `select_host`/`open_session` costs multiple ssh round trips (probe, mkdir, rsync push,
  possibly git-mirror prep) per session open — currently amortized by cdx's per-command
  granularity (one codex CLI invocation is a whole coding session). agent_pi.py's `run()` is
  called possibly many times per phase (initial send + gate-retry corrections, same session
  id) — opening a fresh remote_dispatch session per `run()` call would multiply mirror
  round-trip cost per phase. A `pi-remote` wrapper invoked once per `PI_PATH` subprocess spawn
  (i.e., once per `run()` call, matching today's local-pi granularity) is consistent with
  today's factory calling convention and keeps agent_pi.py untouched — cost is one open_session
  amortized per pi turn, not per adw_id, matching existing cdx-per-invocation cost profile.

## Design (binding shape; codex/gpt-5.6-terra implements the details)

1. `modules/workstation/bin/pi-remote` — thin CLI: `pi-remote [pi-args...]`. Internally:
   resolve `cwd` (default `os.getcwd()`, honor `-C`-less pi invocation semantics — pi has no
   `-C` flag, so cwd is simply the wrapper's own cwd, forwarded via `remote_dispatch`'s
   existing `find_workdir`/no-workdir-flag path, i.e. treat pi like a tool with no workdir
   flag: mirror `repo_root(cwd)`, run in `container_workspace(root)`).
   MUST additionally mirror `--session-dir` and the raw-output parent dir (both come from
   `PiRequest`, forwarded as pi CLI flags already) round-trip: push before exec, pull back
   after exec, keyed the same way `push_tracked_argv`/`pull_argv` do it today but pointed at
   the out-of-tree data_dir path instead of `root`. This is new code — remote_dispatch has no
   existing helper for a second out-of-tree directory; extend narrowly (e.g. a second rsync
   pair using the same `access`/`ssh_argv` machinery, not a new architecture).
   Fail CLOSED: registry unreadable / no reachable host / mirror push or pull failure → clear
   nonzero exit (reuse `remote_dispatch.EXIT_*` codes) with detail to stderr, NEVER fall back
   to local pi silently.
2. Factory config: new `defaults.pi_spawn` (or similar; final key name is codex's/reviewer's
   call, follow `ConfigDefaults` conventions in `data_types.py`) — default `"local"`, opt-in
   `"remote"` — resolved once per run and threaded to set `PI_PATH` (or an explicit spawn-path
   override in `agent_pi.run`/`agents.py::execute`) to `pi-remote`'s absolute installed path
   when `"remote"`. Config layering per `test_config_layering.py` idiom (central default +
   project overlay). `PI_PATH` env var still wins when set (existing behavior in
   `agent_pi.py`, untouched).
3. Containment record: `pi-remote` calls `remote_dispatch`'s `_append_containment_record`
   equivalent (or the shared helper it wraps) so runs show in `~/.overdeck/containment.jsonl`
   with `tool="pi"` (or `"factory-pi"`), same schema as cdx's records.
4. `agent_pi.py` protocol/tracer/gates/runner commit flow: untouched. Seam is the spawn path
   only (`PI_PATH`), exactly as designed.
5. Tests: `pi-remote` unit tests (fail-closed branches: registry absent, host down, mirror
   push/pull failure) near `modules/systray/tests/test_remote_dispatch*` patterns. One factory
   test proving the config key sets the resolved spawn path (mock `pi-remote`, no real
   dispatch) — must not regress `python3 -m pytest modules/harness/factory/tests/ -q`
   (189/189 baseline recorded this session, not 186 as briefed — reverify after every
   material change).
6. Live proof: BLOCKED until pi is installed + authenticated on at least one buildbox (bare
   host or sandbox image, whichever `pi-remote` targets) — see next executable action.

## Acceptance criteria

- `pi-remote` wrapper: source-controlled, tested, fail-closed (no silent local fallback).
- Factory config opt-in key resolves spawn path; `PI_PATH` env override still works; default
  unchanged (local pi).
- `agent_pi.py` protocol/tracer/gates/commit flow: zero diff.
- `python3 -m pytest modules/harness/factory/tests/ -q` green before and after.
- containment.jsonl gains factory-pi records distinguishable from cdx records.
- Live proof: one minimal factory phase via `pi-remote` completes, containment record names a
  debian host, trace db has the run — OR, if pi-on-box install exceeds this lane, everything
  else lands tested and this file names the exact remaining prerequisite (see below).

## Design decisions resolved this session

- **Target: bare buildbox host over ssh (`agent-seat` role), NOT the podman sandbox
  container.** pi is absent from both, but the container path requires a Containerfile
  change + fleet image rebuild + republish — a second lane the owner did not scope here.
  Bare-host ssh matches how `probe_load`/`ssh_argv` already work in `remote_dispatch` and
  needs only a devtools.json vendor entry + auth sync, not an image bump.
- **Streaming: pi flushes JSONL per-line in real time (confirmed live: first event at
  T+45s during a slow model call, then multiple small writes within the same second).**
  `agent_pi.py`'s tracer/UI observability depends on this. The remote transport MUST NOT
  block-buffer: plain `ssh host 'pi ...'` inherits pi's own line-buffered stdout onto the
  ssh channel unbuffered by default for interactive-shaped pipes — `pi-remote` pipes ssh's
  stdout straight through to its own stdout (no accumulation), preserving line-by-line
  arrival for `agent_pi.py`'s existing `consume_stdout` reader.
- **Process lifecycle / SIGTERM propagation:** `agent_pi.py::run()` already sends
  SIGTERM/SIGKILL to the **local** process group (`kill_process_group(process.pid, ...)`)
  when `pi-remote` is `PI_PATH`. Killing the local ssh client process does not by itself
  kill the remote pi process (ssh without `-tt` does not always propagate SIGHUP/kill to
  the remote command on non-interactive channels). `pi-remote` MUST use `ssh -tt` (or
  equivalent) so a local SIGTERM/SIGKILL to the ssh client tears down the remote pty and
  its child, OR wrap the remote invocation so the remote shell traps and kills pi's pid on
  ssh disconnect (e.g. `ssh ... 'trap "kill -TERM %1 2>/dev/null" HUP TERM EXIT; pi ... & wait'`).
  This is the one place a naive wrapper silently breaks the existing timeout/kill contract
  in `modules/harness/CLAUDE.md` (`starting`/`streaming`/`terminating`/`terminal` states) —
  MUST verify with a real test: start `pi-remote`, SIGTERM it, assert the remote pi process
  is also gone (via ssh `pgrep`).
- **Session-dir / raw-output mirroring is per-invocation, not per-run:** `pi-remote` pushes
  `--session-dir` and the raw-output parent dir before exec and pulls them back after,
  every single call (matches today's local-pi behavior where these are just regular
  filesystem paths pi/agent_pi.py read/write directly — no caching across calls needed,
  correctness over speed; the existing `remote_dispatch` push/pull argv helpers are reused
  for the mechanics, pointed at this second directory pair instead of `root`).
- **Info-only / non-agent invocations (`--list-models`, `--version` probes elsewhere in
  agent_pi.py like `_pi_catalog`, `context_window`) MUST NOT round-trip to a buildbox** —
  `pi-remote` inspects argv and execs local `pi` directly for these (mirrors
  `command_router.py`'s own `is_info_only_invocation` pattern), so `agent_pi.py` stays at
  zero diff even though it calls `pi --list-models` today assuming a fast local call.

## Preserved WIP / next executable action

- Landed to `main`: `55c6e167d` (wrapper + config key + devtools entry + tests) and
  `7439a7845` (fixed the bin-path collision the first commit caused). Both worktrees
  (`wt/factory-pi-remote`, `wt/pi-remote-vendor-fix`) were removed by the lander on
  success, as expected.
- Post-land `bash packaging/deploy-local.sh` did not complete this session — the shared
  deploy queue was congested (≈19-20 queued requests from other concurrent sessions at
  every check); two direct attempts (600s and 1700s budgets) produced zero output and
  no completion, consistent with lock contention rather than a defect in this change. A
  further background attempt was left running past this session's own reporting point.
  **Next executable action: re-run `bash packaging/deploy-local.sh` from
  `/home/user/Projects/overdeck` once the fleet's deploy queue drains** (or just let a
  concurrent session's own deploy pick it up — the change is already merged to `main`,
  so any successful deploy from here forward carries it).
- **After that deploy succeeds:** trigger `buildbox bootstrap` (or wait for the next
  `buildbox-parity.timer` convergence) so `pi`/`pi-agent` and its auth actually reach
  the fleet — only then does item 6's real live proof (one minimal factory phase via
  `pi-remote` against a real, authenticated remote pi) become executable. Until a box
  carries the real `pi-agent` binary, `pi_spawn: remote` stays a project-level opt-in
  nobody has flipped (default remains `local`, zero behavior change to existing runs).

## Constraints carried from owner brief

- Never modify agent_pi.py protocol handling, tracer, gates, runner commit flow.
- Fail closed everywhere; no silent local absorption of load.
- `python3 -m pytest modules/harness/factory/tests/ -q` MUST stay green throughout.
- No hand-scp of unmanaged binaries — installation goes through devtools.json/buildbox
  bootstrap only.
- Land via `.claude/scripts/ship.sh land wt/factory-pi-remote <worktree>` from
  `/home/user/Projects/overdeck`.