# Factory Reliability

audience: AI coding agents first.

status: DONE
task IDs: #1, #2, #151, #171, #181, #182, #183
source request: Restore Factory functionality, visible progress, remote transport, fail-loud supervision, and safe disk admission. Work may flow through SSH before k3s migration completes. On 2026-08-10 the owner reported a 15-hour run shown as running, ordered it killed immediately, required a direct web UI kill action, required run text to remain selectable for copying, and asked what happened to the redesigned `/factory` plan. Task #151 correction fixes the real-process timeout fixture startup race only: `popen_factory` MUST await bounded child readiness before returning so `agent_pi` wall time starts after leader/child tree readiness. On 2026-08-12 owner said “go”: resume task #2 after preservation-first audit; restore root capacity before Git ceremony; recover only minimal current-main workstation retention behavior; keep snapshot `ed8b2edda` source-only; NEVER push stale combined candidate.

## Outcome

Factory accepts work, routes it remotely, exposes authoritative queued/running/succeeded/failed state, preserves results, and rejects unsafe disk conditions before mutation. UI reflects backend truth.

## Acceptance criteria

- Real task enters Factory and reaches a terminal result.
- No silent no-op or detached unobserved worker.
- Transport preserves extension/tooling inputs and result refs.
- Missing metrics veto admission; never coerce absent to zero.
- Disk rejection names host, measured capacity, threshold, and remediation.
- Factory page displays real state and failure reason.
- Every active run exposes a keyboard-accessible kill action that uses the canonical `factory stop <adw_id>` lifecycle and visibly refreshes to terminal state.
- Run titles, prompts, identifiers, logs, and failure text remain browser-selectable and copyable; row expansion never steals drag selection.
- The redesigned `/factory` plan and any preserved WIP are identified from durable plans, refs, and history; missing or abandoned work is reported explicitly rather than inferred.
- #151 real-process fixture waits on explicit child-readiness signal under fixture-local bounded deadline before returning process; failure raises diagnostic detail. No production timeout semantic change, no scheduler-race sleeps, no broad process kills.

## Preserved WIP and receipts

- #181 destination-symlink fix is preserved; prior rerun blocked by Debian1 ENOSPC.
- #182 prior baseline: 177 tests passed with four unresolved P1 findings. Recover exact findings before changing supervisor behavior; do not repeat broad review.
- #183 WIP preserved. Debian2 previously measured 18,042,288 KiB free.
- #171 Factory page was repaired, but that completion is not authoritative while backend remains offline. Re-prove after execution restoration.
- k3s worktree also modifies `modules/harness/factory/adw_modules/agent_pi.py` and `modules/harness/factory/bin/factory`; reconcile rather than overwrite.

## Constraints

- SSH implementation may remain active until equivalent k3s class is proven.
- Do not delete old transport before live replacement proof.
- Any factory phase/retry/gate/permission/commit behavior change requires full Factory suite remotely.
- Never weaken fail-closed behavior to get a green run.

## Execution steps

1. Recover exact #181/#182/#183 WIP refs and dirty files into receipt.
2. Use SSH remote path for immediate functional recovery where k3s class is unproven.
3. Fix transport with Git-ref model shared by k3s where possible.
4. Close four known supervisor P1s; focused regressions first.
5. #151: inject `popen_factory` fixture that blocks only until child readiness or bounded diagnostic failure, then return process for TERM→grace→KILL/drain verification.
6. Run focused timeout test repeatedly, then exactly one mandatory Factory suite.
7. Install live infra.
8. Run real task and verify Factory UI.
9. Land/deploy exact deltas.

## Current receipt

Task #2 ACTIVE. Canonical deploy at `581833a0` contains landed retention commit `b7189994a`; installed monitor entrypoints and timer resolve to deploy clone. Root later regressed to 8.7 GiB free because safe eligible worktrees accumulated faster than bounded passes reclaimed them. Four additional installed passes reaped 18, then 13, then 0, then 0 worktrees and restored 18 GiB free without force-removal or uncertain-state deletion. This clears the immediate floor but does not yet prove sustained admission under live creation.

Tasks #10 and #11 source landed on `origin/main` through `74b53fea4`. Task #11 is DONE: focused remote-GC regression passed; `rb-mirror-gc` was reinstalled on Debian1/2/3; installed script SHA-256 `3e5ad350854c3509fefae612f958c9573703dbf5d78729cacd31244982fba6c7` exactly matches generated landed source on every host; all timers are enabled and active. Task #10 remains ACTIVE: admission helper SHA-256 `5af5a1a8e014b70f5e99b4f8392ad91ba92f048620c4cdd54941b1b7064f3902` is installed on Debian1/2/3 and each host correctly refuses new work because root free space is below 15 GiB (about 6/10/5 GiB), while scratch remains above floor. Full host hardening and scratch migration remain deliberately unrun because existing migration logic can rename storage still held by active seats. Canonical local deploy remains blocked by unrelated dirty deploy-clone `_human-session` WIP; preserve it unchanged.

Task #1 factory kill/copy UI is complete. Run `b9892096` was already failed in SQLite; canonical stop and reconcile confirmed no live work remained. Reviewed implementation commit `3c68a0ef` and receipt commit `747fe53f` landed on `origin/main`. Required UI gates passed: 19 focused FactoryContent tests, full deck-ui tests/typecheck, web build/typecheck, slopgate, and independent review. The initial web typecheck failure was stale remote source; current-tree rerun confirmed `LinkButton` import and passed with 0 errors/warnings/hints. Emergency install applied only the bounded three-file diff, then landed release `.build-747fe53f2b33e8d3be18f4e46432a93cc155e081-4qzbqj` deployed normally. Live `/factory` returns HTTP 200 and its active assets contain `Stop run` plus `Expand run details`. Deploy printed shim-drift hazards only for unrelated dirty worktrees; live shims match `origin/main`, so those signals are preserved owner WIP and were not modified. The seat bootstrap's detail-free `pnpm install: 1 warnings` was outside the repo command; the actual typecheck reported 0 warnings.

Task #151 completed from `989a17d4`. `test_timeout_kills_group_and_drains_child_streams` passed 10 consecutive runs (3.43–3.57s each). Full Factory suite ran once: `144 passed in 70.61s`. Fixture-local readiness guard waits for explicit child file before `agent_pi` begins wall timing; early leader exit or five-second readiness absence fails with PID/path diagnostics, and timeout cleanup targets only its spawned process group.

## Next executable action

No executable disk-retention action remains. Root currently has 16 GiB free; installed `od-worktree` rejects before mutation below 15 GiB; workstation and buildbox retention timers are armed. Canonical deploy retry remains owned by the unrelated dirty deploy-clone `_human-session` workstream; NEVER modify that WIP from this plan.