# Factory on k3s

audience: AI coding agents first.

status: BLOCKED
worker: none; repair dispatch `bl1io3br2` started no process because Debian3 agent-seat capacity was 3/3
task IDs: #96, #175
source request: Replace hand-rolled schedulers with k3s without stopping customer work. k3s is an incremental migration lane, never a global delivery gate.
review requests:
- Verifier audit of `factory-agent-k3s.mjs`: assess whether created input, result, and recovery refs lack deletion and permanently retain WIP/session data. Candidate must receive a code-cited CONFIRMED/PLAUSIBLE/REFUTED verdict.
- Verifier-only audit: assess whether crafted direct `FACTORY_AGENT_ATTEMPT_ID` reaches `writeRecovery` filename without sanitization, traverses `STATE_DIR`, and overwrites a writable `.json` path. Inspect script plus caller/attempt-ID source; default `PLAUSIBLE`; assess realistic reachability; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.
- Verifier-only audit: assess whether pod `runAsUser/runAsGroup: 1000` can write plain `emptyDir` workspace without `fsGroup`; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.
- Verifier-only audit: check claim that `factory-agent-k3s.mjs` line 204 invokes `git push --no-verify`; inspect exact target and classify `CONFIRMED`/`PLAUSIBLE`/`REFUTED` against the governing no-bypass rule.
- Verifier-only audit: compare `factory-agent-k3s.mjs` `buildFactoryJobManifest` with sibling `k3s-remote-build.mjs` `buildJobManifest`; determine whether duplicated lifecycle/security/resource boilerplate can drift; return code-cited CONFIRMED/PLAUSIBLE/REFUTED only.
- Verifier-only credential-boundary audit: assess whether host `GH_TOKEN` enters the untrusted coding container with repository/private-repository power beyond publishing generated refs, enabling prompt-injection exfiltration or arbitrary-ref pushes. Inspect credential scope and caller context; default `PLAUSIBLE`; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.

## Outcome

One real Factory task completes through Kubernetes: submit → restricted Job → immutable input → authoritative status → Git result ref → safe result application → visible success/failure. Migrate additional job classes only after this vertical slice works.

## Acceptance criteria

- Immutable GHCR image by digest.
- Ephemeral Git input/result refs.
- Credentials via Kubernetes Secret.
- Lifecycle/status through Kubernetes API.
- Restricted namespace, pod, RBAC, resources, and network policy.
- One installed Factory entrypoint launches and observes a real Job.
- Result application validates repository/ref/worktree safety.
- Failure is owner-visible and fail loud.
- Every ref created for an attempt has a bounded lifecycle: cleanup runs after all pre-submit, observation, result-application, and recovery-recording failures; intentionally retained recovery data has an explicit expiry/acknowledged-consumption deletion path.
- Coding pods receive no reusable host GitHub credential; result publication uses a per-attempt, repository-bound credential restricted server-side to exactly the generated result/recovery refs, and tests prove arbitrary refs and exfiltration paths are denied.

## Preserved WIP

Worktree: `/home/user/Projects/overdeck/.worktrees/factory-agents-k3s`

Files:

- `modules/workstation/claude/bin/k3s-toolchain-provision`
- `modules/workstation/claude/test/k3s-toolchain-provision.test.sh`
- `modules/workstation/claude/lib/factory-agent-k3s.mjs`
- `modules/workstation/claude/tests/factory-agent-k3s.test.mjs`
- `modules/workstation/claude/bin/factory-result-apply.py`
- `modules/harness/factory/adw_modules/agent_pi.py`
- `modules/harness/factory/bin/factory`
- `modules/sandbox/image/Containerfile`
- `packaging/deploy-local.sh`
- `lib/deckctl/sync.sh`
- `tests/os/deckctl-sync.test.sh`
- Verifier finding: `factory-agent-k3s.mjs` has normal-path ref deletion, but cleanup is not guaranteed on every fatal path and failure recovery refs deliberately remain without a bounded deletion workflow.

Receipts already obtained:

- Provisioner focused tests: 7/7 passed remotely after correcting the stale SSH mock's return contract.
- Launcher focused tests: 4/4 passed remotely.
- Cluster lists Debian1/2/3.
- Caller can create Job/Secret and read pods in `overdeck-builds`.
- GHCR image digest exists: `sha256:8b5a6220094dec6c8c07932b09345b09df4ed16e96f904117787b22d69e5b6f7`, tag `factory-latest`.
- `gh` auth has `repo` + `read:packages`, not `write:packages`.
- Local Podman probe failed before start with exit 125; never repair/use local Podman for this lane.
- Full Factory suite was blocked because remote Python lacked pytest; provision dependency in immutable image/toolchain, never laptop.

## WIP defects to replace, not preserve

Independent verifier finding: `createSecret` duplicates `required` status handling (`factory-agent-k3s.mjs:33-35` vs `86-90`). It currently omits `error?.message` fallback and a reusable operation label, so command-failure reporting can drift as either path changes. Verdict: CONFIRMED; centralize with `required(kube(...), "create credential Secret")` while preserving Secret-specific semantics.

Independent verifier finding: image digest policy is duplicated and diverged. `validateImageDigest` (`factory-agent-k3s.mjs:49-55`) rejects registry ports and all `localhost/` references; sibling `immutableImage` (`k3s-remote-build.mjs:64-70`) permits registry ports and does not reject localhost. Verdict: CONFIRMED; consolidate one policy/parser and align tests.

Independent verifier finding: host `GH_TOKEN`/`GITHUB_TOKEN` (or `gh auth token`) is copied into the Job Secret (`factory-agent-k3s.mjs:201-212`, `534-535`) and injected into both containers as `GH_TOKEN` (`:405-414`, `:438-457`). The untrusted agent runs with it and configures Git to present it to any remote (`:359-379`); no code limits server-side token scope or ref targets. Existing receipt confirms `gh` auth has `repo` scope. Verdict: CONFIRMED; replace this credential with a short-lived per-attempt GitHub App installation token enforced to the generated refs, or move result publication to a trusted controller that verifies a signed artifact.

Completed independent review `a3ba147e42b5b9056` confirmed additional ship blockers:

- Generated argv invokes Pi twice: `agentArgv()` includes `pi`, then the shell executes `pi "$@"` (`factory-agent-k3s.mjs:377`, `:484`).
- UID/GID 1000 containers mount plain root-owned `emptyDir` volumes without pod `fsGroup`, so workspace/session initialization can fail permission denied (`:437`, `:470`).
- External extension paths remain host paths and do not exist in the pod (`:487`).
- Blob-based result application loses executable modes and rename-source deletion semantics (`:231`, `:242`).
- Job observation emits no heartbeat/event stream and installs no cancellation handler; supervisor termination can orphan a running Job (`:536`).
- Init-container failure can produce no terminated `agent` state; current status handling exits 125 before recovery metadata/cleanup.
- Result application has a TOCTOU window between `guardLocalEdits` and final Python application; preserve per-operation local-edit protection through the atomic apply boundary.

Maintenance-only findings MUST NOT displace blockers: duplicated container security literals, one-use credential helper, dead `rev-parse` output, duplicated atomic-write/status/image/manifest policies, and repeated path canonicalization.

Current launcher violates required architecture through:

- SSH workspace fan-out/fallback;
- `hostPath`;
- node affinity;
- runtime PATH mutation;
- local-only image reference;
- pod-log result streaming.

Namespace lacks ResourceQuota, LimitRange, NetworkPolicy, and image-pull Secret configuration.

Preserve useful Secret, Job identity, result-safety, and recovery contracts while replacing transport.

## Forbidden architecture

- No SSH workspace fan-out.
- No hostPath.
- No node affinity.
- No manual per-node image distribution.
- No runtime PATH surgery.
- No ConfigMap workspace archive.
- No pod-log result transport.

## Constraints

- Cleanup MUST NOT delete a ref still required for recovery before durable recovery metadata is atomically recorded.
- Recovery ref retention MUST be bounded by explicit acknowledgement/expiry cleanup; process termination alone MUST NOT decide retention.

## Execution steps

1. Replace workspace transport with ephemeral Git input ref.
2. Pin image digest and configure pull credentials.
3. Publish result through ephemeral Git result ref.
4. Replace log-based status/results with Kubernetes API state plus explicit result metadata.
5. Make ref lifecycle total: cleanup input/result refs on every pre-submit, observation, and result-application exit; retain failed recovery ref only under explicit expiry/acknowledgement cleanup.
6. Add namespace/pod/RBAC/resource/network restrictions.
7. Update focused tests before broad suite.
8. Run focused tests through SSH remote-only executor.
9. Install live entrypoints.
10. Launch one real Factory task and record full receipt.
11. Land/deploy exact delta.
12. Migrate job classes incrementally: test → build → browser → coding → privileged.

## Current receipt

Git-ref transport candidate now exists in the preserved worktree: immutable GHCR digest, ephemeral input/result/recovery refs, `emptyDir`, Kubernetes API polling, and no SSH/hostPath/node-affinity/log-result path. Seat-local focused tests reported 4/4; remote verification was not obtained because the seat lacked SSH credentials. Independent review confirmed `applyStagedResult` is test-only dead code; production applies through `factory-result-apply.py`, so the JS symlink/traversal test does not cover the live applier. Attempt-ID traversal verification blocked: mandated Codex run exited 12 because Debian1's sandbox launcher started no container; log: `/home/user/.local/state/overdeck/systray/runtime/logs/cdx/20260810-120217-1281668.log`. Ref-lifecycle verifier result: PLAUSIBLE. Normal-path deletion exists at `deleteEphemeralRefs`/`deleteRemoteRef` (`factory-agent-k3s.mjs:306-315`, invoked at `541`), so “no deletion path” is false. But `publishInputRef` precedes Job creation (`509`); `fail()` exits without `finally` (`23-25`), stranding refs after rejection or later fatal paths. Failed recovery refs are intentionally omitted from deletion (`309-310`) and have no explicit expiry/acknowledgement cleanup. Result-apply efficiency claim REFUTED: `classifyResultChanges` (`factory-agent-k3s.mjs:269-278`) runs four buffered `git diff` commands (R/T/D/ACM), while local-edit guards (`:246-267`) and staging (`:280-287`) can add buffered `git show` calls per changed/deleted file. One-use `gitCredentialSecret` helper is a confirmed simplification candidate: it is defined at `factory-agent-k3s.mjs:210-212` and called once at `:535`; inline `{ "GH_TOKEN": resolveGitToken() }` preserves semantics with less indirection. Image-parser drift verifier result: CONFIRMED. `validateImageDigest` (`factory-agent-k3s.mjs:49-55`) rejects `registry:port/...@sha256:...` and `localhost/...`, while sibling `immutableImage` (`k3s-remote-build.mjs:64-70`) accepts the former and does not reject the latter. Nothing is installed, landed, deployed, or cluster-proven.

Independent credential-boundary verifier result: CONFIRMED. `resolveGitToken` accepts host `GH_TOKEN`/`GITHUB_TOKEN` or `gh auth token` (`factory-agent-k3s.mjs:201-212`); `main` writes it into the Job Secret (`:534-535`); the manifest exposes it as `GH_TOKEN` to init and agent containers (`:405-414`, `:438-457`); agent Git uses it for arbitrary remote operations (`:359-379`). No scope/ref restriction exists in the reviewed path. Required next implementation: remove host credential from the pod and publish through a trusted controller or a short-lived, per-attempt, repository/ref-restricted credential.

Manifest-drift verifier result: CONFIRMED. `buildFactoryJobManifest` (`factory-agent-k3s.mjs:424-477`) duplicates lifecycle/security/resource boilerplate independently defined by `buildJobManifest` (`k3s-remote-build.mjs:137-190`), with no shared policy/helper. Divergence is already present: factory hard-codes timeout/TTL/resources/image pull policy, while sibling validates bounds and accepts options/env (`k3s-remote-build.mjs:108-114`, `:156`, `:174`, `:179-182`); lifecycle cleanup is also separate (`factory-agent-k3s.mjs:541` vs `k3s-remote-build.mjs:353-370`).

No-bypass verifier result: REFUTED for inspected WIP. `factory-agent-k3s.mjs:204` is credential lookup, and push argv at `:217`/`:307` contains no `--no-verify`; governing prohibition remains applicable if a different revision contains that flag.

## Next executable action

Reject the current candidate from installation. Repair in blocker order: remove reusable host GitHub credentials from untrusted pods; correct Pi argv; make mounted volumes writable under non-root policy; transport external extensions; preserve complete Git tree semantics; close local-edit TOCTOU; add streamed heartbeat/events plus signal-safe Job/ref cleanup; handle init-container failure with recovery metadata. Add executable integration regressions for every seam, then run the focused launcher suite through the authoritative remote-only executor before namespace policy/install/real Job work. Consolidation and micro-efficiency findings remain follow-up only unless required by a blocker fix.
