# Factory on k3s

audience: AI coding agents first.

status: IDLE
worker: none; requested commit complete, not landed or deployed
source request update (2026-08-14): Fix direct-GitHub k3s Factory result transport on current main plus `c5da638e03762c905cd36a7fb141f16f8012217d` if needed. Remove base64 Git-bundle bytes from Kubernetes logs. Use shared `emptyDir` result volume plus restricted non-root resource-bounded sidecar; worker writes `result.bundle` and completion marker; trusted submitter waits for worker termination/marker, copies exact bytes through `kubectl exec`, validates bundle ancestry, and deletes Job with bounded timeout/failure cleanup. Keep controller-only publication. Do not add Gitea, proxy, object store, encryption, or credential gates. Focused tests MUST prove logs contain no bundle bytes/marker and exact handoff; full Factory suite MUST pass; commit with terse message.
source request update (2026-08-13, coordinator): Land candidate `fc6adff0e4edb819a4dee9822a0eeae3548e29f9` end-to-end through canonical `.claude/scripts/ship.sh`; integrate current GitHub `main`; rerun affected focused/full/deploy gates; deploy landed source through canonical lock; prove installed launcher provenance, cleanup service/timer, cleanup entrypoint, smallest installed seam, and zero residual resources/refs/mailboxes; update plan/index to DONE with receipts.
source request update (2026-08-13, coordinator): Repair canonical deployment robustly: stage and validate `buildbox-hosts.json` before atomic replacement; preserve valid existing registry when candidate source is absent; fail closed without destructive deletion; test missing source, preservation, invalid replacement, atomic success, and rollback. Install local-infra fix live first, run canonical deployment, prove registry/provenance/cleanup units, then complete harmless real Factory-k3s vertical slice with credential/ref/PVC/resource cleanup receipts. Run relevant deploy tests plus focused launcher and full 174-test Factory suite with zero warnings. Commit only; never merge/push.
worker prior state: exact candidate launcher installed, but canonical deployment is blocked before cleanup unit installation by destructive `buildbox-hosts.json` sync ordering; live Job intentionally not started without cleanup safety
source request update (2026-08-13): Resume install/live proof from verified healthy source `/home/user/Projects/overdeck/.git/healthy-sources/candidate-f451ca73.git` at exact commit `f451ca73fb0f98a156ce63603b4ff16415fa9c40`; never use poisoned canonical source; run canonical `packaging/deploy-local.sh` from clean no-local clone; prove installed provenance, cleanup units, harmless real vertical slice, credential/ref restrictions, visible state, and complete resource/ref/mailbox cleanup; do not push or merge candidate to main.
worker prior receipt: exact candidate cannot be installed because repository object `44742e6abe0970bb913ceb3b69516292776c7b1b` is missing, preventing a clean isolated deploy clone; live cluster API discovery also fails
task IDs: #96, #175
source request: Replace hand-rolled schedulers with k3s without stopping customer work. k3s is an incremental migration lane, never a global delivery gate.
review requests:
- Verifier audit of `factory-agent-k3s.mjs`: assess whether created input, result, and recovery refs lack deletion and permanently retain WIP/session data. Candidate must receive a code-cited CONFIRMED/PLAUSIBLE/REFUTED verdict.
- Verifier-only audit: assess whether crafted direct `FACTORY_AGENT_ATTEMPT_ID` reaches `writeRecovery` filename without sanitization, traverses `STATE_DIR`, and overwrites a writable `.json` path. Inspect script plus caller/attempt-ID source; default `PLAUSIBLE`; assess realistic reachability; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.
- Verifier-only audit: assess whether pod `runAsUser/runAsGroup: 1000` can write plain `emptyDir` workspace without `fsGroup`; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.
- Verifier-only audit: check claim that `factory-agent-k3s.mjs` line 204 invokes `git push --no-verify`; inspect exact target and classify `CONFIRMED`/`PLAUSIBLE`/`REFUTED` against the governing no-bypass rule.
- Verifier-only audit: compare `factory-agent-k3s.mjs` `buildFactoryJobManifest` with sibling `k3s-remote-build.mjs` `buildJobManifest`; determine whether duplicated lifecycle/security/resource boilerplate can drift; return code-cited CONFIRMED/PLAUSIBLE/REFUTED only.
- Verifier-only credential-boundary audit: assess whether host `GH_TOKEN` enters the untrusted coding container with repository/private-repository power beyond publishing generated refs, enabling prompt-injection exfiltration or arbitrary-ref pushes. Inspect credential scope and caller context; default `PLAUSIBLE`; return code-cited `CONFIRMED`/`PLAUSIBLE`/`REFUTED` only.

## Outcome

One real Factory task completes through Kubernetes: submit → restricted Job → immutable input → authoritative status → Git result ref → safe result application → visible success/failure. Migrate additional job classes only after this vertical slice works.

## Acceptance criteria

- Immutable GHCR image by digest.
- Ephemeral Git input/result refs.
- Credentials via Kubernetes Secret.
- Lifecycle/status through Kubernetes API.
- Restricted namespace, pod, RBAC, resources, and network policy.
- One installed Factory entrypoint launches and observes a real Job.
- Result application validates repository/ref/worktree safety.
- Failure is owner-visible and fail loud.
- Every ref created for an attempt has a bounded lifecycle: cleanup runs after all pre-submit, observation, result-application, and recovery-recording failures; intentionally retained recovery data has an explicit expiry/acknowledged-consumption deletion path.
- Direct-GitHub result transport never writes repository content or completion-marker content to container logs.
- Worker writes `result.bundle` plus completion marker to shared `emptyDir`; restricted sidecar remains available for trusted `kubectl exec` byte copy after worker termination.
- Trusted submitter requires worker termination and marker, copies exact bundle bytes from sidecar, validates bundle ancestry before controller-only publication, then deletes Job.
- Result handoff, timeout, failure, and cleanup are bounded; sidecar runs non-root with locked security context and small resource requests/limits.
- Focused tests prove forbidden log transport is absent and exact file handoff is present; mandatory full Factory suite passes.

## Preserved WIP

Worktree: `/home/user/Projects/overdeck/.worktrees/factory-k3s-repair`

Files:

- `modules/workstation/claude/bin/k3s-toolchain-provision`
- `modules/workstation/claude/test/k3s-toolchain-provision.test.sh`
- `modules/workstation/claude/lib/factory-agent-k3s.mjs`
- `modules/workstation/claude/tests/factory-agent-k3s.test.mjs`
- `modules/workstation/claude/bin/factory-result-apply.py`
- `modules/harness/factory/adw_modules/agent_pi.py`
- `modules/harness/factory/bin/factory`
- `modules/sandbox/image/Containerfile`
- `packaging/deploy-local.sh`
- `lib/deckctl/sync.sh`
- `tests/os/deckctl-sync.test.sh`
- Verifier finding: `factory-agent-k3s.mjs` has normal-path ref deletion, but cleanup is not guaranteed on every fatal path and failure recovery refs deliberately remain without a bounded deletion workflow.

Receipts already obtained:

- Provisioner focused tests: 7/7 passed remotely after correcting the stale SSH mock's return contract.
- Launcher focused tests: 4/4 passed remotely.
- Cluster lists Debian1/2/3.
- Caller can create Job/Secret and read pods in `overdeck-builds`.
- GHCR image digest exists: `sha256:8b5a6220094dec6c8c07932b09345b09df4ed16e96f904117787b22d69e5b6f7`, tag `factory-latest`.
- `gh` auth has `repo` + `read:packages`, not `write:packages`.
- Local Podman probe failed before start with exit 125; never repair/use local Podman for this lane.
- Full Factory suite was blocked because remote Python lacked pytest; provision dependency in immutable image/toolchain, never laptop.

## WIP defects to replace, not preserve

Independent verifier finding: `createSecret` duplicates `required` status handling (`factory-agent-k3s.mjs:33-35` vs `86-90`). It currently omits `error?.message` fallback and a reusable operation label, so command-failure reporting can drift as either path changes. Verdict: CONFIRMED; centralize with `required(kube(...), "create credential Secret")` while preserving Secret-specific semantics.

Independent verifier finding: image digest policy is duplicated and diverged. `validateImageDigest` (`factory-agent-k3s.mjs:49-55`) rejects registry ports and all `localhost/` references; sibling `immutableImage` (`k3s-remote-build.mjs:64-70`) permits registry ports and does not reject localhost. Verdict: CONFIRMED; consolidate one policy/parser and align tests.

Independent verifier finding: host `GH_TOKEN`/`GITHUB_TOKEN` (or `gh auth token`) is copied into the Job Secret (`factory-agent-k3s.mjs:201-212`, `534-535`) and injected into both containers as `GH_TOKEN` (`:405-414`, `:438-457`). The untrusted agent runs with it and configures Git to present it to any remote (`:359-379`); no code limits server-side token scope or ref targets. Existing receipt confirms `gh` auth has `repo` scope. Verdict: CONFIRMED; replace this credential with a short-lived per-attempt GitHub App installation token enforced to the generated refs, or move result publication to a trusted controller that verifies a signed artifact.

Completed independent review `a3ba147e42b5b9056` confirmed additional ship blockers:

- Generated argv invokes Pi twice: `agentArgv()` includes `pi`, then the shell executes `pi "$@"` (`factory-agent-k3s.mjs:377`, `:484`).
- UID/GID 1000 containers mount plain root-owned `emptyDir` volumes without pod `fsGroup`, so workspace/session initialization can fail permission denied (`:437`, `:470`).
- External extension paths remain host paths and do not exist in the pod (`:487`).
- Blob-based result application loses executable modes and rename-source deletion semantics (`:231`, `:242`).
- Job observation emits no heartbeat/event stream and installs no cancellation handler; supervisor termination can orphan a running Job (`:536`).
- Init-container failure can produce no terminated `agent` state; current status handling exits 125 before recovery metadata/cleanup.
- Result application has a TOCTOU window between `guardLocalEdits` and final Python application; preserve per-operation local-edit protection through the atomic apply boundary.

Maintenance-only findings MUST NOT displace blockers: duplicated container security literals, one-use credential helper, dead `rev-parse` output, duplicated atomic-write/status/image/manifest policies, and repeated path canonicalization.

Current launcher violates required architecture through:

- SSH workspace fan-out/fallback;
- `hostPath`;
- node affinity;
- runtime PATH mutation;
- local-only image reference;
- pod-log result streaming.

Namespace lacks ResourceQuota, LimitRange, NetworkPolicy, and image-pull Secret configuration.

Preserve useful Secret, Job identity, result-safety, and recovery contracts while replacing transport.

## Forbidden architecture

- No SSH workspace fan-out.
- No hostPath.
- No node affinity.
- No manual per-node image distribution.
- No runtime PATH surgery.
- No ConfigMap workspace archive.
- No pod-log result transport.

## Constraints

- Cleanup MUST NOT delete a ref still required for recovery before durable recovery metadata is atomically recorded.
- Recovery ref retention MUST be bounded by explicit acknowledgement/expiry cleanup; process termination alone MUST NOT decide retention.

## Execution steps

1. Replace workspace transport with ephemeral Git input ref.
2. Pin image digest and configure pull credentials.
3. Publish result through ephemeral Git result ref.
4. Replace log-based status/results with Kubernetes API state plus explicit result metadata.
5. Make ref lifecycle total: cleanup input/result refs on every pre-submit, observation, and result-application exit; retain failed recovery ref only under explicit expiry/acknowledgement cleanup.
6. Add namespace/pod/RBAC/resource/network restrictions.
7. Update focused tests before broad suite.
8. Run focused tests through SSH remote-only executor.
9. Install live entrypoints.
10. Launch one real Factory task and record full receipt.
11. Land/deploy exact delta.
12. Migrate job classes incrementally: test → build → browser → coding → privileged.

## Current receipt

2026-08-13 current receipt supersedes older receipts below. Healthy bare source `/home/user/Projects/overdeck/.git/healthy-sources/candidate-f451ca73.git` passed exact commit and connectivity checks for `f451ca73fb0f98a156ce63603b4ff16415fa9c40`. Clean no-local liveproof clone: `/home/user/Projects/overdeck/.worktrees/factory-k3s-liveproof-f451ca73`.

Canonical registry deployment is repaired and installed. Candidate registry is validated before mutation; missing/invalid candidates preserve the valid live registry; successful replacement uses atomic symlink rename and keeps a snapshot backup. Live registry stayed byte-identical at SHA-256 `61cd599c5987be4b738b00e450d66ebd5df1a7ead25d44c799f55f0f37829082`. Regression suite: 69/69. Fleet transport suite: 23/23. `engine.mjs` diagnostic was checked against this delta: `existsSync` remains used at lines 127, 267, and 363; no introduced unused import exists.

Installed launcher resolves to this liveproof clone and matches source byte-for-byte at SHA-256 `d045ee414e236a8362fdd862bc891eb8fa7c8c3c7318e9103e7b68afa799f2d7`. Installed symlink execution is fixed and tested. `factory-k3s-cleanup.timer` is loaded, enabled, active/waiting. `factory-k3s-cleanup.service` is loaded and its last run completed successfully. `overdeck-ghcr-pull` exists; mailbox and Job manifests use it. Pods remain credential-free for Git: only selected Pi provider auth/models enter the immutable Secret; no repository URL, GitHub token, credential helper, service-account token, or arbitrary-ref authority enters the pod. Trusted host alone validates, publishes, applies, and deletes launcher-derived refs.

Live canary reached the Kubernetes scheduler with immutable image `ghcr.io/alexcodeplace/overdeck-agent-sandbox@sha256:8b5a6220094dec6c8c07932b09345b09df4ed16e96f904117787b22d69e5b6f7`. Image-pull authentication is fixed. Current attempt could not schedule because all Ready nodes are tainted `node.kubernetes.io/disk-pressure:NoSchedule`: Debian1 since 2026-08-11T16:49:41Z, Debian2 since 2026-08-12T20:15:41Z, Debian3 since 2026-08-12T15:10:18Z. No broad toleration was added.

Exact cleanup receipt: unrecoverable old attempts `3ac8b810683d96f6d5e2` and `8d89fec198551f7cca03` were matched against recorded names/UIDs/refs before deletion; absent resources/refs counted as idempotent success; only their mailbox/state/lock artifacts were removed. Unrelated CLEANED receipt `c850b50ac5321fd15300` was preserved. No Factory Jobs, Pods, or attempt Secrets remain. Exact orphan PVCs `fa-3ac8b810683d96f6d5e2-49ee40edc0ea3a08-mailbox` UID `ad8c2d88-612f-4c56-93ba-743b7931b7ca` and `fa-8d89fec198551f7cca03-28a93f5393d0652a-mailbox` UID `500fdeb2-3979-418c-984b-d9bf52d61bc2` accepted deletion at `2026-08-12T20:22:29Z` and remain terminating under `kubernetes.io/pvc-protection`; no force/finalizer bypass used. Generated result/recovery refs are absent. Canary worktree remains unchanged.

Final gates after all repairs: deploy sync 69/69, fleet 23/23, focused launcher 32/32, mandatory Factory 174/174, `git diff --check`; zero warnings.

PVC-mailbox/trusted-host publisher repair is verified in `/home/user/Projects/overdeck/.worktrees/factory-k3s-repair`. Coding pods receive no reusable GitHub/Git credential, repository URL, credential helper, publisher sidecar, service-account token, or arbitrary push path. Trusted host verifies signed bundles and publishes only launcher-derived result/recovery refs after untrusted pods are deleted.

Crash recovery is state-aware and lease-safe. Heartbeats renew durable ownership; live owner processes, active Jobs, and unknown Kubernetes states cannot be cleaned by a second launcher. Terminal `RUNNING` attempts resume extraction. `RESULT_READY`/`RECOVERY_READY` resume verification. `PUBLISHING` resumes fixed-ref publication. `PUBLISHED` resumes apply/recovery recording and exact ref cleanup. Recovery metadata is durable before PVC deletion. Successful ref deletion failures and signal cleanup failures create expiring retry receipts; only the selected failure-recovery ref is retained. Random execution generations keep repeated identical commands independent while preserving external attempt metadata.

Final receipts: 19/19 focused launcher tests passed without warnings; mandatory Factory suite 174/174 passed in 98.28 seconds without warnings; Node syntax checks, exact ESLint unused-declaration check, credential/dead-symbol search, and `git diff --check` passed. Tests cover every persisted state, separate-process live concurrency, repeated identical commands, HMAC tampering, credential denial, real publish/apply/delete with modes and deletions, durable deletion retry, and signal recovery-ref selection. Nothing was merged, pushed, installed, deployed, or cluster-proven, per coordinator scope.

Landing-preparation receipt: fetched `origin/main` and merged it cleanly as `ec6d191a1`; no conflict resolution or post-review production/test/unit/deploy code change was required. Branch is 0 behind and 8 commits ahead of `origin/main`. Net delta remains seven intended files. Focused launcher tests pass 31/31 in 22.01 seconds; mandatory Factory suite passes 174/174 in 144.49 seconds. Node and shell syntax, isolated systemd service/timer verification, exact ESLint `no-unused-vars:error`, forbidden credential/bypass/host-path string scan, post-review code-drift check, and `git diff --check` pass. Direct host `systemd-analyze verify` also inspected an unrelated distro unit with a removed `CPUAccounting=` option and could not resolve the intentionally uninstalled `/home/user/.claude/lib/factory-agent-k3s.mjs`; isolated verification replaced only that runtime path with `/usr/bin/true` and passed cleanly. Nothing was pushed, merged to main, installed, or deployed.

Earlier credential-boundary audit follows for history:

Credential-boundary follow-up inspected current `origin/main` and live GitHub repository configuration. Existing controller-owned dark delivery verifies signed rollout evidence only; it has no Git publication proxy, installation-token minting, or exact-ref admission path. GitHub evidence: active CLI credential is a reusable classic token with `repo` scope; repository App-installation lookup returns HTTP 401; deploy-key list is empty; Actions environment list is empty; repository secret and variable lists are empty; source search found no App token minting or trusted publisher; repository ruleset API returns HTTP 403 because rulesets are unavailable for this private repository on the current plan. A GitHub App installation token would restrict repository/permissions but not exact refs, so it alone cannot satisfy server-side generated-ref enforcement. Smallest missing prerequisite: deploy a trusted controller publication endpoint holding repository write authority and accepting a short-lived signed per-attempt envelope bound to repository plus exact input/result/recovery refs, or enable a Git host/ruleset mechanism that enforces those exact refs for a dedicated actor.

Local review fixes are implemented: removed unused `mkdtempSync`, `rmSync`, `RESULT_APPLY`, `lstatSync`, and `realpathSync`; removed dead staged/Python application path; production `applyResultRef` now fetches and verifies the advertised commit, rejects symlink/tree modes, preserves executable modes and deletions, rejects renames/traversal/protected paths, and rechecks local edits immediately before apply. Failure recovery now preserves the exact recorded result/recovery ref. Transient ref deletion failures create expiring durable cleanup receipts, and expiry processing retries deletion before removing receipts. Coding container denial tests prove no credential environment, credential mount, helper, or arbitrary push command. Focused suite passes 13/13 with no warnings. Node syntax and `git diff --check` pass. Mandatory Factory suite was attempted twice after the correction but the execution environment timed out before a terminal verdict; first attempt also repaired a stale virtual-environment link and emitted that environment warning. Last clean terminal full-suite receipt remains 174/174 from the preceding commit; it is not claimed for this corrected delta.

Earlier candidate receipt follows for history:

Git-ref transport candidate now exists in the preserved worktree: immutable GHCR digest, ephemeral input/result/recovery refs, `emptyDir`, Kubernetes API polling, and no SSH/hostPath/node-affinity/log-result path. Seat-local focused tests reported 4/4; remote verification was not obtained because the seat lacked SSH credentials. Independent review confirmed `applyStagedResult` is test-only dead code; production applies through `factory-result-apply.py`, so the JS symlink/traversal test does not cover the live applier. Attempt-ID traversal verification blocked: mandated Codex run exited 12 because Debian1's sandbox launcher started no container; log: `/home/user/.local/state/overdeck/systray/runtime/logs/cdx/20260810-120217-1281668.log`. Ref-lifecycle verifier result: PLAUSIBLE. Normal-path deletion exists at `deleteEphemeralRefs`/`deleteRemoteRef` (`factory-agent-k3s.mjs:306-315`, invoked at `541`), so “no deletion path” is false. But `publishInputRef` precedes Job creation (`509`); `fail()` exits without `finally` (`23-25`), stranding refs after rejection or later fatal paths. Failed recovery refs are intentionally omitted from deletion (`309-310`) and have no explicit expiry/acknowledgement cleanup. Result-apply efficiency claim REFUTED: `classifyResultChanges` (`factory-agent-k3s.mjs:269-278`) runs four buffered `git diff` commands (R/T/D/ACM), while local-edit guards (`:246-267`) and staging (`:280-287`) can add buffered `git show` calls per changed/deleted file. One-use `gitCredentialSecret` helper is a confirmed simplification candidate: it is defined at `factory-agent-k3s.mjs:210-212` and called once at `:535`; inline `{ "GH_TOKEN": resolveGitToken() }` preserves semantics with less indirection. Image-parser drift verifier result: CONFIRMED. `validateImageDigest` (`factory-agent-k3s.mjs:49-55`) rejects `registry:port/...@sha256:...` and `localhost/...`, while sibling `immutableImage` (`k3s-remote-build.mjs:64-70`) accepts the former and does not reject the latter. Nothing is installed, landed, deployed, or cluster-proven.

Independent credential-boundary verifier result: CONFIRMED. `resolveGitToken` accepts host `GH_TOKEN`/`GITHUB_TOKEN` or `gh auth token` (`factory-agent-k3s.mjs:201-212`); `main` writes it into the Job Secret (`:534-535`); the manifest exposes it as `GH_TOKEN` to init and agent containers (`:405-414`, `:438-457`); agent Git uses it for arbitrary remote operations (`:359-379`). No scope/ref restriction exists in the reviewed path. Required next implementation: remove host credential from the pod and publish through a trusted controller or a short-lived, per-attempt, repository/ref-restricted credential.

Manifest-drift verifier result: CONFIRMED. `buildFactoryJobManifest` (`factory-agent-k3s.mjs:424-477`) duplicates lifecycle/security/resource boilerplate independently defined by `buildJobManifest` (`k3s-remote-build.mjs:137-190`), with no shared policy/helper. Divergence is already present: factory hard-codes timeout/TTL/resources/image pull policy, while sibling validates bounds and accepts options/env (`k3s-remote-build.mjs:108-114`, `:156`, `:174`, `:179-182`); lifecycle cleanup is also separate (`factory-agent-k3s.mjs:541` vs `k3s-remote-build.mjs:353-370`).

No-bypass verifier result: REFUTED for inspected WIP. `factory-agent-k3s.mjs:204` is credential lookup, and push argv at `:217`/`:307` contains no `--no-verify`; governing prohibition remains applicable if a different revision contains that flag.

Final repair receipt: all six re-review blockers are implemented in the preserved worktree. Cancellation preserves resumable terminal artifacts; launcher/recovery use exclusive attempt locks, atomic guarded transitions, lease renewal, and live Kubernetes checks; `APPLYING`/`APPLIED` resume verifies the exact result; independent timer cleanup records bounded per-resource retries for Jobs, Pods, PVCs, Secrets, refs, and mailboxes; Git quarantine enforces bundle, object-count, individual-object, and expanded-total limits; signing keys are deleted after verification and abandoned mailboxes are pruned by age/size without touching active attempts. Focused launcher tests pass 24/24; mandatory Factory suite passes 174/174; Node syntax and `git diff --check` pass without warnings under `LC_ALL=C`. No merge, push, install, deploy, or cluster mutation occurred.

Second independent-review receipt: unacknowledged/unexpired recovery refs, PVCs, and mailboxes are retained until acknowledgement or expiry; live attempt locks are skipped while malformed locks still fail; terminal extraction intent is durable before Job deletion; failed/exhausted cleanup remains owner-visible and keeps scheduled capped-backoff retries; local deployment installs and enables the cleanup timer using existing unit ownership conventions; repository mutation holds a canonical lock through final guard, apply, verification, and durable `APPLIED`, with exact reverse-patch rollback on state-write or verification failure. Focused tests pass 26/26 and mandatory Factory tests pass 174/174. Node syntax, shell syntax, warning, credential/dead-symbol, security-string, and diff checks pass. No merge, push, install, deploy, or cluster mutation occurred.

Final blocker receipt: signal cleanup now treats durable `TERMINAL` as artifact-bearing, preserving `extraction_pending` for restart extraction. Recovery acknowledgement has a production `--ack-recovery <attempt-id>` path. Acknowledgement and expiry only mark durable metadata; attempt cleanup is the sole ref-deletion owner, treats an absent remote ref as success, writes `CLEANED` before atomically removing matching recovery metadata, and remains idempotent after a crash or repeated timer run. Unacknowledged/unexpired recoveries remain retained. Focused tests pass 28/28 and mandatory Factory tests pass 174/174. Node syntax, shell/deployment syntax, warning, credential/dead-symbol, security-string, and diff checks pass. No merge, push, install, deploy, or cluster mutation occurred.

Post-review lifecycle receipt: recovery-expiry no longer accepts an unused repository parameter and every callsite is clean. `TERMINAL` restart uses persisted exit/extraction identity and creates the launcher-owned PVC extraction Pod without querying the original Job-owned Pod, so original Pod garbage collection cannot destroy recovery. Recovery resource cleanup now persists `CLEANUP_FINALIZING` before metadata removal; restart/timer resumes finalization, metadata deletion failure remains visible/retryable, and only a completed metadata deletion permits `CLEANED`. Exact ESLint `no-unused-vars:error` passes. Focused tests pass 28/28 and mandatory Factory tests pass 174/174; Node/shell/deploy syntax, warning, security, credential/dead-symbol, and diff checks pass. No merge, push, install, deploy, or cluster mutation occurred.

Final production-path receipt: persisted `TERMINAL` attempts with a terminal or missing Job always resume launcher-owned PVC extraction using the durable exit code; production reconciliation never cleans them solely because the Job disappeared. Normal launcher reconciliation leaves `CLEANUP_FINALIZING` untouched, while the scheduled cleanup worker removes matching recovery metadata and converges to `CLEANED` across restart/repeated timer runs. Every production expiry invocation is now zero-argument, so repository paths cannot be interpreted as clocks; a production `--cleanup` test proves future unacknowledged recovery remains retained. Exact ESLint unused-declaration and expiry call-signature checks pass. Focused tests pass 31/31 and mandatory Factory tests pass 174/174; Node/shell/deploy syntax, warning, security, credential/dead-symbol, and diff checks pass. No merge, push, install, deploy, or cluster mutation occurred.

2026-08-13 terminal receipt: chunked extraction now validates Pod/source allowlists, size plus SHA-256 framing, 256 KiB bounded chunks, three idempotent retries, wrapped-base64 decoding, reconstructed digest, and atomic local replacement. Regression suite passes 33/33; mandatory Factory 174/174; deployment 69/69; syntax, unit verification, and diff checks clean. Installed candidate hash: `088c0bc9ace533dc9be16b8cc02d53ed87aaac6d93ad6eb5a41021eb031a3024`.

Preserved attempt `6ecb134183bc95c85b7c` transferred and hash-verified its 229,898,991-byte historical bundle. Trusted validation correctly rejected protected path `.rb-origin`; nothing was published or applied. Lifecycle repair now records validation failure and schedules exact cleanup rather than looping. A subsequent reconciliation transfer reached byte 125,829,120 before mailbox Pod UID `ee2324a3-9b64-40f7-b604-a92064d17927` was evicted at `2026-08-13T05:08:38Z`: Debian1 ephemeral-storage threshold 5,895,516,247 bytes, available 5,800,592 KiB; container usage 12 KiB. PVC `fa-6ecb134183bc95c85b7c-ab1b7b9b4c704304-mailbox` remains Bound; durable attempt remains preserved; no result/recovery ref exists; canary worktree unchanged. Cleanup timer remains active but latest service execution drifted to status 203 after concurrent installed-path removal; tested candidate symlink was restored.

2026-08-13 emergency runtime receipt: canonical manifest owns `claude:lib` from `/home/user/.local/share/overdeck/deploy`; concurrent deployment reconverged `/home/user/.claude/lib` there, where unlanded `factory-agent-k3s.mjs` was absent, causing cleanup service exit 203. Under `/home/user/.local/share/overdeck/deploy.lock`, exact candidate bytes were atomically installed at canonical target. Source, canonical target, and installed path SHA-256: `088c0bc9ace533dc9be16b8cc02d53ed87aaac6d93ad6eb5a41021eb031a3024`. Real installed `--cleanup` exits 0; cleanup service `Result=success`, `ExecMainStatus=0`; timer enabled and active. Preserved PVC UID `96dc20d1-2d54-4295-a950-43bc7b6eeaed` remained Bound.

Extraction Pod eviction diagnosis: all result bytes live on PVC; mailbox container consumed only 12 KiB node ephemeral storage but declared zero request. Kubernetes selected it during node pressure. Add explicit bounded `ephemeral-storage` request/limit to mailbox Pod; NEVER move artifacts to node-local storage, add toleration, or pin node.

2026-08-13 terminal success receipt: canonical installed launcher/source SHA-256 `530a2389865041f9288035ea7fb20b2e088e109b5a58157aee00d824583a9581`. Extraction Pod declares `ephemeral-storage` request `16Mi`, limit `64Mi`; artifacts remain PVC-only. Worker excludes exact transport metadata `.rb-origin` and `.rb-epoch` from result commits. Compact result promotion reads signed bundle only after quarantine verification, exact-parent check, object-count/size bounds, and protected-path classification. Every persisted result state converts validation rejection into durable cleanup.

Installed canary attempt `cd511021c308260f5294`, Job `fa-cd511021c308260f5294-7d52cfef41cb6d13`, scheduled normally on Debian3. Immutable image digest remained `sha256:8b5a6220094dec6c8c07932b09345b09df4ed16e96f904117787b22d69e5b6f7`; status `Complete`; fixed result ref `refs/heads/factory-result/cd511021c308260f5294/7d52cfef41cb6d13`; rc 0. Trusted host verified, published, applied allow-empty result, deleted fixed ref, and converged attempt to `CLEANED`. Exact final state: zero labeled Job/Pods/PVC/Secrets, no result/recovery refs, mailbox absent, clean canary worktree. Disposable canary repositories removed.

Final verification: focused launcher 33/33; mandatory Factory 174/174; deployment sync command passed with prior exact total 69/69; Node syntax and `git diff --check` clean. Installed cleanup service `Result=success`, `ExecMainStatus=0`; timer enabled and active. No Git credential, repository URL, credential helper, service-account token, arbitrary publication authority, node pinning, or toleration entered Pod manifests. Nothing pushed or merged.

2026-08-13 landing receipt: assigned clean detached worktree matched candidate `fc6adff0e4edb819a4dee9822a0eeae3548e29f9`. Its isolated `origin` intentionally points to healthy candidate storage, so current GitHub `main` `11a299417bd30a5c3d09d1e43c6de3cfa161b51e` was fetched into separate `github/main`. Merge commit `e368df06` preserved both sides of the only production conflict in `lib/deckctl/sync.sh`; variable naming follows current main with unchanged registry-source behavior. Current main added earlier registry backups, exposing a test ordering assumption; `tests/os/deckctl-sync.test.sh` now finds the exact preserved symlink rather than selecting a directory-order result. Clean post-merge gates under `LC_ALL=C`: focused launcher 33/33; mandatory Factory 174/174; deployment sync 70/70; shell syntax and `git diff --check` clean.

2026-08-13 final landing/deploy receipt: canonical ship workflow landed candidate as GitHub `main` commit `8ee207afea966cb0ff2b560559ee6433632b3ebc`; candidate gates completed without reported failure or warning. Initial post-land deploy correctly refused an abandoned broken staging symlink; exact orphan was verified ownerless and removed. Deploy then exposed missing recorded sandbox images on all three build boxes. Canonical provision initially failed on Debian1 because empty Podman metadata retained 11 GB orphan overlay storage and the filesystem had only 5.3 GB free. Debian1 had zero containers/images/volumes; safe Podman reset reclaimed storage to 19 GB free. Canonical all-node provision completed and installed expected `localhost/overdeck-agent-sandbox:d1c7f17660b1` on Debian1/2/3. Debian minimal-image `update-alternatives` skipped absent manual-page links; executable installation and image builds succeeded, so warnings are benign upstream packaging output. Final `packaging/deploy-local.sh` exited 0. Unrelated shim-drift output names preserved edits in other sessions; live shims match `origin/main`, so no foreign WIP was changed.

Installed proof: concurrent authorized landings advanced deployed HEAD to `58a3c3a8f4c33dedbb94a72830bd5b07ca4e6e68`; landed Factory commit `8ee207afea966cb0ff2b560559ee6433632b3ebc` is its verified ancestor. Candidate, deployed source, and `/home/user/.claude/lib/factory-agent-k3s.mjs` all hash to `530a2389865041f9288035ea7fb20b2e088e109b5a58157aee00d824583a9581`; installed path resolves to deployed source. Installed `--cleanup` exits 0. Cleanup service is loaded with `Result=success`, `ExecMainStatus=0`; timer is loaded, enabled, and active. Exact configured-cluster query returns rc 0 with zero managed Jobs/Pods/PVCs/Secrets. Remote factory input/result/recovery refs: zero. Local Factory mailbox/recovery state files: zero. Existing Debian3 canary receipt remains rc 0 with trusted publication/application and `CLEANED`. Final gates: focused launcher 33/33; mandatory Factory 174/174; deploy sync 70/70; shell syntax and `git diff --check` clean.

2026-08-14 direct-GitHub transport repair: applied required production/test/runtime delta from `c5da638e03762c905cd36a7fb141f16f8012217d` onto current main, then removed log transport. Worker atomically writes `/result/result.bundle` and empty `/result/complete`; restricted non-root `result-reader` sidecar mounts only bounded `emptyDir` result volume read-only and has 1m/8Mi requests, 10m/16Mi limits. Trusted submitter waits for exact worker termination plus marker, copies raw bytes with bounded `kubectl exec`, verifies bundle and input ancestry before controller-only publication, and deletes Job in bounded cleanup. Worker logs contain neither bundle encoding nor marker output. Focused transport tests: 23/23 in 20.70s. Static compile and `git diff --check`: clean. Mandatory Factory suite: 189/189 in 393.90s. Verified delta committed; authoritative SHA is the plan file's containing commit.

## Next executable action

Land/deploy only if a later owner request expands the explicit commit-only scope.

## 2026-08-16 k3s Factory image-refresh receipt

The published Factory Kubernetes image is refreshable without hand-editing the
admission policy. The credentialed orchestrator procedure is: build the GHCR
image from the repository's `modules` build context; push it and record the
returned immutable `sha256:<64 lowercase hex>` digest; run
`modules/harness/factory/kubernetes/update-approved-image.sh <digest>`; land
the resulting pin change; run `kubectl apply -f
modules/harness/factory/kubernetes/runtime.yaml`; then resubmit the Factory
run.

Today's drift lesson: landing the new digest is not a completed pin update.
The cluster keeps the prior ValidatingAdmissionPolicy until `kubectl apply`
runs. That landed-but-not-applied state caused the `workload containers are not
approved` denial, so application of `runtime.yaml` is mandatory before
resubmission.

## Automated image refresh — ratified 2026-08-16, not yet built

Owner ruling: the manual refresh chain above becomes a deploy stage. Rationale:
two of 2026-08-16's failures were the same disease — an artifact lagging its
landed source (runtime.yaml landed but never applied; image factory predating
`--skip-plan`). Automation kills that class.

Design (composes existing machinery):

1. **Trigger = content hash, already solved.** The sandbox image build already
   computes a build-context content hash (`sandbox_image_context_hash`, used by
   `sandbox-provision --check` for converged-vs-drifted). The deploy pipeline
   already classifies changes (docs-only detector). Compose them: deploy ran +
   factory/image context hash changed since the last pushed digest → rebuild.
2. **Credential locality.** The deploy runs on the laptop where the GHCR token
   legitimately lives server-side; the push never nears a sandbox — consistent
   with registry-gateway doctrine.
3. **Apply stops being forgettable.** build → push → `update-approved-image.sh
   <digest>` → land → `kubectl apply -f runtime.yaml` becomes a pipeline stage,
   so landed-but-not-applied becomes impossible.

Design constraints (caveats to build around, not skip):

- **Pin-commit loop termination.** The digest lands in source (APPROVED_IMAGE +
  runtime.yaml pins); deploy→land→deploy is a cycle. The context hash makes it
  converge (second pass sees an unchanged context, does nothing) — but the
  termination gets a TEST, not an assumption.
- **Build from landed main only, never the working tree.** An auto-built
  trusted image from a dirty tree would be the deploy-clone bug's worst form.
- **Semantic shift, stated deliberately:** the digest pin stops meaning "a
  human-audited image" and starts meaning "whatever gated main produces."
  Consistent with the trust model (main is the gated trunk), but it is a real
  change and is recorded here on purpose.

End-state: "two factory codebases" collapses into one codebase plus one
artifact that tracks it within a deploy cycle.