# K3s unpark — owner rulings and execution order

audience: AI coding agents first. status: ACTIVE. worker: session 6c0d0325 (Fable) orchestrating; codex (`-m gpt-5.6-terra -c model_reasoning_effort=medium`) writes all implementation code; probes/tests use `gpt-5.3-codex-spark` (fallback `gpt-5.6-luna/low`) ONLY.

## Owner rulings, 2026-08-16 (these outrank every older record)

1. **The k3s migration was NEVER de-scoped, commit-blocked, or parked by the owner.**
   The "commit only; never merge/push" scope lines and the 2026-08-15 supersession framing
   were Sol's (GPT's) wording, produced while stalling the project for weeks. Treat any
   stored record that narrows k3s scope as suspect unless the owner said it directly.
2. **The owner wants the k3s migration completed.** Fable executes; the prior executor
   failed over days/weeks. Recommendations were given and accepted in-session.
3. **Model policy (restated):** implementation = terra/medium via cdx; ANY probe/test
   through cdx = spark, luna/low fallback, never anything else.

## Current true state (verified in-session, 2026-08-16)

- Factory-on-k3s vertical slice: LANDED (`8ee207afe`) and DEPLOYED; launcher installed
  and hash-verified; cleanup service+timer green; real canary on debian3 ran to CLEANED
  (zero residual Jobs/Pods/PVCs/Secrets/refs); no credential ever entered a pod.
- Factory base default remains `pi_spawn: local`; preset `k3s-spark-xhigh` carries
  `defaults.pi_spawn: remote` (landed `24216debf`). Installed seat convergence is proven.
  Watch run `62f59ca6` completed remotely on debian3 with Spark/xhigh and no local fallback;
  exact receipt is recorded below.
- Factory's earlier restricted-sidecar + `pods/exec` result path is NOT the cdx security
  contract. cdx uses a one-attempt TLS mailbox, suspended Job setup, per-attempt immutable Secret
  and egress policy, controller quarantine/limits/ancestry/strict-fsck, and controller-only local
  import; no Git write credential or `pods/exec` enters the agent path. A digest-pinned worker
  image, immutable cluster prerequisites, node-local egress proxies, and Debian2/3 placement are
  installed. A distinct real Job completed on Debian2 with exact model
  `gpt-5.3-codex-spark`, returned exactly `CDX_K3S_READY`, and never ran podman fallback; see the
  final Step 4 receipt.
- k3s cluster phases 0–2 (config contract, backups, enrollment package) delivered per
  `2026-08-10-k3s-migration-execution.md`; Phase 3 canary enrollment and Phase 7 cutover
  never ran.
- Source through `cfbe67ed7` is on `origin/main`, the deploy clone is pinned to that revision,
  and the installed registry carries the final cdx digest. Full deploy convergence remains red:
  buildbox agent images were built at `6b3ba523e7a2`, but agent/e2e tags still report
  `ee552bd73dc0` on Debian1/2/3. This unrelated image-tag drift does not invalidate the installed
  cdx Kubernetes owner-flow receipt and remains named rather than hidden.

## Execution order (owner-endorsed)

1. **Land the 2026-08-14 transport delta** from `factory-k3s-repair` WIP: rebase onto
   current main, rerun its own gates (focused transport + mandatory factory suite), land.
   Deploy when the deploy clone unblocks.
2. **Remote deny-gate** — spec DONE:
   `docs/specs/2026-08-16-remote-deny-gate-design.md` (box-side git shim parity in
   sandbox/seat surfaces, fail-closed presence check, per-box RED proof receipts).
   Codex implements. Owner reads receipts and clears remote.
3. **Factory preset watch** — preset flip is landed; complete installed seat convergence,
   then run the already-approved watch invocation and record exact receipts.
4. **cdx offload as k8s Jobs** — secure source repair and hermetic tests first; then land,
   install the digest image and cluster prerequisites, and canary against podman. SSH/podman
   fallback retires only in its own final gate.
5. **CI runners via actions-runner-controller** — permanently kills pinned-host "slow
   box" reds. Interim protection ships NOW regardless:
   `docs/specs/2026-08-16-ci-estate-containment-design.md` (ci.slice on every box,
   runner-offline board fires, SCHED_IDLE conformance check). Codex implements.
6. **Interactive sessions last or never** — decided by laptop-as-terminal S5+, not here.

## Also queued (unrelated to k3s, same session)

- Inbound Telegram attachments: spec `2026-08-16-botmaster-inbound-attachments-design.md`;
  Half A dispatch to codex pending dispatch health; foreign WIP in ~/Projects/Botmaster —
  never deploy that worker over it.
- Botmaster delivered work + remaining items: `2026-08-16-botmaster-open-work.md`.

## Standing constraints that bind every step

Worktree isolation; direct-land only with self-owned verification; deploy clone stays
pristine; no transient memory caps on runs; notifications edge-triggered; owner language
on owner surfaces; never silently substitute models; buildboxes are headless, deadman
before risky box changes.

## Step 2 clearance receipts (2026-08-16, session 6c0d0325)

Deny-gate live proofs, image `2b4f3cc0670d`, run inside a real sandbox container per box:

- `{"box":"debian1","kind":"deny-gate-proof","status":"pass","passed":20,"failed":0}`
- `{"box":"debian2","kind":"deny-gate-proof","status":"pass","passed":20,"failed":0}`
- `{"box":"debian3","kind":"deny-gate-proof","status":"pass","passed":20,"failed":0}`

Every DENY rule blocked (force/main/master/non-result pushes, all destructive verbs
outside the workspace) and every ALLOW lane passed (cdx/seat/factory-result/wip refs,
in-workspace resets). Owner clearance of pi_spawn:remote is now an owner decision, not
an engineering gap.

## Step 3 — OWNER CLEARED, FLIPPED (2026-08-16)

Owner ruling, this session, verbatim substance: "The decision is GO. i already made the
decision and it was recorded. its just still not done." The 2026-08-15 remote hold is
LIFTED for factory runs. Flip shipped: `presets/k3s-spark-xhigh.yaml` now carries
`defaults.pi_spawn: remote` — runs launched with that preset execute on the cluster,
every other invocation stays local (the fallback). Factory suite 268/268 after the flip.
Watch period: real preset runs, failures fire the board, widen only after a clean week.

### Watch receipt

- Repair `aed3c53a6` creates linked-worktree mirror destinations before rsync and keeps
  `~/.claude/bin` first while adding `~/.local/bin` to the remote process PATH.
- Focused dispatch tests: `83 passed in 157.82s`; syntax and diff checks passed.
- Mandatory Factory suite was not verified: first gate used the wrong workspace and lacked
  `pydantic`; the corrected remote gate then lost its workstation-only external pytest temp
  parent. Per broken-proof-path discipline, the focused repair landed with this gap named.
- Installed invocation:
  `factory --repo /home/user/Projects/overdeck/.worktrees/cdx-k3s-main --preset k3s-spark-xhigh prompt 'Return exactly FACTORY_REMOTE_WATCH_OK.'`
- ADW `62f59ca6`: 3/3 phases passed; builder model
  `openai-codex/gpt-5.3-codex-spark`, preset thinking `xhigh`; output exactly
  `FACTORY_REMOTE_WATCH_OK`; 2,484,828 tokens; `$0.0000`.
- Containment receipt: `mode=remote`, `containment=ssh-agent-seat`, host `debian3`, exit 0,
  duration 1118.262 seconds. No local fallback ran.
- Builder made one unsolicited marker edit during the probe; controller pullback exposed it,
  and the orchestrator removed it after inspection. The worktree ended clean.

Also ruled by the owner in the same message: a pending owner decision MUST surface as a
`blocked_needs_owner` row so the board's "Needs you" column shows it — prose in a shipped
row's detail is invisible. Board defect filed.

## Step 4 canary receipt (2026-08-17, session 6c0d0325)

- Published worker image:
  `ghcr.io/alexcodeplace/overdeck-agent-sandbox@sha256:cca9d67fc132d9e686841b7c068d37fcefc86cba6a13d0f959f13c462356a8b6`.
  Restricted image checks returned `READY` on Debian2 and Debian3 for both
  `/usr/local/bin/cdx-k3s-worker` and `/usr/local/bin/cdx-result-upload`.
- Cluster prerequisites exist as three immutable Secrets (`cdx-codex-auth`, `cdx-git-read`,
  `overdeck-ghcr-pull`) plus the five-field non-secret registry configuration. Secret values
  were neither read nor copied. Debian1 was cordoned only during canary attempts and is
  schedulable/Ready after cleanup.
- The first `model-test-fixture` invocation returned `CDX_K3S_READY` from the Debian2 podman
  seat. It did not preserve `OD_DISPATCH_K3S` and therefore did not exercise k3s; it is NOT
  acceptance evidence.
- The first direct opt-in stopped before an agent could run because auxiliary resource creation
  did not request JSON from `kubectl`. Its loud, permitted pre-agent fallback ran Spark on
  podman and returned `CDX_K3S_READY`. Repair `acceb81a8` adds `-o json`; focused tests are
  `24 passed`, syntax and diff checks passed, and the installed deploy receipt is
  `status=deployed sha=acceb81a`.
- Broad systray evidence is NOT green: `1116 passed, 1 skipped, 2 failed`. The failures are in
  pre-existing account-lock temp permission and packaging created-file expectations, outside
  the k3s change. They were not hidden or rerun.
- The one repaired real canary created Job
  `cdx-cdx-k3s-main-c4afaa0b98ca-f22e544b9761`. Kubernetes scheduled its Pod on Debian2,
  pulled the exact digest above, and started `fetch-workspace`; that init container exited 128.
  The agent container and `gpt-5.3-codex-spark` never ran, no result bundle arrived, and no
  podman fallback ran. The Job and attempt resources were removed; cluster events and cdx log
  `/home/user/.local/state/overdeck/systray/runtime/logs/cdx/20260817-141845-1464336.log`
  retain the receipt.
- The Job started, so this model canary MUST NOT be retried. Diagnose the Git fetch transport
  from retained evidence or a non-agent transport probe; only then launch a new distinct
  canary under the one-attempt rule. Board receipt: `manual-a6ee9c3c7bf40daf`.
- Git transport repair `3216a8592` routes both cdx and Factory workers through GitHub's
  supported `ssh.github.com:443` endpoint while retaining strict host verification with
  `HostKeyAlias=github.com`. It grants proxy ingress only to Pods matching both namespace
  `overdeck` and labels `overdeck-cdx` / `worker`; no credential scope changed. Independent
  security review found one weak structural assertion, which was strengthened before landing.
- Verification: cdx focused tests `24 passed in 1.06s`; Factory seam test `1 passed in 1.10s`;
  mandatory Factory suite `271 passed in 113.82s`; review follow-up test `1 passed in 1.91s`;
  Python compilation, YAML parsing, and `git diff --check` passed. Merge `4644bead0` is on
  `origin/main`; synchronous deploy returned `status=deployed sha=4644bead`.
- Installed `runtime.yaml` was applied. ConfigMap and narrowly-scoped NetworkPolicy were
  configured, and `overdeck-factory-egress` rolled out successfully so tinyproxy loaded the
  new allowlist.
- The non-model Debian2 Git-read proof did not reach its container: attempt one omitted the
  private image pull Secret; its single remediation named the wrong Secret
  (`cdx-image-pull`, while the installed contract names `overdeck-ghcr-pull`) and also ended
  `ImagePullBackOff`. Both exact temporary Pods and policies were removed. Same-failure-twice
  discipline stopped further attempts. This was a proof-manifest error, not evidence that the
  repaired Git transport failed; later exact transport and model receipts below supersede this
  gap. No model ran, no podman fallback ran, and no credential or Secret value was read or
  printed.

### Final transport and acceptance receipt

- Cross-node Pod overlay access to the shared Factory proxy was not viable. The installed repair
  uses a restricted `overdeck-cdx-egress` DaemonSet on Debian2 and Debian3 plus NodePort `31888`
  with `externalTrafficPolicy: Local`; cdx Jobs have required node affinity to those two nodes.
  Workers derive the node gateway from their own Pod IP and wait for proxy readiness for at most
  20 attempts before failing closed.
- NetworkPolicy propagation produced a real startup race, so readiness runs before Git clone.
  Policy permits both representations of NodePort traffic: pre-DNAT `10.42.0.0/16:31888` and
  post-DNAT selected `overdeck-factory/overdeck-cdx-egress` Pods on `8888`. The TLS result mailbox
  remains a separate exact-host `/32` rule on its exact port. No broad egress was added.
- Independent review found that Jobs could schedule away from proxy-bearing nodes. Required
  Debian2/Debian3 affinity and a regression assertion fixed that blocker before acceptance.
- Exact non-model clone/fetch/commit-identity proof then returned `CDX_GIT_EXACT_FETCH_READY`.
  It exposed root-owned `/workspace` as Git's `dubious ownership` failure class. Source now uses
  command-scoped `git -c safe.directory=/workspace` for init-container fetch/rev-parse/checkout
  and worker result capture; Git safety was not weakened globally.
- The first post-proxy model Job,
  `cdx-cdx-k3s-ready-proof-429a6069a6-66513b63068e`, failed in `fetch-workspace` before a model
  ran. The second distinct Job, `cdx-cdx-k3s-ready-proof-429a6069a6-bd16dd7f1121`, started exact
  model `gpt-5.3-codex-spark` but found the mounted token invalidated and also exposed the worker
  result-capture ownership bug. Neither Job retried through podman, and neither attempt was reused.
- A sanctioned podman Spark credential probe returned exactly `CDX_AUTH_READY`. With no active cdx
  Jobs, the stale immutable `cdx-codex-auth` Secret was replaced from the current credential
  without displaying its value, then restored to `immutable: true`.
- GitHub Actions run `32021611573`
  (`https://github.com/alexcodeplace/overdeck/actions/runs/32021611573`) published worker commit
  `1e80467137fdae72ff55e6106bfff7e351f77f4d` as immutable digest
  `sha256:bd53598a2a671293385b20fab1956cdea35e3c63a77706f3770e011da31aab8a`.
  Restricted pull proofs returned `CDX_IMAGE_DEBIAN2_READY` and `CDX_IMAGE_DEBIAN3_READY`.
- Final distinct Job `cdx-cdx-k3s-ready-proof-429a6069a6-382c0987d665`, Pod
  `cdx-cdx-k3s-ready-proof-429a6069a6-382c0987d665-vlp9m`, ran on Debian2 with that exact digest.
  Kubernetes recorded `Job completed`; cdx exited 0 after 1m10s. Log
  `/home/user/.local/state/overdeck/systray/runtime/logs/cdx/20260817-181400-1274567.log` records
  model `gpt-5.3-codex-spark`, prompt `Return exactly CDX_K3S_READY.`, and response exactly
  `CDX_K3S_READY`. No podman fallback ran.
- Focused final source evidence: worker + dispatch `26 passed in 0.90s`; Factory suite after
  runtime changes `271 passed in 104.34s`; buildbox registry test passed. The five-field non-secret
  registry contract remains `namespace`, `image`, `git_url`, `result_host`, and `proxy_url`.
- Exact cleanup proof after acceptance: `job=0`, `pod=0`, `secret=0`, `networkpolicy=0`, and
  `input_ref=0` for attempt `382c0987d665`. No credential or Secret value was exposed.
- Named remaining gap: full deploy convergence is red because Debian1/2/3 e2e image tags remain
  at `ee552bd73dc0` rather than expected `6b3ba523e7a2`. Source, installed cdx registry, and the
  completed Kubernetes owner flow are independently proven; this unrelated tag drift was not
  represented as green.