# cdx exec on k3s Jobs — design

audience: AI coding agents first. Board row #manual-a89fa135e91ad309; k3s-unpark step 4
(pulled ahead of step 3; the pi_spawn:remote hold was cleared by the owner's GO on
2026-08-16 and the preset flip is landed at 24216debf).
Implementer: codex (`-m gpt-5.6-terra -c model_reasoning_effort=medium`), two parallel
lanes with disjoint files; main session reviews, lands, deploys, and runs every live
proof (sandbox egress blocks the k3s API, kubectl, ssh, and main-push by design).

## Context

`cdx exec` today: `modules/systray/remote_dispatch.py` picks a box by load-with-claims,
pushes a snapshot commit as a git delta to a bare mirror on the box
(`refs/cdx/<sandbox-id>/in`), runs codex in a rootless podman sandbox via
`agent-sandbox`/`sandbox-run`, force-pushes the result to `refs/cdx/<sandbox-id>/out`
(scratch semantics, `6e6bbe1b1`), fetches it home, and applies it as uncommitted changes
with a refuse-on-local-change guard. The Factory lane already runs k3s Jobs with the full
security model: `modules/harness/factory/adw_modules/kubernetes_job.py` (digest-pinned
image, restricted pod, attempt-bound result mailbox, quarantine validation, and
controller-only publication).

## Contract — Lane 1: the Job submitter (`modules/systray/`)

New `modules/systray/k3s_dispatch.py`, selected inside `remote_dispatch.open_session` by
env flag `OD_DISPATCH_K3S=1` (absent = podman path untouched, byte for byte):

- **Job manifest** (build as data, port the shape from `kubernetes_job.py` — port, don't
  import): digest-pinned sandbox image from the exact approved GHCR repository
  `ghcr.io/alexcodeplace/overdeck-agent-sandbox@sha256:<digest>`; the legacy exact
  `overdeck.k3s.local/overdeck-agent-sandbox@sha256:<digest>` identity remains accepted,
  but is not a publication endpoint. Use the immutable `overdeck-ghcr-pull` Secret and
  prove it exists before Job creation. Set `runAsNonRoot`, drop all capabilities,
  no service account token mount, resource requests/limits (cpu 2/4, memory 8Gi/12Gi to
  match `sandbox-run` defaults), and EXPLICIT `ephemeral-storage` requests/limits
  (128Mi/512Mi) — a mailbox pod was evicted on 2026-08-13 for declaring zero.
- **Input transport unchanged:** the same snapshot commit pushed to the same bare mirror
  ref `refs/cdx/<sandbox-id>/in`; the Job's init step clones/fetches from the mirror path
  on the node (hostPath to the bare mirror is FORBIDDEN by the factory security model —
  use the same in-cluster fetch pattern `kubernetes_job.py` uses for its input ref).
- **Result transport:** NEVER mount a Git write credential or grant `pods/exec` to the
  agent/controller path. Controller starts a one-attempt TLS mailbox, creates an immutable
  owner-referenced upload Secret plus egress NetworkPolicy while the Job is suspended, then
  unsuspends it. Worker writes one bounded thin Git bundle and uploads it once. Controller
  accepts only the matching bearer token, exact path, declared length, and SHA-256; imports
  under `prlimit` into a temporary bare quarantine; bounds object count, individual and
  expanded size; requires input ancestry and strict `fsck`; then imports one local
  `refs/cdx-fetch/<sandbox>` ref and reuses the refuse-on-local-change application semantics.
  Worker publishes no remote ref. Job, attempt Secret, NetworkPolicy, mailbox, and unique input
  ref are cleaned after the attempt; a cleanup failure is reported, never hidden.
- **Credential seam:** codex `auth.json` from a Kubernetes Secret (`cdx-codex-auth`,
  created out-of-band by the main session, never by code in this lane), mounted read-only
  at a LITERAL path, copied into `$HOME` by an init command built ONLY from literals plus
  bare `$HOME` — the podman `$VAR`-mangling class (`a252b5861`) is named prior art: no
  `${...}` forms, no shell variables in container commands. Secret absent → fail closed.
- **Fail-closed fallback:** any k3s-path failure before the agent runs (API unreachable,
  Job rejected, Secret absent, image digest unresolvable) falls back to the podman path
  with the failure REPORTED in the cdx output (`cdx: k3s dispatch failed (<reason>);
  ran on podman`), never silently absorbed. A failure AFTER the agent may have run does
  NOT retry anywhere — report and stop (one attempt per crossing).
- **Placement claims:** k3s scheduling replaces load-probe selection; the placement-claim
  files are NOT written for k3s dispatches (the scheduler owns placement), but
  containment recording and logging stay identical.

**Tests (Lane 1):** hermetic with a fake API server object (no cluster): manifest shape
pinned field by field (security context, ephemeral-storage, no SA token); RED first —
Job rejected → podman fallback with reason in output; Secret absent → same; mailbox result
missing after Job completion → fail loudly, no fallback; malformed, oversized, unrelated,
or non-descendant bundle → reject before controller import; `OD_DISPATCH_K3S` unset → the
podman path's argv is byte-identical to today (regression pin). Run:
`python3 -m pytest modules/systray/tests/ -q` green.

## Contract — Lane 2: hosted image publication

- `.github/workflows/cdx-kubernetes-image.yml` builds
  `modules/sandbox/image/Containerfile` on a GitHub-hosted runner whenever that image
  context or the workflow changes. It publishes only to
  `ghcr.io/alexcodeplace/overdeck-agent-sandbox`, using the workflow-scoped
  `GITHUB_TOKEN`, and reports the immutable digest. K3s Spegel's
  `overdeck.k3s.local` alias is a pull-through mirror, not a writable OCI endpoint;
  NEVER route a push to it or stand up another registry to disguise that mismatch.
- Publish an immutable commit tag plus a convenience `cdx-latest` tag. Runtime config
  MUST use the returned digest, never either tag.
- The host registry's optional `k3s` object is the complete non-secret runtime contract:
  `namespace`, digest-pinned `image`, approved credential-free `git_url`
  (`git@github.com:alexcodeplace/overdeck.git` or its exact `ssh://` form), usable IPv4
  `result_host`, and explicit HTTP `proxy_url` with a usable IPv4 endpoint. Reject
  credentials inside allowed values and reject unspecified, loopback, link-local,
  multicast, or reserved endpoints before creating the Job. Missing, partial, unknown,
  or secret-bearing fields fail before Job creation. Kubernetes Secrets remain
  out-of-band cluster state.
- **Tests:** parse the workflow; validate complete registry shape; reject partial and
  secret-bearing registry objects; accept only the exact approved GHCR repository or
  legacy exact mirror alias with a lowercase SHA-256 digest.

## Out of scope (both lanes)

Retiring podman (its own owner-cleared gate); flipping any default; factory preset
changes (step 3, owner-gated); touching `finish-branch.sh` (another lane holds it);
cluster mutations of any kind from inside a codex run.

## Phase 3 — main session only, after review+land+deploy

Create the `cdx-codex-auth`, read-only input Git, and `overdeck-ghcr-pull`
Secrets by hand; configure the namespace and complete non-secret five-field `k3s`
registry object only after its parser deploys. Capture the hosted image digest, verify
both worker executables from that digest on healthy Debian2/3, then canary N real
`OD_DISPATCH_K3S=1` vs podman incumbents; wall-clock + failure-drill comparison per the
unpark plan; one spark probe (`gpt-5.3-codex-spark`) as the READY check; receipts to the
board row and `docs/plans/2026-08-16-k3s-unpark.md`.