# Overdeck k3s migration audit

audience: owner and AI coding agents.

date: 2026-08-10

scope: the supplied `overdeck` main-branch archive and the supplied `k3s-phase0-land` worktree archive.

## Executive verdict

The repository is not at one coherent "k3s migration" stage. It contains two related but materially different migration tracks:

1. **Remote-build scheduler migration** in `docs/specs/2026-08-08-k3s-scheduler-design.md` and `modules/workstation/claude/lib/k3s-remote-build.mjs`. This is an incremental compatibility lane around the existing SSH/buildbox execution model. It is implemented enough for focused tests, but is disabled by default and still uses host-coupled transport.
2. **Factory-on-k3s execution** in `docs/plans/2026-08-10-factory-k3s.md` and `modules/harness/factory/**`. This is the target secure execution model. Its own plan marks it `BLOCKED` and forbids the SSH fan-out, `hostPath`, node-affinity, local-image, PATH-mutation, and pod-log result transport used by the earlier compatibility lane.

These tracks can share cluster installation, node enrollment, policy, observability, backup, and upgrade foundations. They MUST NOT share the same workload transport contract. The old scheduler lane may remain a bounded migration adapter; it must not become the Factory architecture.

The first safe step is therefore **not another live installation**. It is to land the supplied WIP safely, collect the actual cluster/host state without modifying it, and use that receipt to generate the first mutating package.

## What was actually attached

### Main snapshot

- Git revision: `d64358e883e4e8a69f5a383a762a736efbab3266`.
- Subject: `Revert "Freeze legacy remote routing"`.
- It contains the scheduler plan, Factory plan, cluster source-of-truth implementation, legacy K3s remote-build launcher/watcher, and the current Factory Kubernetes runtime.

### `k3s-phase0-land` worktree snapshot

Content comparison against the supplied main snapshot found only two modified repository files:

- `modules/workstation/claude/lib/remote-seat-provision.mjs`
- `modules/workstation/claude/tests/remote-seat-provision.test.mjs`

No worktree-only repository file was present. The worktree's `.git` file points to a path on the owner's machine and cannot reconstruct its branch metadata from the archive alone.

The "Preserved WIP" files named by `docs/plans/2026-08-10-factory-k3s.md` are absent from both attached trees, including:

- `modules/workstation/claude/bin/k3s-toolchain-provision`
- `modules/workstation/claude/lib/factory-agent-k3s.mjs`
- `modules/workstation/claude/bin/factory-result-apply.py`

Therefore this package can safely land and audit the supplied phase-zero delta, but it cannot honestly claim to contain or complete the missing Factory launcher worktree.

## Verified implementation status

| Area | Attached code status | Audit classification |
|---|---|---|
| Fleet declaration and loader | `modules/fleet/fleet.json`, loader, expansion, engine, hardening and CLI tests exist | Implemented in snapshot; live drift not yet measured |
| Host access registry | `buildbox-hosts.json` lists Debian1/2/3 and separate access doors | Implemented; live reachability not yet verified |
| Legacy K3s remote-build submitter | `k3s-remote-build.mjs` exists and has focused tests | Implemented behind an off-by-default flag |
| Controller K3s watcher | `controller/src/k3s-watcher.ts` exists | Additive telemetry; not a full scheduler replacement |
| Factory Kubernetes submission | `kubernetes_job.py`, worker, image workflow and runtime manifest exist | Partial vertical slice; plan still records ship blockers |
| Secure Factory namespace policy | Current `runtime.yaml` lacks the complete quota/limit/network/RBAC boundary required by the plan | Incomplete |
| One-command node onboarding | No transactional enrollment command exists | Not implemented |
| Cluster backup/restore contract | No canonical, tested package was found tying datastore backup to the server token and restore proof | Gap |
| Live cluster receipt | Plans contain historical claims, but no current machine-generated receipt was attached | Unverified |

## Supplied WIP review

The supplied change allows a managed runtime copy such as `settings.json` to replace a pre-existing unmanaged file while retaining that unmanaged file as a backup.

The original WIP had a rollback hole: it could move the existing unmanaged file out of the live path before the new copy and provenance record were durably staged. A copy failure or provenance-commit failure could therefore leave the runtime path missing.

Phase 0 repairs that path as a transaction:

1. Stage the new content and provenance fingerprint before moving the live file.
2. Move an existing managed or unmanaged file to a unique backup only after staging succeeds.
3. Publish the new content atomically.
4. Publish provenance atomically.
5. Restore the original file if publication or provenance commit fails.
6. Retain the old unmanaged file only after the complete transaction succeeds.

Failure-injection tests now cover copy failure and provenance-commit failure. The focused remote-seat test passes in the repository's expected `/home/user` environment.

## Correctness holes and required improvements

### Critical — reconcile the two execution architectures

The scheduler design treats SSH-mirrored workspaces and `hostPath` as migration mechanisms. The newer Factory plan explicitly forbids those mechanisms. Combining them would quietly reintroduce host credentials and node coupling into untrusted coding workloads.

**Decision:** maintain two named workload classes during migration:

- `legacy-remote-build`: compatibility-only; bounded canary; no expansion of privileges or job classes.
- `factory-kubernetes`: immutable Git/artifact input, Kubernetes API lifecycle, trusted result publication, no host credential or `hostPath`.

Shared code is limited to safe primitives such as image-digest validation, Kubernetes naming, resource policy, receipt formatting, and cleanup state machines.

### Critical — live receipts are stale until re-proved

The scheduler plan records a server and agent join, but the attached host registry currently marks Debian2 unreachable and the plans describe prior recovery incidents. A dated markdown receipt is evidence of a past state, not proof of the present state.

**Fix:** Phase 0 adds a read-only collector that records repository revision, Tailscale/SSH doors, node/API status, workloads, policy objects, storage metadata, K3s configuration metadata and hashes, fleet audits, and non-zero probes. It deliberately omits Secret values, K3s token contents, process environments, and systemd environment values.

### Critical — Factory credential boundary is not closed

The Factory plan itself records that the preserved launcher copied a reusable host GitHub token into an untrusted coding pod. The acceptance criteria now require a repository- and attempt-bound publication capability or a trusted publisher outside the coding pod.

**Fix:** no Factory live canary may run until result publication is moved behind a trusted controller/publisher or an equivalently constrained, short-lived server-side capability. A pod-visible general `repo` token is not an acceptable temporary shortcut.

### Critical — result/ref lifecycle is not total

The Factory plan records cleanup gaps across pre-submit, observation, application and recovery failures, and retained recovery refs without expiry/acknowledgement deletion.

**Fix:** implement one explicit attempt state machine. Every created input/result/recovery ref is registered immediately; every exit path runs cleanup; retained recovery material gets an expiry plus an acknowledged-consumption deletion path. Cancellation and controller restart are acceptance tests, not follow-up work.

### High — source of truth is split during enrollment

Desired roles/profiles are declared in `modules/fleet/fleet.json`, while concrete identity and access are declared in `modules/workstation/claude/buildbox-hosts.json`. The older Debian3 runbook still tells the operator to edit a removed `hosts[]` field in `build-remote.json`.

**Fix:** onboarding updates the fleet declaration and host registry as one post-proof Git transaction. It MUST NOT publish either declaration before the machine passes identity, recovery-door, K3s Ready, and proof-Job gates. The stale `build-remote.json hosts[]` instructions must be superseded.

### High — installation state needs one canonical configuration contract

A rerun of the K3s installation script can lose options that are not supplied again. Cluster configuration, version, server URL, node identity, labels/taints and registry settings must not exist only in installer command history.

**Fix:** generate canonical root-owned files under `/etc/rancher/k3s/`, pin the approved K3s version, checksum installed content, and make reruns converge those files before invoking installation/restart. Upgrades become a separate cordon/drain/package with rollback.

### High — do not distribute the static server/agent token to new machines

A static agent token does not expire and may be equivalent to the server token unless explicitly separated. Copying it into scripts, logs or repository files would create a durable cluster credential.

**Fix:** the workstation orchestrator asks the server to create a short-lived bootstrap token, transports it through a protected temporary file/stdin, joins one agent, verifies Ready, then deletes/revokes the token. Receipts store only token identifiers and timestamps, never values.

### High — labels and trust must be applied by the control plane

A joining kubelet must not be allowed to self-declare that it is a trusted builder or privileged workload target.

**Fix:** install with only identity/bootstrap labels required to register, verify the Node object against Tailscale and machine identity, then apply protected scheduling labels and taints from the control plane. Phase 1 must verify Node authorization and NodeRestriction behavior rather than assume it.

### High — single-server recovery is not yet an accepted product feature

The existing design places the control plane on Debian3. That can be a valid initial topology, but migration/cutover is unsafe without a current backup, server-token backup, documented restore target, and a restore drill.

**Fix:** Phase 1 establishes datastore/token backup, integrity verification, off-host copy, retention, and an isolated restore proof before adding workload criticality. HA is a later topology decision; backup/restore is not optional meanwhile.

### High — the old Factory hardening branch cannot be merged directly

The supplied Git metadata includes `origin/wt/factory-kubernetes-job`, five commits ahead and 84 commits behind the supplied `origin/main`. It adds useful egress-proxy and policy work, but a direct merge would mix stale base assumptions with current code.

**Fix:** review and forward-port the useful commits/files onto current main in a fresh worktree. Re-run the complete Factory suite. Never merge the stale branch wholesale.

### Medium — fallback retirement is contradictory

The scheduler plan says to delete the hand-rolled placement path at cutover, while its degradation section relies on the same direct SSH path when K3s is unavailable.

**Fix:** split this into two gates:

1. **Default cutover:** K3s becomes default; direct SSH remains an explicit, owner-visible emergency fallback.
2. **Retirement:** delete the fallback only after measured reliability, restore drills and owner acceptance.

### Medium — one-command onboarding needs a failure ledger and rollback

A long shell script that mutates the host, cluster and Git tree without durable step state would merely compress hours of risk into one opaque command.

**Fix:** enrollment uses a transaction ID and step ledger. Every completed step is idempotent and re-checkable. Before the Git registration commit, rollback can uninstall the new agent and remove its Node object without touching existing nodes. After registration, rollback also reverts the exact registry commit/PR.

## Recommended one-command onboarding contract

The command is run from the trusted workstation after the new machine has joined Tailscale:

```bash
bash tools/k3s/enroll-node.sh debian4
```

The workstation-orchestrated model is preferred over downloading a self-enrollment script onto the new node because repository credentials, cluster-admin kubeconfig, GitHub credentials and long-lived K3s tokens remain off the new machine.

### Prerequisites

- Supported Linux install completed.
- Tailscale installed, authenticated and approved in the tailnet.
- Tailnet policy permits the owner/enrollment identity to reach the machine through Tailscale SSH.
- The local enrollment identity can obtain the required root action on the new machine. This may be a temporary, narrowly-scoped Tailscale SSH policy or an existing sudo path.
- The machine name supplied to the command is unique.

### Transaction

1. Discover exactly one matching Tailscale node and pin its stable identity for this run.
2. Refuse reused hostname, machine ID, Tailscale identity, Node name or registry entry.
3. Capture preflight facts: OS/architecture, disks, clock, DNS, CPU/RAM, required ports and current services.
4. Establish and verify all three recovery doors without binding OpenSSH to the Tailscale address.
5. Converge the buildbox/fleet host profile and prove idempotency.
6. Request a short-lived K3s bootstrap token from the server.
7. Write canonical K3s agent configuration and token file with restrictive permissions; install the pinned version.
8. Wait for the Node, verify its identity and Ready conditions, and delete/revoke the bootstrap token.
9. Apply protected labels/taints from the control plane.
10. Run a node-pinned proof Job with resource limits and no host credentials.
11. Run fleet/buildbox audits and a reboot-survival proof when the phase requires it.
12. Only after all gates pass, update `fleet.json` and `buildbox-hosts.json`, commit, push and open a PR.
13. Produce a redacted receipt archive for review.

### Fail-closed behavior

- No match or multiple Tailscale matches: stop.
- Existing identity collision: stop.
- Missing independent recovery door: stop before K3s install.
- Bootstrap token creation or transport failure: stop; remove temporary token material.
- Node fails Ready/identity proof: uninstall only the candidate agent and delete only its Node object.
- Proof Job fails: keep the machine unregistered and unschedulable.
- Git conflict/push failure: leave the proven node cordoned/unregistered, produce exact recovery commands, and do not edit main.

## Phase-zero verification receipt

Verified in the isolated audit checkout:

- `HOME=/home/user node modules/workstation/claude/tests/remote-seat-provision.test.mjs` — pass.
- `node modules/workstation/claude/tests/k3s-remote-build.test.mjs` — pass.
- `PYTHONPATH=modules/harness/factory python3 -m pytest -q modules/harness/factory/tests/test_kubernetes_job.py` — 6 passed in the audit environment.
- Fleet loader/expansion/engine/hardening/CLI focused tests — pass when run as a non-root user.
- `modules/workstation/claude/tests/k3s-remote-build-integration.sh` — correctly skipped because the audit environment has no live cluster kubeconfig.
- `tools/k3s/test/phase0-discover.test.sh` — pass.

A full buildbox suite was not accepted as evidence in the audit container because the extracted checkout and home paths do not reproduce the owner's installed environment. The phase script runs only deterministic focused gates, captures environmental skips honestly, and collects live evidence on the owner's workstation.

## Conclusion

The migration can be completed through the requested package/receipt loop. The owner should not have to manually recreate hours of commands. The owner will still control the security and physical boundaries: initial OS/Tailscale admission, root authorization, secrets/policies, reboot/console recovery, and protected-main approval. Everything deterministic around those boundaries can be authored, tested, packaged and driven by one script per phase.