# Overdeck k3s migration and one-command node enrollment audience: owner and AI coding agents. status: SUPERSEDED (2026-08-15) — see "Delivered work, never marked" and "Superseded" sections at end of file task IDs: #96, #175; one-command node-onboarding request has no task ID in the supplied snapshot. source request: inspect all Overdeck k3s plans and supplied WIP, correct the plan, then complete migration through phased ZIP + one-script packages. After Tailscale is installed on a new machine, one workstation command must enroll it, prove it, register it and prepare the Git change. ## Outcome Overdeck runs supported execution workloads on a hardened, recoverable K3s cluster. New Linux machines are enrolled from the trusted workstation with one command after Tailscale admission. Every migration phase is delivered as an inspectable changed-files ZIP and one application script; the owner runs it, returns the receipt, and the next phase is generated from observed state. Final changes land on `main` through reviewed branches/PRs. ## Acceptance criteria - One canonical, version-pinned K3s server/agent configuration contract. - Current datastore plus server-token backups, off-host retention and restore proof. - One-command workstation-orchestrated node enrollment after Tailscale installation. - Enrollment is idempotent, collision-safe, token-redacted, rollback-capable and proof-gated. - `fleet.json` and `buildbox-hosts.json` are updated together only after live proof. - Three independent recovery doors survive reboot on every buildbox. - Factory Kubernetes workloads use immutable input/image/result contracts and no reusable host Git credential in an untrusted pod. - Namespace, pod, RBAC, resource and egress restrictions are enforced and tested. - Legacy remote-build K3s canary remains separate from the secure Factory workload contract. - K3s becomes default only after canary metrics and failure drills pass. - Direct SSH fallback is retired in a separate, explicit final gate. - Each phase produces a machine-readable, redacted receipt and exact Git branch/commit/PR status. - No phase mutates the shared main checkout directly. ## Non-goals - No direct merge of the stale `origin/wt/factory-kubernetes-job` branch. - No self-enrollment script that stores repository, cluster-admin or long-lived join credentials on a candidate node. - No global fleet hardening fan-out. - No Factory implementation based on SSH workspace fan-out, `hostPath`, node affinity, local-only images, PATH surgery, ConfigMap archives or pod-log result transport. - No deletion of the emergency SSH fallback in the same phase that makes K3s the default. ## Architecture boundary ### Shared cluster foundation Node enrollment, K3s configuration, image pull, namespaces, RBAC, quotas, limits, network policy, observability, backup, restore and upgrades are common infrastructure. ### Legacy remote-build compatibility lane The existing `k3s-remote-build.mjs` path may be canaried as a bounded transition for existing remote-build work. It remains clearly labeled `legacy-remote-build` and does not establish the Factory security model. ### Factory Kubernetes lane Factory uses immutable Git/artifact input, a digest-pinned image, Kubernetes API lifecycle, explicit result metadata and a trusted publisher/application boundary. Untrusted coding containers receive no reusable host GitHub credential. ## Responsibility boundary ### Work produced remotely by the assistant - Repository and plan audit. - Architecture reconciliation and phase gates. - Production code, tests, manifests, policies, scripts and documentation. - Migration, rollback, validation and receipt tooling. - Changed-files ZIP, checksums and application script per phase. - Safe worktree/branch creation, commit, push and PR automation in the phase script. - Analysis of returned receipts and correction packages. - Forward-porting selected stale-branch work after code review. ### Actions retained by the owner/system boundary - Install the base OS and Tailscale and admit the machine to the tailnet. - Supply/approve root authorization, Tailscale policy, GitHub App/registry policy and secret values. - Run the delivered phase command on the trusted workstation. - Keep physical/console recovery available for reboot or storage work. - Review redacted receipts and any security-policy changes. - Approve/merge protected-main PRs. The phase script performs all deterministic installation, deployment, validation and Git steps available through those authorizations. It never fabricates access it does not have. ## Package/receipt protocol Every phase package contains: - only changed repository files at their original relative paths; - `MANIFEST.md` with base revision, scope, prerequisites and rollback; - a binary-safe Git patch; - SHA-256 checksums; - focused tests; - a standalone `apply-overdeck-k3s-phaseN.sh`. The application script: 1. verifies prerequisites and package checksums; 2. fetches `origin/main` without modifying the main checkout; 3. creates an isolated `.worktrees/` branch; 4. applies the patch with three-way conflict detection; 5. runs phase-specific deterministic tests; 6. performs the phase's live operations only after preflight gates; 7. validates and executes rollback automatically on a failed transaction where safe; 8. commits exact paths; 9. pushes the branch and opens/updates a PR when authentication is available; 10. emits a redacted result archive plus human-readable summary. A phase is not marked done merely because code was generated. It is done after returned live evidence passes the phase gate and the exact change is merged/deployed where required. ## Phases ### Phase 0 — audited baseline and safe landing package **Goal:** establish truthful inputs before any live cluster mutation. **Assistant work:** - compare the supplied main/worktree archives; - review all scheduler, Factory, fleet, enrollment and recovery-door plans; - repair the supplied unmanaged-copy rollback hole and add failure-injection tests; - add a read-only live-state collector; - record the reconciled architecture and phased execution plan; - package/apply/test/branch/push/PR automation. **Owner command:** ```bash bash apply-overdeck-k3s-phase0.sh /home/user/Projects/overdeck ``` **Live mutation:** none to hosts or Kubernetes. Git changes occur only in a new worktree branch. **Gate:** focused tests pass; receipt identifies current server, agents, versions, recovery doors, cluster policy, storage/backup metadata, source-of-truth drift and inaccessible nodes without exposing secret values. **Rollback:** remove the created worktree/branch if desired; no cluster rollback is needed. **Current receipt:** owner execution passed all required Phase 0 repository gates, produced commit `ec69e612bb666a459ba8fff31656616988236e4d`, opened PR #12, and merged to `main` in merge commit `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`. One legacy integration gate was explicitly optional and recorded as failed; it did not block the Phase 0 audit landing. ### Phase 1 — qualification-first control-plane baseline, backup and recovery **Input:** Phase 0 receipt plus the three safe Phase 1 failure receipts. **Assistant work:** - rederive Phase 1 around the live `debian3` topology rather than patching v3; - separate the outer `/usr/local/bin/k3s` launcher from the running extracted executable; - require active service, outer-launcher `/readyz`, and non-empty Ready Node inventory; - model the live `agent-token -> token` layout without archiving a symlink; - separate launcher/runtime version-lock fields and tie the release URL only to the launcher digest; - aggregate complete qualification failures before any K3s configuration mutation; - create, encrypt, verify, and materialize a pre-change recovery archive before publication; - publish the canonical drop-in, contract, and version lock transactionally; - verify the converged server, workstation, and every reachable registry host; - create and verify a post-change recovery archive before finalization; - install the recurring backup timer transactionally; - authorize Git publication only in the final successful live receipt; - make the package launcher commit/push/PR only after that authorization. **Owner action:** run one replacement phase script. No separate manual plan/apply/Git sequence is required. **Gate:** - all repository, failure-injection, fake-cluster, Phase 0, fleet, and focused Factory gates pass; - complete live qualification passes; - pre-change encrypted restore proof predates configuration publication; - K3s remains healthy with unchanged version, launcher, runtime, and datastore type; - post-change encrypted restore proof succeeds; - recurring timer is installed and active; - receipt returns `status=success` and `git_publication_allowed=true`; - only then is the exact candidate committed and optionally pushed as a draft PR. **Rollback:** before server finalization, restore prior managed files/absence and timer state, restart K3s only when needed, and prove readiness. Qualification failures require no K3s rollback and never create a Git commit. **Implementation status:** DONE. The qualification-first replacement passed live qualification, encrypted pre-change and post-change restore proofs, transactional convergence and recurring-timer proof. PR #13 merged and was deployed at `832acc72665e49a1a3f4f19b66fd2810b59d2e48`. A follow-up rollback-safety fix ensures a timer rollback failure cannot block control-plane rollback, with regression coverage and full GitHub gate proof. **Durable references:** - design: `docs/specs/2026-08-10-k3s-control-plane-baseline-design.md`; - rederivation report: `docs/reports/2026-08-11-k3s-phase1-rederivation.md`; - backup/restore runbook: `docs/runbooks/k3s-control-plane-backup-restore.md`; - execution instructions: `docs/runbooks/k3s-phase1-agent-instructions.md`; - entrypoint: `tools/k3s/phase1-control-plane.sh`. ### Phase 2 — one-command enrollment engine, dry-run first **Input:** deployed Phase 1 canonical endpoint/config, version lock, verified recovery baseline, and current fleet/buildbox registries. **Assistant work:** - implement `tools/k3s/enroll-node.sh ` and a stdlib-only workstation orchestrator; - add checksum-verified, ephemeral, read-only candidate and control-plane helpers; - add Tailscale identity discovery, exact uniqueness checks and supported-host preflight; - encode the complete 17-step Phase 3 transaction and rollback order in a deterministic content-addressed plan; - add a durable plan-bound transaction ledger with strict ordering and resume/rollback contracts; - model the three independent recovery doors and serial convergence policy; - represent a 600-second bootstrap token without recording its value; - define pinned K3s agent, Node identity, protected-label, quarantine and node-pinned proof-Job contracts; - generate atomic paired `fleet.json` + `buildbox-hosts.json` previews with `execution=none`; - recursively reject secret-like output; - independently validate receipts before Git publication; - add sanitized fixture, unit/failure tests, deterministic shell transaction, documentation and interactive presentation; - hard-disable every live mutation command until a reviewed Phase 3 canary package. **Owner action:** run the standalone Phase 2 package launcher. Default execution uses the sanitized `debian4` fixture; `--live-candidate` performs read-only qualification against a real OS + Tailscale-ready candidate. Return the generated result archive and keep the draft PR unmerged pending review. **Gate:** - package and launcher checksums pass; - repository changes are applied only in a fresh isolated worktree from current `origin/main`; - all required unit, failure-injection, helper, receipt and deterministic transaction gates pass; - generated plan and registry previews are byte-identical across repeated fixture runs; - the plan has exactly 17 steps and contains no secret values; - source registry digests remain unchanged; - candidate preview uses `execution=none` and has no order/fallback membership; - all live/candidate/cluster/registry mutation flags are false; - `phase3_authorized=false`; - only an independently validated receipt may authorize a candidate branch commit/push/draft PR. **Rollback:** Phase 2 has no live cluster or host mutation to undo. A failure leaves only the isolated worktree, gate logs and a receipt with `git_publication_allowed=false`. Preview registries must never be copied over tracked registries manually. **Implementation status:** implementation and distribution package complete against post-Phase-1 main. Live canary enrollment remains locked to Phase 3. **Durable references:** - design: `docs/specs/2026-08-11-k3s-node-enrollment-design.md`; - implementation report: `docs/reports/2026-08-11-k3s-phase2-implementation.md`; - owner runbook: `docs/runbooks/k3s-node-enrollment.md`; - agent instructions: `docs/runbooks/k3s-phase2-agent-instructions.md`; - presentation: `docs/mockups/k3s-phase2-enrollment.html`; - entrypoint: `tools/k3s/enroll-node.sh`. ### Phase 3 — canary enrollment of one real machine **Input:** Phase 2 dry-run receipt and an OS + Tailscale-ready candidate. **Owner command:** ```bash bash tools/k3s/enroll-node.sh debian4 ``` **Assistant-authored transaction:** discovery → identity pin → root/recovery-door convergence → bootstrap token → K3s join → Ready/identity verification → protected scheduling metadata → proof Job → reboot proof where approved → fleet/buildbox audit → registry commit/PR → receipt. **Gate:** candidate survives reboot with three doors, returns Ready, runs only the intended proof workload, is clean under fleet/buildbox audit, and appears in both declarations in one reviewed commit. **Rollback:** cordon/drain candidate, delete only candidate workloads/Node, uninstall candidate K3s agent, restore prior host config, revoke token and revert candidate registry change. ### Phase 4 — secure shared workload substrate **Input:** stable control plane and at least two proven nodes or an explicitly accepted single-node limitation. **Assistant work:** - create dedicated namespaces and service accounts; - implement least-privilege Role/RoleBinding, ResourceQuota, LimitRange and Pod Security labels/admission; - implement default-deny ingress/egress plus tested explicit destinations; - configure digest-pinned image pull and registry credentials; - verify trusted-controller placement and protected node labels; - review the five stale Factory hardening commits and forward-port only current, useful pieces; - add policy probes and rollback manifests. **Owner action:** approve registry/GitHub App/Tailscale policy and run one phase script. **Gate:** allowed probes pass; arbitrary egress, privileged pod creation, host mounts, arbitrary service-account use and untrusted-node placement fail. ### Phase 5 — one real Factory Kubernetes vertical slice **Assistant work:** - replace missing/stale preserved WIP on current main; - create immutable input and digest-pinned worker execution; - move result publication behind a trusted, attempt-bound publisher; - implement API-backed heartbeat, cancellation and restart recovery; - implement total ref lifecycle with bounded recovery retention; - preserve file modes, renames and deletions during safe result application; - close TOCTOU and local-edit guards at the atomic apply boundary; - run the complete Factory test suite and one real owner-visible task. **Owner action:** approve the narrowly-scoped GitHub App/credential boundary and run the package. **Gate:** submit → restricted Job → visible heartbeat → result publication → safe apply → cleanup succeeds; arbitrary ref push/token exfiltration tests fail; cancellation and init failure produce durable owner-visible outcomes. ### Phase 6 — legacy remote-build K3s canary and observability **Assistant work:** - keep `k3s_enabled` off globally; - add explicit canary selection, workload labels and stop switch; - measure queue time, dispatch latency, completion/failure classification and fallback use; - reconcile controller watcher status with Kubernetes Job truth; - test API outage, node loss, Job timeout and ambiguous submit; - prevent compatibility host credentials from crossing into Factory workloads. **Owner action:** run a bounded canary package and identify allowed canary jobs. **Gate:** canary success/failure is no worse than the direct path under agreed thresholds; every fallback is explicit and visible; no customer-wide cutover. ### Phase 7 — default cutover and failure drills **Assistant work:** - make K3s default for the accepted workload classes; - keep direct SSH fallback as an owner-visible emergency lane; - add SLO dashboards/alerts, capacity and disk-pressure gates; - perform API outage, server restart, agent loss, image-pull failure, network-policy and backup-restore drills; - make rollback one command. **Owner action:** approve the cutover window and run the package; retain console access during reboot/restore drills. **Gate:** agreed observation period passes; no silent local/workstation spill; recovery and rollback receipts pass. ### Phase 8 — legacy retirement, upgrades and final main convergence **Assistant work:** - delete hand-rolled scheduler/fallback code only after owner acceptance; - remove legacy hostPath/SSH workspace transport and stale instructions; - finalize automated node enrollment/decommission, token rotation, backup retention and upgrade runbooks; - prove rolling upgrade/rollback on one agent before server upgrade; - consolidate plans/status, land final PRs and verify installed entrypoints from main. **Owner action:** explicitly authorize fallback deletion and final protected-main merges. **Gate:** no production caller depends on removed paths; restore and upgrade proofs pass; every package is merged/deployed; docs and source of truth match live state. ## Phase ordering invariants - Never harden or reboot all nodes in one transaction. - Never register a candidate in Git before live proof. - Never use a non-expiring join token in package files, command history or receipts. - Never merge the stale Factory branch directly. - Never permit the legacy workload transport to define Factory security. - Never delete fallback during the default-cutover phase. - Never infer reboot survival from a service restart. - Never proceed from a historical markdown receipt when a current machine receipt is available. ## Current receipt - Supplied archives audited against one another. - Only two supplied worktree files differ from main; missing Factory preserved WIP confirmed absent. - Supplied runtime-copy migration hardened with transactional rollback and failure-injection tests. - Read-only Phase 0 collector and its offline test added. - Detailed audit recorded at `docs/reports/2026-08-10-k3s-migration-audit.md`. - Phase 0 changed-files package/application script generation in progress. - Phase 0 commit `ec69e612bb666a459ba8fff31656616988236e4d` merged through PR #12 at `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`. - Phase 1 v1–v3 stopped safely before persistent cluster mutation and are superseded by the qualification-first replacement. - The replacement replays the live launcher/runtime and agent-token topology, proves a pre-change backup before configuration publication, and permits Git publication only after full live success. - Phase 1 has no product UI, primitive, design-system, or Astryx delta. ## Next executable action Run `apply-overdeck-k3s-phase1-qualified.sh /home/user/Projects/overdeck` on the trusted workstation and return the generated `overdeck-k3s-phase1-result-*.tar.gz`. Keep any draft pull request unmerged until that receipt is reviewed. ## Delivered work, never marked (2026-08-15) Measured 2026-08-15: this line of work has 159 commits and 35 branches, including a commit titled "Record successful k3s Factory acceptance" (`c5fb2b8bd`, verified ancestor of `origin/main`) and a dedicated `refs/heads/receipt/factory-k3s-delivery` ref. Phases 0-2 are landed and deployed per the receipts above (Phase 0 PR #12, Phase 1 PR #13 deployed `832acc72`, Phase 2 package complete). This is NOT abandoned or stalled work — it is delivered work that the bookkeeping never caught up to, discovered while reconciling the plan index against git evidence (see [2026-08-15-plan-status-self-reconciling.md](2026-08-15-plan-status-self-reconciling.md)). ## Superseded (2026-08-15) [Laptop as terminal](2026-08-15-laptop-as-terminal.md) is now the registered destination for where agent work runs, and it puts k3s explicitly out of scope for that goal. Live canary enrollment (Phase 3) and default cutover (Phase 7) were never run; this plan is superseded for further execution, not erased — the delivered phases above stand, and this file remains the record of the architecture and what was actually shipped.