# Overdeck k3s migration and one-command node enrollment audience: owner and AI coding agents. status: ACTIVE task IDs: #96, #175; one-command node-onboarding request has no task ID in the supplied snapshot. source request: inspect all Overdeck k3s plans and supplied WIP, correct the plan, then complete migration through phased ZIP + one-script packages. After Tailscale is installed on a new machine, one workstation command must enroll it, prove it, register it and prepare the Git change. ## Outcome Overdeck runs supported execution workloads on a hardened, recoverable K3s cluster. New Linux machines are enrolled from the trusted workstation with one command after Tailscale admission. Every migration phase is delivered as an inspectable changed-files ZIP and one application script; the owner runs it, returns the receipt, and the next phase is generated from observed state. Final changes land on `main` through reviewed branches/PRs. ## Acceptance criteria - One canonical, version-pinned K3s server/agent configuration contract. - Current datastore plus server-token backups, off-host retention and restore proof. - One-command workstation-orchestrated node enrollment after Tailscale installation. - Enrollment is idempotent, collision-safe, token-redacted, rollback-capable and proof-gated. - `fleet.json` and `buildbox-hosts.json` are updated together only after live proof. - Three independent recovery doors survive reboot on every buildbox. - Factory Kubernetes workloads use immutable input/image/result contracts and no reusable host Git credential in an untrusted pod. - Namespace, pod, RBAC, resource and egress restrictions are enforced and tested. - Legacy remote-build K3s canary remains separate from the secure Factory workload contract. - K3s becomes default only after canary metrics and failure drills pass. - Direct SSH fallback is retired in a separate, explicit final gate. - Each phase produces a machine-readable, redacted receipt and exact Git branch/commit/PR status. - No phase mutates the shared main checkout directly. ## Non-goals - No direct merge of the stale `origin/wt/factory-kubernetes-job` branch. - No self-enrollment script that stores repository, cluster-admin or long-lived join credentials on a candidate node. - No global fleet hardening fan-out. - No Factory implementation based on SSH workspace fan-out, `hostPath`, node affinity, local-only images, PATH surgery, ConfigMap archives or pod-log result transport. - No deletion of the emergency SSH fallback in the same phase that makes K3s the default. ## Architecture boundary ### Shared cluster foundation Node enrollment, K3s configuration, image pull, namespaces, RBAC, quotas, limits, network policy, observability, backup, restore and upgrades are common infrastructure. ### Legacy remote-build compatibility lane The existing `k3s-remote-build.mjs` path may be canaried as a bounded transition for existing remote-build work. It remains clearly labeled `legacy-remote-build` and does not establish the Factory security model. ### Factory Kubernetes lane Factory uses immutable Git/artifact input, a digest-pinned image, Kubernetes API lifecycle, explicit result metadata and a trusted publisher/application boundary. Untrusted coding containers receive no reusable host GitHub credential. ## Responsibility boundary ### Work produced remotely by the assistant - Repository and plan audit. - Architecture reconciliation and phase gates. - Production code, tests, manifests, policies, scripts and documentation. - Migration, rollback, validation and receipt tooling. - Changed-files ZIP, checksums and application script per phase. - Safe worktree/branch creation, commit, push and PR automation in the phase script. - Analysis of returned receipts and correction packages. - Forward-porting selected stale-branch work after code review. ### Actions retained by the owner/system boundary - Install the base OS and Tailscale and admit the machine to the tailnet. - Supply/approve root authorization, Tailscale policy, GitHub App/registry policy and secret values. - Run the delivered phase command on the trusted workstation. - Keep physical/console recovery available for reboot or storage work. - Review redacted receipts and any security-policy changes. - Approve/merge protected-main PRs. The phase script performs all deterministic installation, deployment, validation and Git steps available through those authorizations. It never fabricates access it does not have. ## Package/receipt protocol Every phase package contains: - only changed repository files at their original relative paths; - `MANIFEST.md` with base revision, scope, prerequisites and rollback; - a binary-safe Git patch; - SHA-256 checksums; - focused tests; - a standalone `apply-overdeck-k3s-phaseN.sh`. The application script: 1. verifies prerequisites and package checksums; 2. fetches `origin/main` without modifying the main checkout; 3. creates an isolated `.worktrees/` branch; 4. applies the patch with three-way conflict detection; 5. runs phase-specific deterministic tests; 6. performs the phase's live operations only after preflight gates; 7. validates and executes rollback automatically on a failed transaction where safe; 8. commits exact paths; 9. pushes the branch and opens/updates a PR when authentication is available; 10. emits a redacted result archive plus human-readable summary. A phase is not marked done merely because code was generated. It is done after returned live evidence passes the phase gate and the exact change is merged/deployed where required. ## Phases ### Phase 0 — audited baseline and safe landing package **Goal:** establish truthful inputs before any live cluster mutation. **Assistant work:** - compare the supplied main/worktree archives; - review all scheduler, Factory, fleet, enrollment and recovery-door plans; - repair the supplied unmanaged-copy rollback hole and add failure-injection tests; - add a read-only live-state collector; - record the reconciled architecture and phased execution plan; - package/apply/test/branch/push/PR automation. **Owner command:** ```bash bash apply-overdeck-k3s-phase0.sh /home/user/Projects/overdeck ``` **Live mutation:** none to hosts or Kubernetes. Git changes occur only in a new worktree branch. **Gate:** focused tests pass; receipt identifies current server, agents, versions, recovery doors, cluster policy, storage/backup metadata, source-of-truth drift and inaccessible nodes without exposing secret values. **Rollback:** remove the created worktree/branch if desired; no cluster rollback is needed. **Current receipt:** owner execution passed all required Phase 0 repository gates, produced commit `ec69e612bb666a459ba8fff31656616988236e4d`, opened PR #12, and merged to `main` in merge commit `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`. One legacy integration gate was explicitly optional and recorded as failed; it did not block the Phase 0 audit landing. ### Phase 1 — qualification-first control-plane baseline, backup and recovery **Input:** Phase 0 receipt plus the three safe Phase 1 failure receipts. **Assistant work:** - rederive Phase 1 around the live `debian3` topology rather than patching v3; - separate the outer `/usr/local/bin/k3s` launcher from the running extracted executable; - require active service, outer-launcher `/readyz`, and non-empty Ready Node inventory; - model the live `agent-token -> token` layout without archiving a symlink; - separate launcher/runtime version-lock fields and tie the release URL only to the launcher digest; - aggregate complete qualification failures before any K3s configuration mutation; - create, encrypt, verify, and materialize a pre-change recovery archive before publication; - publish the canonical drop-in, contract, and version lock transactionally; - verify the converged server, workstation, and every reachable registry host; - create and verify a post-change recovery archive before finalization; - install the recurring backup timer transactionally; - authorize Git publication only in the final successful live receipt; - make the package launcher commit/push/PR only after that authorization. **Owner action:** run one replacement phase script. No separate manual plan/apply/Git sequence is required. **Gate:** - all repository, failure-injection, fake-cluster, Phase 0, fleet, and focused Factory gates pass; - complete live qualification passes; - pre-change encrypted restore proof predates configuration publication; - K3s remains healthy with unchanged version, launcher, runtime, and datastore type; - post-change encrypted restore proof succeeds; - recurring timer is installed and active; - receipt returns `status=success` and `git_publication_allowed=true`; - only then is the exact candidate committed and optionally pushed as a draft PR. **Rollback:** before server finalization, restore prior managed files/absence and timer state, restart K3s only when needed, and prove readiness. Qualification failures require no K3s rollback and never create a Git commit. **Implementation status:** DONE. The qualification-first replacement passed live qualification, encrypted pre-change and post-change restore proofs, transactional convergence and recurring-timer proof. PR #13 merged and was deployed at `832acc72665e49a1a3f4f19b66fd2810b59d2e48`. A follow-up rollback-safety fix ensures a timer rollback failure cannot block control-plane rollback, with regression coverage and full GitHub gate proof. **Durable references:** - design: `docs/specs/2026-08-10-k3s-control-plane-baseline-design.md`; - rederivation report: `docs/reports/2026-08-11-k3s-phase1-rederivation.md`; - backup/restore runbook: `docs/runbooks/k3s-control-plane-backup-restore.md`; - execution instructions: `docs/runbooks/k3s-phase1-agent-instructions.md`; - entrypoint: `tools/k3s/phase1-control-plane.sh`. ### Phase 2 — one-command enrollment engine, dry-run first **Input:** deployed Phase 1 canonical endpoint/config, version lock, verified recovery baseline, and current fleet/buildbox registries. **Assistant work:** - implement `tools/k3s/enroll-node.sh ` and a stdlib-only workstation orchestrator; - add checksum-verified, ephemeral, read-only candidate and control-plane helpers; - add Tailscale identity discovery, exact uniqueness checks and supported-host preflight; - encode the complete 17-step Phase 3 transaction and rollback order in a deterministic content-addressed plan; - add a durable plan-bound transaction ledger with strict ordering and resume/rollback contracts; - model the three independent recovery doors and serial convergence policy; - represent a 600-second bootstrap token without recording its value; - define pinned K3s agent, Node identity, protected-label, quarantine and node-pinned proof-Job contracts; - generate atomic paired `fleet.json` + `buildbox-hosts.json` previews with `execution=none`; - recursively reject secret-like output; - independently validate receipts before Git publication; - add sanitized fixture, unit/failure tests, deterministic shell transaction, documentation and interactive presentation; - hard-disable every live mutation command until a reviewed Phase 3 canary package. **Owner action:** run the standalone Phase 2 package launcher. Default execution uses the sanitized `debian4` fixture; `--live-candidate` performs read-only qualification against a real OS + Tailscale-ready candidate. Return the generated result archive and keep the draft PR unmerged pending review. **Gate:** - package and launcher checksums pass; - repository changes are applied only in a fresh isolated worktree from current `origin/main`; - all required unit, failure-injection, helper, receipt and deterministic transaction gates pass; - generated plan and registry previews are byte-identical across repeated fixture runs; - the plan has exactly 17 steps and contains no secret values; - source registry digests remain unchanged; - candidate preview uses `execution=none` and has no order/fallback membership; - all live/candidate/cluster/registry mutation flags are false; - `phase3_authorized=false`; - only an independently validated receipt may authorize a candidate branch commit/push/draft PR. **Rollback:** Phase 2 has no live cluster or host mutation to undo. A failure leaves only the isolated worktree, gate logs and a receipt with `git_publication_allowed=false`. Preview registries must never be copied over tracked registries manually. **Implementation status:** implementation and distribution package complete against post-Phase-1 main. Live canary enrollment remains locked to Phase 3. **Durable references:** - design: `docs/specs/2026-08-11-k3s-node-enrollment-design.md`; - implementation report: `docs/reports/2026-08-11-k3s-phase2-implementation.md`; - owner runbook: `docs/runbooks/k3s-node-enrollment.md`; - agent instructions: `docs/runbooks/k3s-phase2-agent-instructions.md`; - presentation: `docs/mockups/k3s-phase2-enrollment.html`; - entrypoint: `tools/k3s/enroll-node.sh`. ### Phase 3 — canary enrollment of one real machine **Input:** Phase 2 dry-run receipt and an OS + Tailscale-ready candidate. **Owner command:** ```bash bash tools/k3s/enroll-node.sh debian4 ``` **Assistant-authored transaction:** discovery → identity pin → root/recovery-door convergence → bootstrap token → K3s join → Ready/identity verification → protected scheduling metadata → proof Job → reboot proof where approved → fleet/buildbox audit → registry commit/PR → receipt. **Gate:** candidate survives reboot with three doors, returns Ready, runs only the intended proof workload, is clean under fleet/buildbox audit, and appears in both declarations in one reviewed commit. **Rollback:** cordon/drain candidate, delete only candidate workloads/Node, uninstall candidate K3s agent, restore prior host config, revoke token and revert candidate registry change. ### Phase 4 — secure shared workload substrate **Input:** stable control plane and at least two proven nodes or an explicitly accepted single-node limitation. **Assistant work:** - create dedicated namespaces and service accounts; - implement least-privilege Role/RoleBinding, ResourceQuota, LimitRange and Pod Security labels/admission; - implement default-deny ingress/egress plus tested explicit destinations; - configure digest-pinned image pull and registry credentials; - verify trusted-controller placement and protected node labels; - review the five stale Factory hardening commits and forward-port only current, useful pieces; - add policy probes and rollback manifests. **Owner action:** approve registry/GitHub App/Tailscale policy and run one phase script. **Gate:** allowed probes pass; arbitrary egress, privileged pod creation, host mounts, arbitrary service-account use and untrusted-node placement fail. ### Phase 5 — one real Factory Kubernetes vertical slice **Assistant work:** - replace missing/stale preserved WIP on current main; - create immutable input and digest-pinned worker execution; - move result publication behind a trusted, attempt-bound publisher; - implement API-backed heartbeat, cancellation and restart recovery; - implement total ref lifecycle with bounded recovery retention; - preserve file modes, renames and deletions during safe result application; - close TOCTOU and local-edit guards at the atomic apply boundary; - run the complete Factory test suite and one real owner-visible task. **Owner action:** approve the narrowly-scoped GitHub App/credential boundary and run the package. **Gate:** submit → restricted Job → visible heartbeat → result publication → safe apply → cleanup succeeds; arbitrary ref push/token exfiltration tests fail; cancellation and init failure produce durable owner-visible outcomes. ### Phase 6 — legacy remote-build K3s canary and observability **Assistant work:** - keep `k3s_enabled` off globally; - add explicit canary selection, workload labels and stop switch; - measure queue time, dispatch latency, completion/failure classification and fallback use; - reconcile controller watcher status with Kubernetes Job truth; - test API outage, node loss, Job timeout and ambiguous submit; - prevent compatibility host credentials from crossing into Factory workloads. **Owner action:** run a bounded canary package and identify allowed canary jobs. **Gate:** canary success/failure is no worse than the direct path under agreed thresholds; every fallback is explicit and visible; no customer-wide cutover. ### Phase 7 — default cutover and failure drills **Assistant work:** - make K3s default for the accepted workload classes; - keep direct SSH fallback as an owner-visible emergency lane; - add SLO dashboards/alerts, capacity and disk-pressure gates; - perform API outage, server restart, agent loss, image-pull failure, network-policy and backup-restore drills; - make rollback one command. **Owner action:** approve the cutover window and run the package; retain console access during reboot/restore drills. **Gate:** agreed observation period passes; no silent local/workstation spill; recovery and rollback receipts pass. ### Phase 8 — legacy retirement, upgrades and final main convergence **Assistant work:** - delete hand-rolled scheduler/fallback code only after owner acceptance; - remove legacy hostPath/SSH workspace transport and stale instructions; - finalize automated node enrollment/decommission, token rotation, backup retention and upgrade runbooks; - prove rolling upgrade/rollback on one agent before server upgrade; - consolidate plans/status, land final PRs and verify installed entrypoints from main. **Owner action:** explicitly authorize fallback deletion and final protected-main merges. **Gate:** no production caller depends on removed paths; restore and upgrade proofs pass; every package is merged/deployed; docs and source of truth match live state. ## Phase ordering invariants - Never harden or reboot all nodes in one transaction. - Never register a candidate in Git before live proof. - Never use a non-expiring join token in package files, command history or receipts. - Never merge the stale Factory branch directly. - Never permit the legacy workload transport to define Factory security. - Never delete fallback during the default-cutover phase. - Never infer reboot survival from a service restart. - Never proceed from a historical markdown receipt when a current machine receipt is available. ## Current receipt - Supplied archives audited against one another. - Only two supplied worktree files differ from main; missing Factory preserved WIP confirmed absent. - Supplied runtime-copy migration hardened with transactional rollback and failure-injection tests. - Read-only Phase 0 collector and its offline test added. - Detailed audit recorded at `docs/reports/2026-08-10-k3s-migration-audit.md`. - Phase 0 changed-files package/application script generation in progress. - Phase 0 commit `ec69e612bb666a459ba8fff31656616988236e4d` merged through PR #12 at `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`. - Phase 1 v1–v3 stopped safely before persistent cluster mutation and are superseded by the qualification-first replacement. - The replacement replays the live launcher/runtime and agent-token topology, proves a pre-change backup before configuration publication, and permits Git publication only after full live success. - Phase 1 has no product UI, primitive, design-system, or Astryx delta. ## Next executable action Run `apply-overdeck-k3s-phase1-qualified.sh /home/user/Projects/overdeck` on the trusted workstation and return the generated `overdeck-k3s-phase1-result-*.tar.gz`. Keep any draft pull request unmerged until that receipt is reviewed.