# Overdeck k3s migration and one-command node enrollment

audience: owner and AI coding agents.

status: ACTIVE

task IDs: #96, #175; one-command node-onboarding request has no task ID in the supplied snapshot.

source request: inspect all Overdeck k3s plans and supplied WIP, correct the plan, then complete migration through phased ZIP + one-script packages. After Tailscale is installed on a new machine, one workstation command must enroll it, prove it, register it and prepare the Git change.

## Outcome

Overdeck runs supported execution workloads on a hardened, recoverable K3s cluster. New Linux machines are enrolled from the trusted workstation with one command after Tailscale admission. Every migration phase is delivered as an inspectable changed-files ZIP and one application script; the owner runs it, returns the receipt, and the next phase is generated from observed state. Final changes land on `main` through reviewed branches/PRs.

## Acceptance criteria

- One canonical, version-pinned K3s server/agent configuration contract.
- Current datastore plus server-token backups, off-host retention and restore proof.
- One-command workstation-orchestrated node enrollment after Tailscale installation.
- Enrollment is idempotent, collision-safe, token-redacted, rollback-capable and proof-gated.
- `fleet.json` and `buildbox-hosts.json` are updated together only after live proof.
- Three independent recovery doors survive reboot on every buildbox.
- Factory Kubernetes workloads use immutable input/image/result contracts and no reusable host Git credential in an untrusted pod.
- Namespace, pod, RBAC, resource and egress restrictions are enforced and tested.
- Legacy remote-build K3s canary remains separate from the secure Factory workload contract.
- K3s becomes default only after canary metrics and failure drills pass.
- Direct SSH fallback is retired in a separate, explicit final gate.
- Each phase produces a machine-readable, redacted receipt and exact Git branch/commit/PR status.
- No phase mutates the shared main checkout directly.

## Non-goals

- No direct merge of the stale `origin/wt/factory-kubernetes-job` branch.
- No self-enrollment script that stores repository, cluster-admin or long-lived join credentials on a candidate node.
- No global fleet hardening fan-out.
- No Factory implementation based on SSH workspace fan-out, `hostPath`, node affinity, local-only images, PATH surgery, ConfigMap archives or pod-log result transport.
- No deletion of the emergency SSH fallback in the same phase that makes K3s the default.

## Architecture boundary

### Shared cluster foundation

Node enrollment, K3s configuration, image pull, namespaces, RBAC, quotas, limits, network policy, observability, backup, restore and upgrades are common infrastructure.

### Legacy remote-build compatibility lane

The existing `k3s-remote-build.mjs` path may be canaried as a bounded transition for existing remote-build work. It remains clearly labeled `legacy-remote-build` and does not establish the Factory security model.

### Factory Kubernetes lane

Factory uses immutable Git/artifact input, a digest-pinned image, Kubernetes API lifecycle, explicit result metadata and a trusted publisher/application boundary. Untrusted coding containers receive no reusable host GitHub credential.

## Responsibility boundary

### Work produced remotely by the assistant

- Repository and plan audit.
- Architecture reconciliation and phase gates.
- Production code, tests, manifests, policies, scripts and documentation.
- Migration, rollback, validation and receipt tooling.
- Changed-files ZIP, checksums and application script per phase.
- Safe worktree/branch creation, commit, push and PR automation in the phase script.
- Analysis of returned receipts and correction packages.
- Forward-porting selected stale-branch work after code review.

### Actions retained by the owner/system boundary

- Install the base OS and Tailscale and admit the machine to the tailnet.
- Supply/approve root authorization, Tailscale policy, GitHub App/registry policy and secret values.
- Run the delivered phase command on the trusted workstation.
- Keep physical/console recovery available for reboot or storage work.
- Review redacted receipts and any security-policy changes.
- Approve/merge protected-main PRs.

The phase script performs all deterministic installation, deployment, validation and Git steps available through those authorizations. It never fabricates access it does not have.

## Package/receipt protocol

Every phase package contains:

- only changed repository files at their original relative paths;
- `MANIFEST.md` with base revision, scope, prerequisites and rollback;
- a binary-safe Git patch;
- SHA-256 checksums;
- focused tests;
- a standalone `apply-overdeck-k3s-phaseN.sh`.

The application script:

1. verifies prerequisites and package checksums;
2. fetches `origin/main` without modifying the main checkout;
3. creates an isolated `.worktrees/<phase>` branch;
4. applies the patch with three-way conflict detection;
5. runs phase-specific deterministic tests;
6. performs the phase's live operations only after preflight gates;
7. validates and executes rollback automatically on a failed transaction where safe;
8. commits exact paths;
9. pushes the branch and opens/updates a PR when authentication is available;
10. emits a redacted result archive plus human-readable summary.

A phase is not marked done merely because code was generated. It is done after returned live evidence passes the phase gate and the exact change is merged/deployed where required.

## Phases

### Phase 0 — audited baseline and safe landing package

**Goal:** establish truthful inputs before any live cluster mutation.

**Assistant work:**

- compare the supplied main/worktree archives;
- review all scheduler, Factory, fleet, enrollment and recovery-door plans;
- repair the supplied unmanaged-copy rollback hole and add failure-injection tests;
- add a read-only live-state collector;
- record the reconciled architecture and phased execution plan;
- package/apply/test/branch/push/PR automation.

**Owner command:**

```bash
bash apply-overdeck-k3s-phase0.sh /home/user/Projects/overdeck
```

**Live mutation:** none to hosts or Kubernetes. Git changes occur only in a new worktree branch.

**Gate:** focused tests pass; receipt identifies current server, agents, versions, recovery doors, cluster policy, storage/backup metadata, source-of-truth drift and inaccessible nodes without exposing secret values.

**Rollback:** remove the created worktree/branch if desired; no cluster rollback is needed.

**Current receipt:** owner execution passed all required Phase 0 repository gates, produced commit `ec69e612bb666a459ba8fff31656616988236e4d`, opened PR #12, and merged to `main` in merge commit `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`. One legacy integration gate was explicitly optional and recorded as failed; it did not block the Phase 0 audit landing.

### Phase 1 — qualification-first control-plane baseline, backup and recovery

**Input:** Phase 0 receipt plus the three safe Phase 1 failure receipts.

**Assistant work:**

- rederive Phase 1 around the live `debian3` topology rather than patching v3;
- separate the outer `/usr/local/bin/k3s` launcher from the running extracted executable;
- require active service, outer-launcher `/readyz`, and non-empty Ready Node inventory;
- model the live `agent-token -> token` layout without archiving a symlink;
- separate launcher/runtime version-lock fields and tie the release URL only to the launcher digest;
- aggregate complete qualification failures before any K3s configuration mutation;
- create, encrypt, verify, and materialize a pre-change recovery archive before publication;
- publish the canonical drop-in, contract, and version lock transactionally;
- verify the converged server, workstation, and every reachable registry host;
- create and verify a post-change recovery archive before finalization;
- install the recurring backup timer transactionally;
- authorize Git publication only in the final successful live receipt;
- make the package launcher commit/push/PR only after that authorization.

**Owner action:** run one replacement phase script. No separate manual plan/apply/Git sequence is required.

**Gate:**

- all repository, failure-injection, fake-cluster, Phase 0, fleet, and focused Factory gates pass;
- complete live qualification passes;
- pre-change encrypted restore proof predates configuration publication;
- K3s remains healthy with unchanged version, launcher, runtime, and datastore type;
- post-change encrypted restore proof succeeds;
- recurring timer is installed and active;
- receipt returns `status=success` and `git_publication_allowed=true`;
- only then is the exact candidate committed and optionally pushed as a draft PR.

**Rollback:** before server finalization, restore prior managed files/absence and timer state, restart K3s only when needed, and prove readiness. Qualification failures require no K3s rollback and never create a Git commit.

**Implementation status:** DONE. The qualification-first replacement passed live qualification, encrypted pre-change and post-change restore proofs, transactional convergence and recurring-timer proof. PR #13 merged and was deployed at `832acc72665e49a1a3f4f19b66fd2810b59d2e48`. A follow-up rollback-safety fix ensures a timer rollback failure cannot block control-plane rollback, with regression coverage and full GitHub gate proof.

**Durable references:**

- design: `docs/specs/2026-08-10-k3s-control-plane-baseline-design.md`;
- rederivation report: `docs/reports/2026-08-11-k3s-phase1-rederivation.md`;
- backup/restore runbook: `docs/runbooks/k3s-control-plane-backup-restore.md`;
- execution instructions: `docs/runbooks/k3s-phase1-agent-instructions.md`;
- entrypoint: `tools/k3s/phase1-control-plane.sh`.

### Phase 2 — one-command enrollment engine, dry-run first

**Input:** deployed Phase 1 canonical endpoint/config, version lock, verified recovery baseline, and current fleet/buildbox registries.

**Assistant work:**

- implement `tools/k3s/enroll-node.sh <tailscale-name>` and a stdlib-only workstation orchestrator;
- add checksum-verified, ephemeral, read-only candidate and control-plane helpers;
- add Tailscale identity discovery, exact uniqueness checks and supported-host preflight;
- encode the complete 17-step Phase 3 transaction and rollback order in a deterministic content-addressed plan;
- add a durable plan-bound transaction ledger with strict ordering and resume/rollback contracts;
- model the three independent recovery doors and serial convergence policy;
- represent a 600-second bootstrap token without recording its value;
- define pinned K3s agent, Node identity, protected-label, quarantine and node-pinned proof-Job contracts;
- generate atomic paired `fleet.json` + `buildbox-hosts.json` previews with `execution=none`;
- recursively reject secret-like output;
- independently validate receipts before Git publication;
- add sanitized fixture, unit/failure tests, deterministic shell transaction, documentation and interactive presentation;
- hard-disable every live mutation command until a reviewed Phase 3 canary package.

**Owner action:** run the standalone Phase 2 package launcher. Default execution uses the sanitized `debian4` fixture; `--live-candidate` performs read-only qualification against a real OS + Tailscale-ready candidate. Return the generated result archive and keep the draft PR unmerged pending review.

**Gate:**

- package and launcher checksums pass;
- repository changes are applied only in a fresh isolated worktree from current `origin/main`;
- all required unit, failure-injection, helper, receipt and deterministic transaction gates pass;
- generated plan and registry previews are byte-identical across repeated fixture runs;
- the plan has exactly 17 steps and contains no secret values;
- source registry digests remain unchanged;
- candidate preview uses `execution=none` and has no order/fallback membership;
- all live/candidate/cluster/registry mutation flags are false;
- `phase3_authorized=false`;
- only an independently validated receipt may authorize a candidate branch commit/push/draft PR.

**Rollback:** Phase 2 has no live cluster or host mutation to undo. A failure leaves only the isolated worktree, gate logs and a receipt with `git_publication_allowed=false`. Preview registries must never be copied over tracked registries manually.

**Implementation status:** implementation and distribution package complete against post-Phase-1 main. Live canary enrollment remains locked to Phase 3.

**Durable references:**

- design: `docs/specs/2026-08-11-k3s-node-enrollment-design.md`;
- implementation report: `docs/reports/2026-08-11-k3s-phase2-implementation.md`;
- owner runbook: `docs/runbooks/k3s-node-enrollment.md`;
- agent instructions: `docs/runbooks/k3s-phase2-agent-instructions.md`;
- presentation: `docs/mockups/k3s-phase2-enrollment.html`;
- entrypoint: `tools/k3s/enroll-node.sh`.

### Phase 3 — canary enrollment of one real machine

**Input:** Phase 2 dry-run receipt and an OS + Tailscale-ready candidate.

**Owner command:**

```bash
bash tools/k3s/enroll-node.sh debian4
```

**Assistant-authored transaction:** discovery → identity pin → root/recovery-door convergence → bootstrap token → K3s join → Ready/identity verification → protected scheduling metadata → proof Job → reboot proof where approved → fleet/buildbox audit → registry commit/PR → receipt.

**Gate:** candidate survives reboot with three doors, returns Ready, runs only the intended proof workload, is clean under fleet/buildbox audit, and appears in both declarations in one reviewed commit.

**Rollback:** cordon/drain candidate, delete only candidate workloads/Node, uninstall candidate K3s agent, restore prior host config, revoke token and revert candidate registry change.

### Phase 4 — secure shared workload substrate

**Input:** stable control plane and at least two proven nodes or an explicitly accepted single-node limitation.

**Assistant work:**

- create dedicated namespaces and service accounts;
- implement least-privilege Role/RoleBinding, ResourceQuota, LimitRange and Pod Security labels/admission;
- implement default-deny ingress/egress plus tested explicit destinations;
- configure digest-pinned image pull and registry credentials;
- verify trusted-controller placement and protected node labels;
- review the five stale Factory hardening commits and forward-port only current, useful pieces;
- add policy probes and rollback manifests.

**Owner action:** approve registry/GitHub App/Tailscale policy and run one phase script.

**Gate:** allowed probes pass; arbitrary egress, privileged pod creation, host mounts, arbitrary service-account use and untrusted-node placement fail.

### Phase 5 — one real Factory Kubernetes vertical slice

**Assistant work:**

- replace missing/stale preserved WIP on current main;
- create immutable input and digest-pinned worker execution;
- move result publication behind a trusted, attempt-bound publisher;
- implement API-backed heartbeat, cancellation and restart recovery;
- implement total ref lifecycle with bounded recovery retention;
- preserve file modes, renames and deletions during safe result application;
- close TOCTOU and local-edit guards at the atomic apply boundary;
- run the complete Factory test suite and one real owner-visible task.

**Owner action:** approve the narrowly-scoped GitHub App/credential boundary and run the package.

**Gate:** submit → restricted Job → visible heartbeat → result publication → safe apply → cleanup succeeds; arbitrary ref push/token exfiltration tests fail; cancellation and init failure produce durable owner-visible outcomes.

### Phase 6 — legacy remote-build K3s canary and observability

**Assistant work:**

- keep `k3s_enabled` off globally;
- add explicit canary selection, workload labels and stop switch;
- measure queue time, dispatch latency, completion/failure classification and fallback use;
- reconcile controller watcher status with Kubernetes Job truth;
- test API outage, node loss, Job timeout and ambiguous submit;
- prevent compatibility host credentials from crossing into Factory workloads.

**Owner action:** run a bounded canary package and identify allowed canary jobs.

**Gate:** canary success/failure is no worse than the direct path under agreed thresholds; every fallback is explicit and visible; no customer-wide cutover.

### Phase 7 — default cutover and failure drills

**Assistant work:**

- make K3s default for the accepted workload classes;
- keep direct SSH fallback as an owner-visible emergency lane;
- add SLO dashboards/alerts, capacity and disk-pressure gates;
- perform API outage, server restart, agent loss, image-pull failure, network-policy and backup-restore drills;
- make rollback one command.

**Owner action:** approve the cutover window and run the package; retain console access during reboot/restore drills.

**Gate:** agreed observation period passes; no silent local/workstation spill; recovery and rollback receipts pass.

### Phase 8 — legacy retirement, upgrades and final main convergence

**Assistant work:**

- delete hand-rolled scheduler/fallback code only after owner acceptance;
- remove legacy hostPath/SSH workspace transport and stale instructions;
- finalize automated node enrollment/decommission, token rotation, backup retention and upgrade runbooks;
- prove rolling upgrade/rollback on one agent before server upgrade;
- consolidate plans/status, land final PRs and verify installed entrypoints from main.

**Owner action:** explicitly authorize fallback deletion and final protected-main merges.

**Gate:** no production caller depends on removed paths; restore and upgrade proofs pass; every package is merged/deployed; docs and source of truth match live state.

## Phase ordering invariants

- Never harden or reboot all nodes in one transaction.
- Never register a candidate in Git before live proof.
- Never use a non-expiring join token in package files, command history or receipts.
- Never merge the stale Factory branch directly.
- Never permit the legacy workload transport to define Factory security.
- Never delete fallback during the default-cutover phase.
- Never infer reboot survival from a service restart.
- Never proceed from a historical markdown receipt when a current machine receipt is available.

## Current receipt

- Supplied archives audited against one another.
- Only two supplied worktree files differ from main; missing Factory preserved WIP confirmed absent.
- Supplied runtime-copy migration hardened with transactional rollback and failure-injection tests.
- Read-only Phase 0 collector and its offline test added.
- Detailed audit recorded at `docs/reports/2026-08-10-k3s-migration-audit.md`.
- Phase 0 changed-files package/application script generation in progress.
- Phase 0 commit `ec69e612bb666a459ba8fff31656616988236e4d` merged through PR #12 at `9990b8cb9b5ebf40acba7c22a2b9763ca4134480`.
- Phase 1 v1–v3 stopped safely before persistent cluster mutation and are superseded by the qualification-first replacement.
- The replacement replays the live launcher/runtime and agent-token topology, proves a pre-change backup before configuration publication, and permits Git publication only after full live success.
- Phase 1 has no product UI, primitive, design-system, or Astryx delta.

## Next executable action

Run `apply-overdeck-k3s-phase1-qualified.sh /home/user/Projects/overdeck` on the trusted workstation and return the generated `overdeck-k3s-phase1-result-*.tar.gz`. Keep any draft pull request unmerged until that receipt is reviewed.
