diff --git a/docs/mockups/k3s-phase2-enrollment.html b/docs/mockups/k3s-phase2-enrollment.html new file mode 100644 index 0000000000000000000000000000000000000000..9f6cbec253e7a396ba45a5220e63d26e2830a9b2 --- /dev/null +++ b/docs/mockups/k3s-phase2-enrollment.html @@ -0,0 +1,28 @@ + + + + + +Overdeck K3s — Phase 2 Enrollment + + +
+
Overdeck · K3s migration

One-command enrollment, proven before mutation

Phase 2 turns the future node join into a deterministic, inspectable transaction. It verifies identity, preflight, recovery, trust and registry contracts while keeping every live mutation path locked.

Phase 2 · dry-run only
+
+
17ordered enrollment steps
0live mutations permitted
3independent recovery doors

Owner experience

Install OSSupported Debian-family host
→
Authenticate TailscaleOne online, unique device identity
→
Run one commandPhase 3 canary will execute the reviewed plan
→
Receive proofNode, recovery, workload and Git receipt
tools/k3s/enroll-node.sh debian4 --dry-run \ + --fixture tools/k3s/test/fixtures/phase2-debian4.json
+

Transaction groups

1–3 · Identity and qualification

Tailscale pin, global collision checks, supported-host preflight.

4–7 · Recovery and host convergence

Durable ledger, three-door snapshot, serial access convergence, existing host profile.

8–10 · Temporary join

Ten-minute token metadata, pinned agent configuration, K3s join over tailscale0.

11–15 · Trust proof

Exact Node identity, quarantine, node-pinned Job, token revocation, tokenless restart proof.

16–17 · Source of truth

Paired non-dispatch registry publication and final audit receipt.

+

Trusted workstation

Owns the plan, ephemeral helpers, cluster observation, receipt validation and Git publication.

no reusable token in fileschecksum-verified helpersprotected labels controller-owned

Candidate cannot

Access Git credentials, cluster-admin kubeconfig, permanent K3s bootstrap authority or trusted scheduling state.

no self-trustno dispatch enablementno Phase 2 write command

Phase boundary

Phase 2: inspect, calculate, preview, validate, publish repository implementation.

Phase 3: bind one reviewed plan digest to one real candidate and unlock its transaction.

+

Paired source-of-truth preview

nonecandidate execution state
falsebuild / E2E order membership
falsefallback dependency

The two previews are content-addressed and must be published as one logical transaction. Enrollment cannot silently add dispatch capacity.

+

Package workflow

Verify ZIPSafe paths, checksums, launcher coupling
→
Isolated worktreeCurrent origin/main remains untouched
→
Run gatesUnit, failure, deterministic integration
→
Validate receiptIndependent fail-closed authorization
→
Draft PRNever merge or push main directly

Return overdeck-k3s-phase2-result-<timestamp>.tar.gz to authorize Phase 3 design.

+
+ diff --git a/docs/plans/2026-08-10-k3s-migration-execution.md b/docs/plans/2026-08-10-k3s-migration-execution.md index 94c1aacb617d11e39282a7836ac84353b57ddcc8..ea0f23493a4d5dc89533bbb1a626be7c6c8fe357 100644 --- a/docs/plans/2026-08-10-k3s-migration-execution.md +++ b/docs/plans/2026-08-10-k3s-migration-execution.md @@ -164,7 +164,7 @@ bash apply-overdeck-k3s-phase0.sh /home/user/Projects/overdeck **Rollback:** before server finalization, restore prior managed files/absence and timer state, restart K3s only when needed, and prove readiness. Qualification failures require no K3s rollback and never create a Git commit. -**Implementation status:** Phase 1 v1–v3 are superseded and must not be merged. Their owner runs stopped safely before persistent cluster mutation. The replacement implementation is qualification-first and includes a sanitized replay fixture from the third receipt, 55 Python unit/failure tests, and a full fake-cluster transaction covering pre-change backup → converge → post-change backup → timer → finalization. Live replacement execution is pending. +**Implementation status:** DONE. The qualification-first replacement passed live qualification, encrypted pre-change and post-change restore proofs, transactional convergence and recurring-timer proof. PR #13 merged and was deployed at `832acc72665e49a1a3f4f19b66fd2810b59d2e48`. A follow-up rollback-safety fix ensures a timer rollback failure cannot block control-plane rollback, with regression coverage and full GitHub gate proof. **Durable references:** @@ -176,24 +176,51 @@ bash apply-overdeck-k3s-phase0.sh /home/user/Projects/overdeck ### Phase 2 — one-command enrollment engine, dry-run first -**Input:** canonical endpoint/config and recovery-door facts from Phase 1. +**Input:** deployed Phase 1 canonical endpoint/config, version lock, verified recovery baseline, and current fleet/buildbox registries. **Assistant work:** -- implement `tools/k3s/enroll-node.sh ` and candidate-host payload; -- add Tailscale identity discovery, uniqueness checks and supported-host preflight; -- add a durable transaction ledger, resumability and exact rollback; -- converge all three recovery doors one host at a time; -- create/revoke short-lived bootstrap tokens without logging values; -- install the pinned K3s agent from canonical config; -- verify Node identity and apply protected labels/taints from the control plane; -- run a node-pinned proof Job; -- stage atomic `fleet.json` + `buildbox-hosts.json` updates only after proof; -- add fixture/integration tests and `--plan`/`--dry-run` modes. +- implement `tools/k3s/enroll-node.sh ` and a stdlib-only workstation orchestrator; +- add checksum-verified, ephemeral, read-only candidate and control-plane helpers; +- add Tailscale identity discovery, exact uniqueness checks and supported-host preflight; +- encode the complete 17-step Phase 3 transaction and rollback order in a deterministic content-addressed plan; +- add a durable plan-bound transaction ledger with strict ordering and resume/rollback contracts; +- model the three independent recovery doors and serial convergence policy; +- represent a 600-second bootstrap token without recording its value; +- define pinned K3s agent, Node identity, protected-label, quarantine and node-pinned proof-Job contracts; +- generate atomic paired `fleet.json` + `buildbox-hosts.json` previews with `execution=none`; +- recursively reject secret-like output; +- independently validate receipts before Git publication; +- add sanitized fixture, unit/failure tests, deterministic shell transaction, documentation and interactive presentation; +- hard-disable every live mutation command until a reviewed Phase 3 canary package. + +**Owner action:** run the standalone Phase 2 package launcher. Default execution uses the sanitized `debian4` fixture; `--live-candidate` performs read-only qualification against a real OS + Tailscale-ready candidate. Return the generated result archive and keep the draft PR unmerged pending review. -**Owner action:** run the phase script in dry-run mode against a named candidate or fixture; approve any required tailnet enrollment policy. +**Gate:** + +- package and launcher checksums pass; +- repository changes are applied only in a fresh isolated worktree from current `origin/main`; +- all required unit, failure-injection, helper, receipt and deterministic transaction gates pass; +- generated plan and registry previews are byte-identical across repeated fixture runs; +- the plan has exactly 17 steps and contains no secret values; +- source registry digests remain unchanged; +- candidate preview uses `execution=none` and has no order/fallback membership; +- all live/candidate/cluster/registry mutation flags are false; +- `phase3_authorized=false`; +- only an independently validated receipt may authorize a candidate branch commit/push/draft PR. + +**Rollback:** Phase 2 has no live cluster or host mutation to undo. A failure leaves only the isolated worktree, gate logs and a receipt with `git_publication_allowed=false`. Preview registries must never be copied over tracked registries manually. + +**Implementation status:** implementation and distribution package complete against post-Phase-1 main. Live canary enrollment remains locked to Phase 3. + +**Durable references:** -**Gate:** no live registry or cluster mutation in dry-run; generated plan is deterministic and contains no secret values; all failure paths have tested cleanup. +- design: `docs/specs/2026-08-11-k3s-node-enrollment-design.md`; +- implementation report: `docs/reports/2026-08-11-k3s-phase2-implementation.md`; +- owner runbook: `docs/runbooks/k3s-node-enrollment.md`; +- agent instructions: `docs/runbooks/k3s-phase2-agent-instructions.md`; +- presentation: `docs/mockups/k3s-phase2-enrollment.html`; +- entrypoint: `tools/k3s/enroll-node.sh`. ### Phase 3 — canary enrollment of one real machine diff --git a/docs/plans/INDEX.md b/docs/plans/INDEX.md index 10bc2b159910f11b29d4446a65316aa18ef7a5dd..92937433709165aa7b7cec33d824cea3fd8e8db7 100644 --- a/docs/plans/INDEX.md +++ b/docs/plans/INDEX.md @@ -28,7 +28,7 @@ Read this file at session start, after compaction, and before claiming work. Eve | FIRE | DONE | [Cluster degraded — oversized mirror trips the fleet](2026-08-11-cluster-push-oversize-trip.md) | 15.3 GB non-git `Press.zone` mirror times out the 120 s push budget and marks every host unhealthy | Landed `b86ee4380` + deployed; doctor green on ssh/parity/rsync for all three hosts; debian2 back in both spill orders | | FIRE | IDLE | [Emergency Offload](2026-08-10-emergency-offload.md) | #170 restore remote execution and laptop usability | Whole-agent remote seat completed on Debian2, exit 0; customer payload and spill pending | | FIRE | BLOCKED | [Factory on k3s](2026-08-10-factory-k3s.md) | #96/#175 immutable k3s execution vertical slice | Repair dispatch `bl1io3br2` started nothing: Debian3 seat capacity 3/3; preserved WIP unchanged | -| FIRE | ACTIVE | [K3s migration and node enrollment](2026-08-10-k3s-migration-execution.md) | Reconcile scheduler/Factory tracks; phased cluster migration; one-command post-Tailscale enrollment | Phase 0 merged at `9990b8cb`; Phase 1 v1–v3 safely stopped and superseded; qualification-first replacement prepared with live-topology replay and pre-change recovery gate | +| FIRE | ACTIVE | [K3s migration and node enrollment](2026-08-10-k3s-migration-execution.md) | Reconcile scheduler/Factory tracks; phased cluster migration; one-command post-Tailscale enrollment | Phase 1 merged/deployed at `832acc72`; Phase 2 deterministic zero-mutation enrollment planner and package complete; next gate is owner dry-run receipt and reviewed Phase 3 canary authorization | | 2 | BLOCKED | [Seat Image Node Drift](2026-08-11-seat-image-node-drift.md) | debian1 seat image node v22≠v24; land-queue gate red fleet-wide | Handed to k3s/node-image lane; direct-land escape in use | | 1 | ACTIVE | [Flow-First Operating Model](2026-08-10-flow-first-operating-model.md) | Permanent delivery workflow adjustment | Owner goal active: main session MUST continuously execute; background waits route immediately to another lane, never stop | | 2 | ACTIVE | [Factory Reliability](2026-08-10-factory-reliability.md) | #1/#151/#171/#181/#182/#183 transport, supervision, timeout fixture, disk admission, UI kill/copy | #1 landed/deployed at 747fe53f; #151 passed 10 focused runs and 144-test Factory suite | diff --git a/docs/reports/2026-08-11-k3s-phase2-implementation.md b/docs/reports/2026-08-11-k3s-phase2-implementation.md new file mode 100644 index 0000000000000000000000000000000000000000..4d6bf21d5f5975c638bc4e72ff52c100254854bc --- /dev/null +++ b/docs/reports/2026-08-11-k3s-phase2-implementation.md @@ -0,0 +1,50 @@ +# K3s Phase 2 implementation report + +## Baseline + +Phase 2 was derived from the fresh post-Phase-1 merge checkout at: + +```text +832acc72665e49a1a3f4f19b66fd2810b59d2e48 +``` + +Phase 1 was reported deployed and its encrypted pre/post recovery backups were verified before Phase 2 work began. + +## Delivered implementation + +- trusted-workstation entrypoint `tools/k3s/enroll-node.sh`; +- stdlib-only enrollment orchestrator and common contract library; +- read-only ephemeral candidate and control-plane helpers; +- exact Tailscale identity and collision gates; +- candidate capacity, systemd, cgroup, clock, root-path and pre-existing-agent checks; +- deterministic 17-step transaction and rollback plan; +- durable transaction ledger with strict ordering and plan binding; +- three-door recovery contract; +- temporary-token metadata without token values; +- paired non-dispatch registry previews; +- recursive secret scanning; +- independent receipt validator; +- sanitized live-topology fixture; +- unit, failure-injection, helper, validator and deterministic shell tests; +- design, runbook, agent instructions and interactive migration presentation. + +## Safety improvements found during implementation + +Two defects were corrected before packaging: + +1. Python does not classify the carrier-grade shared-address block as ordinary private IPv4 on every version. Candidate validation now explicitly requires membership in Tailscale's `100.64.0.0/10` range. +2. The initial plan self-check inverted digest equality. It now recomputes the canonical hash domain and requires an exact match. + +The recursive secret scanner was also narrowed so structural metadata such as `bootstrap_token.value_recorded=false` is accepted while scalar token/password/private-key fields and recognizable token/key material are rejected. + +## Deliberate Phase 2 limits + +- no K3s install or join; +- no token creation or revocation; +- no host/recovery-door convergence; +- no Kubernetes labels, taints, Jobs or Node deletion; +- no tracked registry publication; +- no Phase 3 authorization; +- no product UI or new UI primitive. + +The external launcher may publish the repository implementation to a draft Git branch only after an independently validated zero-mutation receipt. diff --git a/docs/runbooks/k3s-node-enrollment.md b/docs/runbooks/k3s-node-enrollment.md new file mode 100644 index 0000000000000000000000000000000000000000..44352c068103f6dacf70bb62d7d34d0156c8ec39 --- /dev/null +++ b/docs/runbooks/k3s-node-enrollment.md @@ -0,0 +1,89 @@ +# K3s node enrollment runbook + +## Phase 2 package execution + +Run the external package launcher as the normal repository owner. Do not use `sudo` around the launcher. + +```bash +./apply-overdeck-k3s-phase2.sh /home/user/Projects/overdeck +``` + +The default run uses the shipped sanitized `debian4` fixture. It cannot contact or modify a real candidate. It creates an isolated worktree, runs the test gates, creates and validates a dry-run receipt, commits the exact Phase 2 repository delta, pushes a candidate branch, and opens a draft pull request when GitHub CLI authentication is available. + +To perform read-only qualification of a real machine that already has its OS and Tailscale installed: + +```bash +./apply-overdeck-k3s-phase2.sh /home/user/Projects/overdeck \ + --candidate debian4 \ + --live-candidate +``` + +That mode reads Tailscale identity, candidate preflight, Phase 1 control-plane contracts and Kubernetes Node inventory. It still cannot install K3s, create a token, change SSH, label a Node, run a proof Job, edit registries, or authorize Phase 3. + +## Package results + +The launcher prints a result archive similar to: + +```text +/home/user/Projects/overdeck-k3s-phase2-result-.tar.gz +``` + +Return that archive for review. Keep the pull request as a draft and unmerged until the receipt is accepted. + +The result contains: + +- outer launcher result; +- gate ledger and logs; +- exact base, branch and candidate commit; +- Phase 2 plan and text summary; +- candidate preflight and uniqueness proof; +- durable unexecuted transaction ledger; +- paired registry previews; +- independent receipt-validation result; +- sanitized package metadata. + +It must not contain SSH keys, K3s token values, kubeconfig contents, Age identities, private keys, process environments, or GitHub credentials. + +## Direct repository command + +After Phase 2 is merged, the repository entrypoint may be used directly: + +```bash +tools/k3s/enroll-node.sh debian4 --dry-run \ + --fixture tools/k3s/test/fixtures/phase2-debian4.json +``` + +For live read-only planning: + +```bash +tools/k3s/enroll-node.sh debian4 --plan +``` + +The direct command writes a new receipt directory in the current directory unless `--receipt-dir` is supplied. + +## Interpreting failure + +A failed Phase 2 run is safe by construction. Confirm the result records: + +```text +git_publication_allowed=false +live_mutation_performed=false +candidate_mutation_performed=false +cluster_mutation_performed=false +registry_mutation_performed=false +phase3_authorized=false +``` + +The launcher leaves its isolated worktree and logs intact. Do not manually copy registry previews into the repository. Return the archive for correction. + +## Phase 3 prerequisite + +Before the canary package is generated, provide: + +1. the accepted Phase 2 result archive; +2. the exact merged Phase 2 commit or a fresh `main` checkout containing it; +3. the intended candidate name; +4. confirmation that the operating system and Tailscale are installed and authenticated; +5. approval for any required Tailscale SSH/tag policy change. + +Phase 3 will be the first package allowed to perform live enrollment. It will bind authorization to one candidate identity and one reviewed plan digest. diff --git a/docs/runbooks/k3s-phase2-agent-instructions.md b/docs/runbooks/k3s-phase2-agent-instructions.md new file mode 100644 index 0000000000000000000000000000000000000000..b8e8ae7f3710128bc4f6cd01e69fa8f94617fe83 --- /dev/null +++ b/docs/runbooks/k3s-phase2-agent-instructions.md @@ -0,0 +1,36 @@ +# Agent instructions — Overdeck K3s Phase 2 + +## Scope + +Apply and verify only the Phase 2 one-command enrollment planning implementation. Phase 2 is dry-run/read-only. Do not join a node, create a K3s token, alter SSH, mutate Kubernetes, update live registries, merge a pull request, or enable dispatch. + +## Required procedure + +1. Run `apply-overdeck-k3s-phase2.sh` as the normal repository owner. +2. Preserve the shared checkout exactly; the launcher must create a fresh isolated worktree from current `origin/main`. +3. Treat package checksums, changed-path allowlist, semantic plan merges, all required gates, and the independent receipt validator as fail-closed. +4. Use the shipped fixture unless the owner explicitly passes `--live-candidate`. +5. Keep any created pull request in draft state. +6. Return the printed `overdeck-k3s-phase2-result-*.tar.gz` archive. + +## Success criteria + +- all required gates pass; +- `plan.json` and registry previews are deterministic across repeated fixture runs; +- the plan has 17 ordered steps and no secret values; +- the candidate preview has `execution=none` and no order/fallback membership; +- source registry digests are unchanged; +- every mutation flag is false; +- `phase3_authorized=false`; +- the independent validator returns `status=passed`; +- the exact package delta is committed and optionally pushed to a draft branch only after receipt authorization. + +## Forbidden actions + +- Do not run the launcher as root. +- Do not invoke `enroll-node.py --mode apply` or the wrapper with `--apply`/`--execute`. +- Do not edit the shared checkout. +- Do not copy preview registries over tracked registries. +- Do not expose or collect token values, kubeconfig content, private keys or environment variables. +- Do not push directly to `main` or merge the draft pull request. +- Do not create or modify product UI primitives. Phase 2 has no product UI scope. diff --git a/docs/specs/2026-08-11-k3s-node-enrollment-design.md b/docs/specs/2026-08-11-k3s-node-enrollment-design.md new file mode 100644 index 0000000000000000000000000000000000000000..89a3caa5148624e0a6940bfd2b08649f09edb218 --- /dev/null +++ b/docs/specs/2026-08-11-k3s-node-enrollment-design.md @@ -0,0 +1,134 @@ +# K3s one-command node enrollment design + +Status: Phase 2 dry-run implementation complete; live mutation remains locked for Phase 3. + +## Objective + +After the operating system and Tailscale are installed and authenticated, the owner-facing enrollment path becomes one workstation command: + +```bash +tools/k3s/enroll-node.sh debian4 +``` + +Phase 2 proves the complete transaction shape without joining a node. Phase 3 will unlock the same reviewed plan for one named canary only. + +## Trust boundary + +The workstation is the trusted controller. A candidate machine never receives Git credentials, a reusable K3s server token, cluster-admin kubeconfig, or authority to label itself trusted. The candidate-side helper is checksum-verified, ephemeral, root-executed through `sudo -n`, read-only in Phase 2, and deleted after inspection. + +The control-plane helper is also checksum-verified, ephemeral, and read-only. It returns only the Phase 1 canonical endpoint/version contracts and health evidence. It never returns the server token, agent token, kubeconfig content, private keys, process environment, or command lines. + +## Phase 2 contract + +Phase 2 supports two modes: + +- `--dry-run --fixture …`: deterministic offline proof used by CI and the packaged launcher. +- `--plan`: read-only inspection of a real OS + Tailscale-ready candidate. + +`--apply` and `--execute` fail closed. There is no Phase 2 mutation subcommand in either remote helper. + +A successful receipt must prove: + +1. exactly one online Tailscale identity matches the requested candidate; +2. candidate name, machine ID and Tailscale IPv4 do not collide with Tailscale, Kubernetes, `fleet.json`, or `buildbox-hosts.json`; +3. the candidate has systemd, cgroup v2, synchronized time, a Tailscale interface, non-interactive root access, at least 4 GiB RAM and 20 GiB free disk, and no active K3s agent; +4. the Phase 1 endpoint, CA digest, pinned K3s version and launcher digest are complete; +5. the transaction contains exactly 17 ordered steps and deterministic rollback actions; +6. the bootstrap token is represented only as metadata with `value_recorded=false` and a 600-second TTL; +7. registry previews add the candidate with `execution=none`, no build/E2E order membership, and no fallback dependency; +8. source registry digests are identical before and after the run; +9. every generated JSON document passes recursive secret scanning; +10. live, candidate, cluster and registry mutation flags are all false. + +## Deterministic transaction + +The plan is content-addressed. Its transaction identifier is derived from candidate identity, Phase 1 contracts, source registry digests, recovery-door declarations and the fixed transaction schema. Timestamps are excluded from `plan.json`, so two runs against identical inputs produce byte-identical plans and registry previews. + +The 17 steps are: + +1. identity discovery; +2. uniqueness gate; +3. supported-host preflight; +4. transaction ledger open; +5. recovery-door snapshot; +6. recovery-door convergence; +7. host-profile convergence; +8. temporary bootstrap-token creation; +9. agent configuration staging; +10. pinned agent install and join; +11. Node identity and Ready gate; +12. protected metadata and quarantine; +13. node-pinned proof Job; +14. bootstrap-token revocation; +15. tokenless restart proof; +16. atomic registry-pair preview/publication; +17. final audit and receipt. + +Steps marked `phase3-write` or `phase3-secret` are descriptive only in Phase 2. The durable ledger remains `planned` with zero events. + +## Recovery doors + +The plan treats the three recovery doors as independent resources: + +- primary OpenSSH on TCP 2222; +- rescue OpenSSH socket on TCP 2223 with independent configuration and host key; +- Tailscale SSH on TCP 22. + +Phase 3 will snapshot and converge one door at a time, proving an already-working door before changing another. It will never make all doors dependent on one daemon or one configuration tree. + +## Cluster join and trust sequence for Phase 3 + +The future mutation path is constrained to: + +```text +identity pin +→ preflight and recovery-door proof +→ short-lived bootstrap token +→ pinned K3s agent join over tailscale0 +→ exact Node identity/Ready proof +→ controller-owned protected labels + pending NoSchedule taint +→ transaction-unique node-pinned proof Job +→ token revocation +→ tokenless agent restart proof +→ paired registry publication with execution=none +→ final receipt +``` + +The candidate cannot remove quarantine or self-assert a trusted scheduling label. Dispatch enablement is a separate reviewed change after enrollment. + +## Registry transaction + +`modules/fleet/fleet.json` and `modules/workstation/claude/buildbox-hosts.json` form one logical transaction. Phase 2 writes previews only. Phase 3 must publish both together with rollback if the second rename fails. The new candidate initially has: + +```json +{ + "execution": "none" +} +``` + +It is excluded from every build order, E2E order and fallback dependency. This prevents enrollment from silently expanding dispatch capacity. + +## Receipt and publication boundary + +The external Phase 2 launcher creates an isolated worktree from current `origin/main`, applies the package, runs all required gates, executes the deterministic dry-run, validates the receipt independently, and only then commits/pushes a draft branch. + +Git publication requires all of the following: + +```text +status=success +phase=2 +mode in {plan,dry-run} +plan_step_count=17 +plan_deterministic=true +secret_scan_passed=true +source_digests_before == source_digests_after +all mutation flags=false +phase3_authorized=false +git_publication_allowed=true +``` + +No package path can push directly to `main` or merge a pull request. + +## UI boundary + +Phase 2 changes no product UI, introduces no UI primitive, and adds no Astryx dependency. The self-contained HTML under `docs/mockups/` is migration documentation, not runtime product UI. diff --git a/tools/k3s/README.md b/tools/k3s/README.md index 5e178a4712db9a95723e587ef5daf9490dc6005b..c6afa71a6e475a9a62567d2366c16312031dd14e 100644 --- a/tools/k3s/README.md +++ b/tools/k3s/README.md @@ -59,3 +59,26 @@ References: - `docs/reports/2026-08-11-k3s-phase1-rederivation.md` - `docs/runbooks/k3s-control-plane-backup-restore.md` - `docs/runbooks/k3s-phase1-agent-instructions.md` + +## Phase 2 — one-command enrollment plan and dry-run + +Primary files: + +- `enroll-node.sh` — owner-facing workstation command; live apply is hard-disabled. +- `enroll-node.py` — identity/preflight collection, deterministic plan, durable ledger and registry previews. +- `lib/phase2_common.py` — plan, safety, secret scanning, ledger and atomic registry-pair contracts. +- `remote/phase2-candidate.py` — ephemeral read-only candidate inspector. +- `remote/phase2-server.py` — ephemeral read-only Phase 1 control-plane contract inspector. +- `validate-enrollment-receipt.py` — independent zero-mutation receipt authorization. +- `test/fixtures/phase2-debian4.json` — sanitized OS + Tailscale-ready candidate fixture. +- `test/phase2-enrollment.test.sh` and `test/test_phase2_*.py` — deterministic, failure-injection and helper tests. + +Phase 2 emits the complete future 17-step transaction but performs no host, Kubernetes, token or tracked-registry mutation. A successful candidate preview uses `execution=none` and is absent from build, E2E and fallback orders. Live canary enrollment remains a separate Phase 3 package bound to one reviewed plan digest. + +References: + +- `docs/specs/2026-08-11-k3s-node-enrollment-design.md` +- `docs/runbooks/k3s-node-enrollment.md` +- `docs/runbooks/k3s-phase2-agent-instructions.md` +- `docs/reports/2026-08-11-k3s-phase2-implementation.md` +- `docs/mockups/k3s-phase2-enrollment.html` diff --git a/tools/k3s/enroll-node.py b/tools/k3s/enroll-node.py new file mode 100755 index 0000000000000000000000000000000000000000..14ff22430e1494c1e96a9ee61557de3b988753cd --- /dev/null +++ b/tools/k3s/enroll-node.py @@ -0,0 +1,521 @@ +#!/usr/bin/env python3 +"""Build a deterministic, secret-free K3s node-enrollment transaction plan. + +Phase 2 intentionally performs no live mutation. It can inspect a real +post-Tailscale candidate or replay the shipped sanitized fixture, validate the +candidate against live/tracked sources of truth, and emit the exact Phase 3 +transaction/rollback plan plus atomic registry previews. +""" +from __future__ import annotations + +import argparse +import json +import os +import secrets +import shlex +import sys +from pathlib import Path +from typing import Any, Callable, Mapping, Sequence + +K3S_DIR = Path(__file__).resolve().parent +LIB_DIR = K3S_DIR / "lib" +if str(LIB_DIR) not in sys.path: + sys.path.insert(0, str(LIB_DIR)) + +from phase1_common import ( # noqa: E402 + atomic_write_json, + build_ssh_command, + exclusive_lock, + load_host_registry, + normalize_path, + read_json, + resolve_host_access, + run_command, + sha256_file, + utc_now, + utc_stamp, +) +from phase2_common import ( # noqa: E402 + CandidateIdentity, + EnrollmentLedger, + Phase2Error, + assert_secret_free, + build_enrollment_plan, + build_registry_previews, + canonical_json_bytes, + recovery_door_contract, + sha256_json, + summarize_plan, + validate_candidate_uniqueness, + validate_host_preflight, +) + +CANDIDATE_HELPER = K3S_DIR / "remote" / "phase2-candidate.py" +SERVER_HELPER = K3S_DIR / "remote" / "phase2-server.py" + + +class Logger: + def __init__(self, path: Path) -> None: + self.path = path + path.parent.mkdir(parents=True, exist_ok=True) + + def __call__(self, message: str) -> None: + line = f"[phase2 {utc_now()}] {message}" + print(line, file=sys.stderr, flush=True) + with self.path.open("a", encoding="utf-8") as handle: + handle.write(line + "\n") + + +def parse_tailscale_status(document: Mapping[str, Any]) -> list[dict[str, Any]]: + peers: list[dict[str, Any]] = [] + + def add(raw: Mapping[str, Any]) -> None: + dns = str(raw.get("DNSName") or "").rstrip(".") + host = str(raw.get("HostName") or "") + ips = [str(item) for item in raw.get("TailscaleIPs", []) if isinstance(item, str)] + if not host and dns: + host = dns.split(".")[0] + if not host and not dns and not ips: + return + peers.append( + { + "name": host, + "dns_name": dns, + "ips": sorted(set(ips)), + "online": bool(raw.get("Online", True)), + } + ) + + self_item = document.get("Self") + if isinstance(self_item, Mapping): + add(self_item) + peer_root = document.get("Peer") + if isinstance(peer_root, Mapping): + for value in peer_root.values(): + if isinstance(value, Mapping): + add(value) + elif isinstance(peer_root, list): + for value in peer_root: + if isinstance(value, Mapping): + add(value) + dedup: dict[tuple[str, str, tuple[str, ...]], dict[str, Any]] = {} + for peer in peers: + key = (peer["name"], peer["dns_name"], tuple(peer["ips"])) + dedup[key] = peer + return sorted(dedup.values(), key=lambda item: (item["name"], item["dns_name"], item["ips"])) + + +def parse_cluster_nodes(document: Mapping[str, Any]) -> list[dict[str, Any]]: + items = document.get("items") + if not isinstance(items, list): + raise Phase2Error("kubectl node document has no items array") + nodes: list[dict[str, Any]] = [] + for item in items: + if not isinstance(item, Mapping): + continue + metadata = item.get("metadata") if isinstance(item.get("metadata"), Mapping) else {} + status = item.get("status") if isinstance(item.get("status"), Mapping) else {} + labels = metadata.get("labels") if isinstance(metadata.get("labels"), Mapping) else {} + conditions = status.get("conditions") if isinstance(status.get("conditions"), list) else [] + addresses = status.get("addresses") if isinstance(status.get("addresses"), list) else [] + ready = "Unknown" + for condition in conditions: + if isinstance(condition, Mapping) and condition.get("type") == "Ready": + ready = str(condition.get("status") or "Unknown") + internal_ip = None + for address in addresses: + if isinstance(address, Mapping) and address.get("type") == "InternalIP": + internal_ip = str(address.get("address") or "") + break + nodes.append( + { + "name": str(metadata.get("name") or ""), + "hostname": str(labels.get("kubernetes.io/hostname") or metadata.get("name") or ""), + "internal_ip": internal_ip, + "ready": ready, + "uid": str(metadata.get("uid") or ""), + } + ) + return sorted(nodes, key=lambda item: item["name"]) + + +def select_tailscale_candidate(peers: Sequence[Mapping[str, Any]], name: str, ssh_user: str) -> CandidateIdentity: + matches = [ + item + for item in peers + if item.get("name") == name or str(item.get("dns_name") or "").split(".")[0] == name + ] + if len(matches) != 1: + raise Phase2Error(f"expected exactly one Tailscale peer named {name!r}; found {len(matches)}") + peer = matches[0] + ipv4 = [value for value in peer.get("ips", []) if isinstance(value, str) and ":" not in value] + if len(ipv4) != 1: + raise Phase2Error(f"candidate {name!r} must have exactly one Tailscale IPv4; found {len(ipv4)}") + # machine-id and OS facts are filled after the remote candidate helper runs. + return CandidateIdentity.from_mapping( + { + "name": name, + "dns_name": str(peer.get("dns_name") or name), + "tailscale_ipv4": ipv4[0], + "machine_id": "0" * 32, + "os_id": "debian", + "os_version_id": "unknown", + "architecture": "x86_64", + "ssh_user": ssh_user, + "rustdesk": None, + } + ) + + +class Orchestrator: + def __init__(self, args: argparse.Namespace) -> None: + self.args = args + self.repo_root = normalize_path(args.repo_root) + self.fleet_path = normalize_path(args.fleet or self.repo_root / "modules/fleet/fleet.json") + self.hosts_path = normalize_path( + args.host_registry or self.repo_root / "modules/workstation/claude/buildbox-hosts.json" + ) + self.kubeconfig = normalize_path(args.kubeconfig) + self.fixture_path = normalize_path(args.fixture) if args.fixture else None + default_receipt = ( + Path.home() / ".local/state/overdeck/k3s-enrollment" / f"phase2-{utc_stamp()}-{os.getpid()}" + ) + self.receipt_dir = normalize_path(args.receipt_dir or default_receipt) + if self.receipt_dir.exists() or self.receipt_dir.is_symlink(): + raise Phase2Error(f"receipt path already exists: {self.receipt_dir}") + self.receipt_dir.mkdir(parents=True, mode=0o700) + (self.receipt_dir / "logs").mkdir(mode=0o700) + (self.receipt_dir / "registry-preview").mkdir(mode=0o700) + self.log = Logger(self.receipt_dir / "logs" / "phase2.log") + self.lock_file = normalize_path( + args.lock_file or Path.home() / ".local/state/overdeck" / f"k3s-enroll-{args.candidate}.lock" + ) + self.candidate_helper_remote: str | None = None + self.server_helper_remote: str | None = None + self.candidate_ssh: list[str] | None = None + self.server_ssh: list[str] | None = None + self.result: dict[str, Any] = { + "schema_version": 1, + "phase": 2, + "mode": args.mode, + "status": "running", + "started_utc": utc_now(), + "candidate": args.candidate, + "repo_root": str(self.repo_root), + "receipt_dir": str(self.receipt_dir), + "fixture": str(self.fixture_path) if self.fixture_path else None, + "steps": [], + "live_mutation_performed": False, + "candidate_mutation_performed": False, + "cluster_mutation_performed": False, + "registry_mutation_performed": False, + "phase3_authorized": False, + "git_publication_allowed": False, + } + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + + def add_step(self, name: str, status: str, **detail: Any) -> None: + item = {"name": name, "status": status, "at_utc": utc_now()} + item.update(detail) + assert_secret_free(item, f"step {name}") + self.result["steps"].append(item) + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + + def save_json(self, relative: str, value: Any) -> None: + assert_secret_free(value, relative) + atomic_write_json(self.receipt_dir / relative, value, mode=0o600) + + def _source_digests(self) -> dict[str, str]: + return { + "fleet.json": sha256_file(self.fleet_path), + "buildbox-hosts.json": sha256_file(self.hosts_path), + } + + def _load_fixture(self) -> dict[str, Any]: + if self.fixture_path is None: + raise Phase2Error("fixture path is not configured") + document = read_json(self.fixture_path) + if not isinstance(document, dict) or document.get("schema_version") != 1: + raise Phase2Error(f"unsupported Phase 2 fixture: {self.fixture_path}") + self.add_step("fixture-load", "passed", fixture_name=document.get("fixture_name")) + return document + + def _candidate_ssh_command(self, dns_name: str, user: str) -> list[str]: + return [ + "ssh", + "-p", + "22", + "-o", + "BatchMode=yes", + "-o", + f"ConnectTimeout={self.args.ssh_timeout}", + "-o", + "ServerAliveInterval=15", + "-o", + "ServerAliveCountMax=2", + "-o", + "StrictHostKeyChecking=yes", + f"{user}@{dns_name}", + ] + + def _upload_helper(self, ssh: Sequence[str], helper: Path, label: str) -> str: + if not helper.is_file() or helper.is_symlink(): + raise Phase2Error(f"unsafe {label} helper: {helper}") + remote = f"/tmp/overdeck-k3s-phase2-{label}-{os.getpid()}-{secrets.token_hex(4)}.py" + run_command(list(ssh) + ["umask 077; cat > " + shlex.quote(remote)], input_bytes=helper.read_bytes(), timeout=60) + run_command(list(ssh) + [shlex.join(["chmod", "0700", remote])], timeout=30) + expected = sha256_file(helper) + completed = run_command(list(ssh) + [shlex.join(["sha256sum", remote])], timeout=30) + actual = str(completed.stdout).split()[0] + if actual != expected: + raise Phase2Error(f"{label} helper SHA-256 mismatch") + self.add_step(f"{label}-helper-upload", "passed", sha256=expected, persistence="ephemeral") + return remote + + def _remote_json(self, ssh: Sequence[str], helper: str, *arguments: str, sudo: bool = True) -> dict[str, Any]: + argv = (["sudo", "-n"] if sudo else []) + ["/usr/bin/python3", helper, *arguments] + completed = run_command(list(ssh) + [shlex.join(argv)], check=False, timeout=180) + stdout = completed.stdout if isinstance(completed.stdout, str) else completed.stdout.decode(errors="replace") + stderr = completed.stderr if isinstance(completed.stderr, str) else completed.stderr.decode(errors="replace") + try: + payload = json.loads(stdout) + except json.JSONDecodeError as exc: + raise Phase2Error( + f"remote helper returned invalid JSON (exit {completed.returncode}); stderr={stderr[-1200:]!r}" + ) from exc + assert_secret_free(payload, "remote helper response") + if completed.returncode != 0 or payload.get("status") == "error": + raise Phase2Error(str(payload.get("error") or f"remote helper exited {completed.returncode}")) + return payload + + def _collect_live(self) -> dict[str, Any]: + tailscale_status = run_command(["tailscale", "status", "--json"], timeout=30) + try: + tailscale_document = json.loads(str(tailscale_status.stdout)) + except json.JSONDecodeError as exc: + raise Phase2Error("tailscale status --json returned invalid JSON") from exc + peers = parse_tailscale_status(tailscale_document) + provisional = select_tailscale_candidate(peers, self.args.candidate, self.args.candidate_user) + self.candidate_ssh = self._candidate_ssh_command(provisional.dns_name, provisional.ssh_user) + self.candidate_helper_remote = self._upload_helper(self.candidate_ssh, CANDIDATE_HELPER, "candidate") + candidate = self._remote_json(self.candidate_ssh, self.candidate_helper_remote, "inspect") + identity_value = dict(candidate.get("identity") or {}) + identity_value.update( + { + "name": provisional.name, + "dns_name": provisional.dns_name, + "tailscale_ipv4": provisional.tailscale_ipv4, + "ssh_user": provisional.ssh_user, + } + ) + identity = CandidateIdentity.from_mapping(identity_value) + + registry = load_host_registry(self.hosts_path) + server_access = resolve_host_access( + registry, + self.args.server, + preferred_door=self.args.server_ssh_door, + require_reachable=True, + ) + self.server_ssh = build_ssh_command(server_access, timeout=self.args.ssh_timeout) + self.server_helper_remote = self._upload_helper(self.server_ssh, SERVER_HELPER, "server") + server = self._remote_json(self.server_ssh, self.server_helper_remote, "inspect") + + nodes_result = run_command( + ["kubectl", "--kubeconfig", str(self.kubeconfig), "get", "nodes", "-o", "json"], timeout=45 + ) + try: + nodes_document = json.loads(str(nodes_result.stdout)) + except json.JSONDecodeError as exc: + raise Phase2Error("kubectl get nodes returned invalid JSON") from exc + cluster_nodes = parse_cluster_nodes(nodes_document) + return { + "candidate": identity.as_dict(), + "candidate_preflight": candidate.get("preflight"), + "tailscale_peers": peers, + "cluster_nodes": cluster_nodes, + "control_plane": server.get("control_plane"), + "version_lock": server.get("version_lock"), + } + + def _cleanup_helpers(self) -> None: + for ssh, path, label in ( + (self.candidate_ssh, self.candidate_helper_remote, "candidate"), + (self.server_ssh, self.server_helper_remote, "server"), + ): + if ssh and path: + completed = run_command(list(ssh) + [shlex.join(["rm", "-f", path])], check=False, timeout=30) + if completed.returncode != 0: + self.result.setdefault("cleanup_warnings", []).append(f"{label} helper cleanup failed") + self.candidate_helper_remote = None + self.server_helper_remote = None + + def run_locked(self) -> int: + before = self._source_digests() + try: + if self.args.mode == "apply": + raise Phase2Error( + "live enrollment is hard-disabled in Phase 2; use the reviewed Phase 3 canary package" + ) + source = self._load_fixture() if self.fixture_path else self._collect_live() + identity = CandidateIdentity.from_mapping(source.get("candidate") or {}) + if identity.name != self.args.candidate: + raise Phase2Error( + f"candidate argument {self.args.candidate!r} does not match discovered identity {identity.name!r}" + ) + fleet = read_json(self.fleet_path) + hosts = read_json(self.hosts_path) + if not isinstance(fleet, dict) or not isinstance(hosts, dict): + raise Phase2Error("source registries must be JSON objects") + uniqueness = validate_candidate_uniqueness( + identity, + fleet, + hosts, + source.get("cluster_nodes") or [], + source.get("tailscale_peers") or [], + ) + self.save_json("uniqueness.json", uniqueness) + self.add_step("candidate-uniqueness", "passed") + preflight = validate_host_preflight(source.get("candidate_preflight") or {}) + self.save_json("candidate-preflight.json", preflight) + self.add_step("candidate-preflight", "passed") + + source_digests = self._source_digests() + plan = build_enrollment_plan( + identity, + control_plane=source.get("control_plane") or {}, + version_lock=source.get("version_lock") or {}, + source_digests=source_digests, + recovery_doors=recovery_door_contract(identity), + ) + self.save_json("plan.json", plan) + (self.receipt_dir / "PLAN.txt").write_text(summarize_plan(plan), encoding="utf-8") + os.chmod(self.receipt_dir / "PLAN.txt", 0o600) + ledger = EnrollmentLedger(self.receipt_dir / "transaction-ledger.json", plan, create=True) + if ledger.next_step() != "identity-discovery" or ledger.pending_rollback(): + raise Phase2Error("new enrollment ledger did not start from a clean state") + self.add_step("deterministic-plan", "passed", plan_sha256=plan["plan_sha256"], step_count=len(plan["steps"])) + + fleet_preview, hosts_preview, pair = build_registry_previews(fleet, hosts, identity) + self.save_json("registry-preview/fleet.json", fleet_preview) + self.save_json("registry-preview/buildbox-hosts.json", hosts_preview) + self.save_json("registry-preview/pair.json", pair) + self.add_step("registry-preview", "passed", execution="none", ordered=False) + + after = self._source_digests() + if before != after: + raise Phase2Error("dry-run changed a tracked source registry") + plan_roundtrip = json.loads(canonical_json_bytes(plan).decode("utf-8")) + if sha256_json({k: v for k, v in plan_roundtrip.items() if k != "plan_sha256"}) != plan["plan_sha256"]: + raise Phase2Error("plan digest self-check failed") + assert_secret_free( + { + "plan": plan, + "fleet_preview": fleet_preview, + "hosts_preview": hosts_preview, + "pair": pair, + }, + "Phase 2 outputs", + ) + self.result.update( + { + "status": "success", + "finished_utc": utc_now(), + "candidate_identity": identity.as_dict(), + "plan_sha256": plan["plan_sha256"], + "transaction_id": plan["transaction_id"], + "plan_step_count": len(plan["steps"]), + "plan_deterministic": True, + "secret_scan_passed": True, + "source_digests_before": before, + "source_digests_after": after, + "registry_preview": pair, + "live_mutation_performed": False, + "candidate_mutation_performed": False, + "cluster_mutation_performed": False, + "registry_mutation_performed": False, + "phase3_authorized": False, + "git_publication_allowed": True, + "next_phase": 3, + } + ) + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + self.log("Phase 2 dry-run plan completed without live mutation") + return 0 + except (Exception, KeyboardInterrupt) as exc: + error = str(exc) or type(exc).__name__ + self.log(f"failure: {error}") + after = self._source_digests() + self.result.update( + { + "status": "failed", + "finished_utc": utc_now(), + "error": error, + "source_digests_before": before, + "source_digests_after": after, + "live_mutation_performed": False, + "candidate_mutation_performed": False, + "cluster_mutation_performed": False, + "registry_mutation_performed": False, + "phase3_authorized": False, + "git_publication_allowed": False, + } + ) + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + return 2 + finally: + self._cleanup_helpers() + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + + def run(self) -> int: + try: + with exclusive_lock(self.lock_file): + return self.run_locked() + except (Exception, KeyboardInterrupt) as exc: + error = str(exc) or type(exc).__name__ + self.result.update( + { + "status": "failed", + "finished_utc": utc_now(), + "error": error, + "git_publication_allowed": False, + } + ) + atomic_write_json(self.receipt_dir / "phase2-result.json", self.result) + self.log(f"failure before locked execution: {error}") + return 2 + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("candidate", help="Tailscale machine name, for example debian4") + parser.add_argument("--mode", choices=["plan", "dry-run", "apply"], default="dry-run") + parser.add_argument("--repo-root", default=str(K3S_DIR.parent.parent)) + parser.add_argument("--fixture", help="Sanitized fixture JSON; skips all live SSH/Kubernetes calls") + parser.add_argument("--fleet") + parser.add_argument("--host-registry") + parser.add_argument("--server", default="debian3") + parser.add_argument("--server-ssh-door", choices=["tailscale_ip", "tailscale_ssh", "lan"], default="tailscale_ip") + parser.add_argument("--candidate-user", default="user") + parser.add_argument("--kubeconfig", default=str(Path.home() / ".kube/config-buildboxes")) + parser.add_argument("--receipt-dir") + parser.add_argument("--lock-file") + parser.add_argument("--ssh-timeout", type=int, default=12) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + if args.ssh_timeout < 1 or args.ssh_timeout > 120: + print("phase2: --ssh-timeout must be between 1 and 120", file=sys.stderr) + return 2 + try: + orchestrator = Orchestrator(args) + except Exception as exc: + print(f"phase2: {exc}", file=sys.stderr) + return 2 + return orchestrator.run() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/k3s/enroll-node.sh b/tools/k3s/enroll-node.sh new file mode 100755 index 0000000000000000000000000000000000000000..deaa8739bd849843f00fc4f4ca4d539622725bb7 --- /dev/null +++ b/tools/k3s/enroll-node.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Trusted-workstation entrypoint for K3s node enrollment planning. +set -euo pipefail +umask 077 +export PYTHONDONTWRITEBYTECODE=1 + +usage() { + cat <<'USAGE' +Usage: + tools/k3s/enroll-node.sh CANDIDATE [--plan|--dry-run] [options] + +Examples: + # Phase 2 deterministic fixture proof (no live calls or mutations) + tools/k3s/enroll-node.sh debian4 --dry-run \ + --fixture tools/k3s/test/fixtures/phase2-debian4.json + + # Phase 2 read-only plan against an OS + Tailscale-ready candidate + tools/k3s/enroll-node.sh debian4 --plan + +Options are forwarded to enroll-node.py. Phase 2 hard-disables live enrollment; +actual mutation is unlocked only by the reviewed Phase 3 canary package. +USAGE +} + +(($# >= 1)) || { usage >&2; exit 2; } +case "${1:-}" in -h|--help) usage; exit 0;; esac +CANDIDATE=$1 +shift +MODE=dry-run +ARGS=() +while (($#)); do + case "$1" in + --plan) MODE=plan; shift ;; + --dry-run) MODE=dry-run; shift ;; + --apply|--execute) + printf 'enroll-node: live enrollment is hard-disabled in Phase 2; use the reviewed Phase 3 package\n' >&2 + exit 2 + ;; + -h|--help) usage; exit 0 ;; + *) ARGS+=("$1"); shift ;; + esac +done + +HERE=$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +exec python3 "$HERE/enroll-node.py" "$CANDIDATE" --mode "$MODE" "${ARGS[@]}" diff --git a/tools/k3s/lib/phase2_common.py b/tools/k3s/lib/phase2_common.py new file mode 100755 index 0000000000000000000000000000000000000000..4d6cee11baad63f3ae645d31c8301bfebfb0e749 --- /dev/null +++ b/tools/k3s/lib/phase2_common.py @@ -0,0 +1,768 @@ +#!/usr/bin/env python3 +"""Shared safety, planning, ledger, and registry helpers for K3s Phase 2. + +Phase 2 is deliberately dry-run only. These helpers make the complete future +transaction inspectable without granting the Phase 2 launcher a live mutation +surface. Phase 3 may reuse the same ledger and plan contracts after a reviewed +receipt authorizes a single named candidate. +""" +from __future__ import annotations + +import copy +import hashlib +import ipaddress +import json +import os +import re +import stat +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable, Mapping, Sequence + +from phase1_common import atomic_write_json, read_json, safe_name, utc_now + + +class Phase2Error(RuntimeError): + """Raised when a Phase 2 contract cannot be proven safely.""" + + +HOSTNAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$") +MACHINE_ID_RE = re.compile(r"^[0-9a-f]{32}$") +HEX64_RE = re.compile(r"^[0-9a-f]{64}$") +BOOTSTRAP_TOKEN_RE = re.compile(r"(?i)\b[a-z0-9]{6}\.[a-z0-9]{16}\b") +K3S_SECURE_TOKEN_RE = re.compile(r"(?i)\bK10[0-9a-f]{64}::[a-z0-9_-]+:[^\s]+") +AGE_SECRET_RE = re.compile(r"AGE-SECRET-KEY-[A-Z0-9-]+") +PRIVATE_KEY_RE = re.compile(r"-----BEGIN (?:OPENSSH |RSA |EC |)PRIVATE KEY-----") +SENSITIVE_KEY_TERMS = { + "password", + "secret_value", + "credential_value", + "private_key", + "client_key", + "identity_value", + "authorization_value", + "kubeconfig_data", + "server_token", + "agent_token", + "token_value", +} +TAILSCALE_IPV4_NETWORK = ipaddress.ip_network("100.64.0.0/10") + +PLAN_STEP_DEFINITIONS: tuple[tuple[str, str, str, tuple[str, ...]], ...] = ( + ( + "identity-discovery", + "Resolve exactly one online Tailscale peer and pin name, DNS name and IPv4.", + "read-only", + (), + ), + ( + "uniqueness-gate", + "Reject candidate name, machine-id or Tailscale-IP collisions across Tailscale, Kubernetes and both registries.", + "read-only", + (), + ), + ( + "supported-host-preflight", + "Verify supported Linux/systemd/cgroup architecture, capacity, clock and non-interactive root path.", + "read-only", + (), + ), + ( + "transaction-ledger-open", + "Create the candidate-scoped durable transaction ledger and bind it to the deterministic plan digest.", + "phase3-write", + ("remove-ledger-after-conclusive-rollback",), + ), + ( + "recovery-door-snapshot", + "Snapshot current 2222, 2223 and Tailscale-SSH state before touching access configuration.", + "phase3-write", + ("restore-recovery-door-snapshot",), + ), + ( + "recovery-door-converge", + "Converge primary SSH, independent rescue SSH and Tailscale SSH serially, proving the prior door before each transition.", + "phase3-write", + ("restore-recovery-door-snapshot", "prove-at-least-one-door"), + ), + ( + "host-profile-converge", + "Converge the declared buildbox host profile through the existing fleet/buildbox machinery.", + "phase3-write", + ("restore-host-profile-snapshot",), + ), + ( + "bootstrap-token-create", + "Create one short-lived agent bootstrap token on the server without recording its value.", + "phase3-secret", + ("revoke-bootstrap-token",), + ), + ( + "agent-config-stage", + "Stage pinned K3s agent configuration using the canonical endpoint, node IP and tailscale0 interface.", + "phase3-write", + ("restore-agent-config-snapshot",), + ), + ( + "agent-install-and-join", + "Install the pinned launcher and start the K3s agent with the temporary token supplied only over a protected pipe.", + "phase3-write", + ("stop-and-uninstall-agent", "restore-agent-config-snapshot"), + ), + ( + "node-identity-ready-gate", + "Require one Kubernetes Node whose name, UID, machine-id evidence and InternalIP match the pinned candidate.", + "read-only", + ("delete-candidate-node",), + ), + ( + "protected-metadata-quarantine", + "Apply controller-owned protected labels plus a pending NoSchedule taint; the candidate cannot self-assert trust.", + "phase3-write", + ("remove-protected-metadata", "cordon-candidate"), + ), + ( + "node-pinned-proof-job", + "Run a transaction-unique proof Job pinned to the candidate and capture API-backed completion evidence.", + "phase3-write", + ("delete-proof-job",), + ), + ( + "bootstrap-token-revoke", + "Revoke the temporary token after proof and verify the agent remains Ready using issued client credentials.", + "phase3-write", + (), + ), + ( + "tokenless-restart-proof", + "Restart the agent without bootstrap material and repeat identity/Ready checks.", + "phase3-write", + ("stop-and-uninstall-agent",), + ), + ( + "registry-pair-preview", + "Generate validated fleet.json and buildbox-hosts.json replacements with execution=none and no order membership.", + "dry-run-output", + ("restore-registry-pair",), + ), + ( + "final-audit-and-receipt", + "Prove all recovery doors and source-of-truth invariants, then emit the redacted enrollment receipt.", + "read-only", + (), + ), +) + + +@dataclass(frozen=True) +class CandidateIdentity: + name: str + dns_name: str + tailscale_ipv4: str + machine_id: str + os_id: str + os_version_id: str + architecture: str + ssh_user: str + rustdesk: str | None = None + + @classmethod + def from_mapping(cls, value: Mapping[str, Any]) -> "CandidateIdentity": + required = { + "name", + "dns_name", + "tailscale_ipv4", + "machine_id", + "os_id", + "os_version_id", + "architecture", + "ssh_user", + } + missing = sorted(required - set(value)) + if missing: + raise Phase2Error(f"candidate identity is missing: {', '.join(missing)}") + candidate = cls( + name=str(value["name"]), + dns_name=str(value["dns_name"]), + tailscale_ipv4=str(value["tailscale_ipv4"]), + machine_id=str(value["machine_id"]), + os_id=str(value["os_id"]), + os_version_id=str(value["os_version_id"]), + architecture=str(value["architecture"]), + ssh_user=str(value["ssh_user"]), + rustdesk=None if value.get("rustdesk") in {None, ""} else str(value.get("rustdesk")), + ) + candidate.validate() + return candidate + + def validate(self) -> None: + if not HOSTNAME_RE.fullmatch(self.name): + raise Phase2Error(f"invalid candidate name: {self.name!r}") + if not self.dns_name or any(ch.isspace() for ch in self.dns_name): + raise Phase2Error("candidate DNS name is invalid") + try: + ip = ipaddress.ip_address(self.tailscale_ipv4) + except ValueError as exc: + raise Phase2Error(f"candidate Tailscale address is invalid: {self.tailscale_ipv4}") from exc + if ip.version != 4 or ip not in TAILSCALE_IPV4_NETWORK: + raise Phase2Error("candidate must use a Tailscale IPv4 address in 100.64.0.0/10") + if not MACHINE_ID_RE.fullmatch(self.machine_id): + raise Phase2Error("candidate machine-id must be 32 lowercase hex characters") + if self.os_id not in {"debian", "ubuntu"}: + raise Phase2Error(f"unsupported candidate OS: {self.os_id!r}") + if self.architecture not in {"x86_64", "amd64"}: + raise Phase2Error(f"unsupported candidate architecture: {self.architecture!r}") + if not HOSTNAME_RE.fullmatch(self.ssh_user): + raise Phase2Error(f"invalid candidate SSH user: {self.ssh_user!r}") + if self.rustdesk is not None and not re.fullmatch(r"[0-9]{6,16}", self.rustdesk): + raise Phase2Error("RustDesk peer id must be 6-16 digits or null") + + def as_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "dns_name": self.dns_name, + "tailscale_ipv4": self.tailscale_ipv4, + "machine_id": self.machine_id, + "os_id": self.os_id, + "os_version_id": self.os_version_id, + "architecture": self.architecture, + "ssh_user": self.ssh_user, + "rustdesk": self.rustdesk, + } + + +def canonical_json_bytes(value: Any) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + + +def sha256_json(value: Any) -> str: + return hashlib.sha256(canonical_json_bytes(value)).hexdigest() + + +def deep_copy_json(value: Any) -> Any: + return json.loads(json.dumps(value)) + + +def validate_sha256(value: str, label: str) -> None: + if not HEX64_RE.fullmatch(value): + raise Phase2Error(f"{label} must be a lowercase SHA-256 digest") + + +def find_secret_strings(value: Any, *, path: str = "$", findings: list[str] | None = None) -> list[str]: + out = findings if findings is not None else [] + if isinstance(value, Mapping): + for key, child in value.items(): + normalized = str(key).lower().replace("-", "_") + if any(term in normalized for term in SENSITIVE_KEY_TERMS): + permitted = child is None or child is False or ( + isinstance(child, str) and child in {"", "", "not-recorded"} + ) + if not permitted: + out.append(f"{path}.{key}: sensitive key carries a value") + find_secret_strings(child, path=f"{path}.{key}", findings=out) + elif isinstance(value, list): + for index, child in enumerate(value): + find_secret_strings(child, path=f"{path}[{index}]", findings=out) + elif isinstance(value, str): + for label, pattern in ( + ("bootstrap token", BOOTSTRAP_TOKEN_RE), + ("K3s secure token", K3S_SECURE_TOKEN_RE), + ("age secret key", AGE_SECRET_RE), + ("private key", PRIVATE_KEY_RE), + ): + if pattern.search(value): + out.append(f"{path}: contains {label}") + return out + + +def assert_secret_free(value: Any, label: str = "document") -> None: + findings = find_secret_strings(value) + if findings: + raise Phase2Error(f"{label} contains secret-like material: {'; '.join(findings[:8])}") + + +def validate_candidate_uniqueness( + identity: CandidateIdentity, + fleet: Mapping[str, Any], + hosts: Mapping[str, Any], + cluster_nodes: Sequence[Mapping[str, Any]], + tailscale_peers: Sequence[Mapping[str, Any]], +) -> dict[str, Any]: + collisions: list[str] = [] + nodes = fleet.get("nodes") + if isinstance(nodes, Mapping) and identity.name in nodes: + collisions.append(f"fleet node name {identity.name!r} already exists") + host_items = hosts.get("hosts") + if not isinstance(host_items, list): + raise Phase2Error("buildbox registry has no hosts array") + for host in host_items: + if not isinstance(host, Mapping): + continue + if host.get("name") == identity.name or host.get("ssh_alias") == identity.name: + collisions.append(f"buildbox host name/alias {identity.name!r} already exists") + if host.get("machine_id") == identity.machine_id: + collisions.append(f"machine-id already belongs to buildbox host {host.get('name')!r}") + access = host.get("access") + if isinstance(access, Mapping): + for door_name, door in access.items(): + if isinstance(door, Mapping) and door.get("host") == identity.tailscale_ipv4: + collisions.append(f"Tailscale IP already belongs to {host.get('name')!r}.{door_name}") + for node in cluster_nodes: + if not isinstance(node, Mapping): + continue + if node.get("name") == identity.name or node.get("hostname") == identity.name: + collisions.append(f"Kubernetes node name {identity.name!r} already exists") + if node.get("internal_ip") == identity.tailscale_ipv4: + collisions.append(f"Kubernetes InternalIP already belongs to {node.get('name')!r}") + peer_matches = [] + for peer in tailscale_peers: + if not isinstance(peer, Mapping): + continue + names = {str(peer.get("name") or ""), str(peer.get("dns_name") or "").split(".")[0]} + ips = {str(item) for item in peer.get("ips", []) if isinstance(item, str)} + if identity.name in names or identity.dns_name == str(peer.get("dns_name") or "") or identity.tailscale_ipv4 in ips: + peer_matches.append(peer) + exact = [ + peer + for peer in peer_matches + if identity.tailscale_ipv4 in {str(item) for item in peer.get("ips", []) if isinstance(item, str)} + and identity.name + in {str(peer.get("name") or ""), str(peer.get("dns_name") or "").split(".")[0]} + ] + if len(exact) != 1: + collisions.append(f"expected one exact online Tailscale identity; found {len(exact)}") + elif exact[0].get("online") is not True: + collisions.append("the exact Tailscale peer is not online") + if collisions: + raise Phase2Error("candidate uniqueness gate failed: " + "; ".join(collisions)) + return { + "status": "passed", + "candidate": identity.name, + "exact_tailscale_matches": 1, + "fleet_name_unique": True, + "registry_identity_unique": True, + "cluster_identity_unique": True, + } + + +def validate_host_preflight(preflight: Mapping[str, Any]) -> dict[str, Any]: + required_true = ( + "systemd", + "cgroup_v2", + "tailscale_online", + "tailscale_interface", + "sudo_noninteractive", + "clock_synchronized", + ) + failures = [name for name in required_true if preflight.get(name) is not True] + memory = int(preflight.get("memory_bytes") or 0) + disk = int(preflight.get("disk_free_bytes") or 0) + if memory < 4 * 1024**3: + failures.append("memory_bytes<4GiB") + if disk < 20 * 1024**3: + failures.append("disk_free_bytes<20GiB") + existing = str(preflight.get("k3s_agent_state") or "absent") + if existing not in {"absent", "inactive"}: + failures.append(f"k3s_agent_state={existing}") + if failures: + raise Phase2Error("candidate preflight failed: " + ", ".join(failures)) + return { + "status": "passed", + "memory_bytes": memory, + "disk_free_bytes": disk, + "k3s_agent_state": existing, + "required_checks": list(required_true), + } + + +def _validate_fleet_shape(fleet: Mapping[str, Any]) -> None: + if fleet.get("schema_version") != 1 or not isinstance(fleet.get("nodes"), Mapping): + raise Phase2Error("unsupported fleet.json structure") + fallback = fleet.get("fallback") + if not isinstance(fallback, Mapping): + raise Phase2Error("fleet.json fallback block is missing") + + +def _validate_host_registry_shape(hosts: Mapping[str, Any]) -> None: + if hosts.get("schema_version") != 1 or not isinstance(hosts.get("hosts"), list): + raise Phase2Error("unsupported buildbox-hosts.json structure") + if not isinstance(hosts.get("orders"), Mapping): + raise Phase2Error("buildbox registry orders block is missing") + + +def build_registry_previews( + fleet: Mapping[str, Any], + hosts: Mapping[str, Any], + identity: CandidateIdentity, + *, + identity_file: str = "~/.ssh/id_ed25519_buildbox", +) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any]]: + _validate_fleet_shape(fleet) + _validate_host_registry_shape(hosts) + fleet_out = deep_copy_json(fleet) + hosts_out = deep_copy_json(hosts) + if identity.name in fleet_out["nodes"]: + raise Phase2Error(f"fleet preview would replace existing node {identity.name!r}") + if any(item.get("name") == identity.name for item in hosts_out["hosts"] if isinstance(item, dict)): + raise Phase2Error(f"host preview would replace existing host {identity.name!r}") + fleet_out["nodes"][identity.name] = { + "transport": "ssh", + "host_ref": identity.name, + "roles": ["builder", "agent-runtime"], + "profiles": ["shared-agent-tools", "buildbox", "buildbox-root"], + "execution": "none", + } + host_entry = { + "name": identity.name, + "ssh_alias": identity.name, + "state": "reachable", + "machine_id": identity.machine_id, + "roles": ["builder", "agent-seat", "e2e", "agent-sandbox"], + "access": { + "lan": None, + "tailscale_ip": { + "host": identity.tailscale_ipv4, + "port": 2222, + "user": identity.ssh_user, + "identity_file": identity_file, + }, + "tailscale_ssh": { + "host": identity.dns_name, + "port": 22, + "user": identity.ssh_user, + "identity_file": None, + }, + }, + "rustdesk": identity.rustdesk, + "notes": "Planned Phase 3 enrollment; execution remains disabled until live proof and a separate dispatch-authorization change.", + } + hosts_out["hosts"].append(host_entry) + for order_name, order in hosts_out["orders"].items(): + if identity.name in order: + raise Phase2Error(f"candidate must not enter registry order {order_name!r} during enrollment") + for name in fleet_out.get("fallback", {}).get("requires_all_unavailable", []): + if name == identity.name: + raise Phase2Error("candidate must not enter fallback dependencies during enrollment") + proof = { + "schema_version": 1, + "candidate": identity.name, + "fleet_sha256": sha256_json(fleet_out), + "hosts_sha256": sha256_json(hosts_out), + "execution": "none", + "in_build_order": False, + "in_e2e_order": False, + "in_fallback_dependencies": False, + "pair_commit_required": True, + } + assert_secret_free(fleet_out, "fleet preview") + assert_secret_free(hosts_out, "buildbox registry preview") + return fleet_out, hosts_out, proof + + +def build_enrollment_plan( + identity: CandidateIdentity, + *, + control_plane: Mapping[str, Any], + version_lock: Mapping[str, Any], + source_digests: Mapping[str, str], + recovery_doors: Sequence[Mapping[str, Any]], +) -> dict[str, Any]: + endpoint = str(control_plane.get("api", {}).get("endpoint") or "") + ca_sha = str(control_plane.get("api", {}).get("cacerts_sha256") or "") + version = str(version_lock.get("version") or "") + launcher = version_lock.get("launcher") if isinstance(version_lock.get("launcher"), Mapping) else {} + launcher_sha = str(launcher.get("sha256") or "") + if not endpoint.startswith("https://"): + raise Phase2Error("control-plane contract has no canonical HTTPS endpoint") + validate_sha256(ca_sha, "control-plane CA hash") + validate_sha256(launcher_sha, "K3s launcher hash") + if not version.startswith("v"): + raise Phase2Error("version lock has no pinned K3s version") + for label, digest in source_digests.items(): + validate_sha256(str(digest), f"source digest {label}") + if len(recovery_doors) != 3: + raise Phase2Error("exactly three recovery-door definitions are required") + base = { + "schema_version": 1, + "phase": 2, + "mode": "dry-run", + "live_mutation_allowed": False, + "candidate": identity.as_dict(), + "cluster": { + "endpoint": endpoint, + "cacerts_sha256": ca_sha, + "k3s_version": version, + "launcher_sha256": launcher_sha, + "new_agent_node_ip": identity.tailscale_ipv4, + "new_agent_flannel_interface": "tailscale0", + }, + "source_digests": dict(sorted((str(k), str(v)) for k, v in source_digests.items())), + "recovery_doors": [deep_copy_json(item) for item in recovery_doors], + "bootstrap_token": { + "kind": "temporary-kubeadm-style-agent-token", + "ttl_seconds": 600, + "value_recorded": False, + "revoke_before_finalization": True, + }, + "scheduling": { + "protected_label_prefix": "node-restriction.kubernetes.io/", + "initial_taint": "overdeck.io/enrollment=pending:NoSchedule", + "proof_job_node_pinned": True, + "registry_execution": "none", + }, + "steps": [], + } + seed = sha256_json(base) + transaction_id = f"od-enroll-{safe_name(identity.name)}-{seed[:16]}" + steps: list[dict[str, Any]] = [] + for index, (name, description, boundary, rollback) in enumerate(PLAN_STEP_DEFINITIONS, start=1): + steps.append( + { + "ordinal": index, + "name": name, + "description": description, + "boundary": boundary, + "resume_policy": "verify-then-continue", + "rollback": list(rollback), + } + ) + base["transaction_id"] = transaction_id + base["steps"] = steps + base["rollback_order"] = [ + item + for step in reversed(steps) + for item in step["rollback"] + if item + ] + base["plan_sha256"] = sha256_json(base) + assert_secret_free(base, "enrollment plan") + return base + + +class EnrollmentLedger: + """Atomic, append-only logical ledger for one enrollment transaction. + + The file itself is rewritten atomically so a torn append cannot make a later + run guess. Event sequencing is validated against the deterministic plan. + """ + + def __init__(self, path: Path, plan: Mapping[str, Any], *, create: bool = False) -> None: + self.path = path + self.plan = deep_copy_json(plan) + if create: + if path.exists() or path.is_symlink(): + raise Phase2Error(f"refusing to replace enrollment ledger: {path}") + document = { + "schema_version": 1, + "transaction_id": plan.get("transaction_id"), + "candidate": plan.get("candidate", {}).get("name"), + "plan_sha256": plan.get("plan_sha256"), + "status": "planned", + "events": [], + "created_utc": utc_now(), + "updated_utc": utc_now(), + } + atomic_write_json(path, document, mode=0o600) + self.document = self._load() + self._validate_binding() + + def _load(self) -> dict[str, Any]: + document = read_json(self.path) + if not isinstance(document, dict) or document.get("schema_version") != 1: + raise Phase2Error(f"unsupported enrollment ledger: {self.path}") + if not isinstance(document.get("events"), list): + raise Phase2Error("enrollment ledger events must be an array") + return document + + def _validate_binding(self) -> None: + for key in ("transaction_id", "plan_sha256"): + if self.document.get(key) != self.plan.get(key): + raise Phase2Error(f"enrollment ledger {key} does not match the plan") + if self.document.get("candidate") != self.plan.get("candidate", {}).get("name"): + raise Phase2Error("enrollment ledger candidate does not match the plan") + + def _write(self) -> None: + self.document["updated_utc"] = utc_now() + assert_secret_free(self.document, "enrollment ledger") + atomic_write_json(self.path, self.document, mode=0o600) + + @property + def completed_steps(self) -> list[str]: + return [ + str(event["step"]) + for event in self.document["events"] + if event.get("event") == "completed" and isinstance(event.get("step"), str) + ] + + def next_step(self) -> str | None: + completed = set(self.completed_steps) + for step in self.plan.get("steps", []): + name = step.get("name") + if isinstance(name, str) and name not in completed: + return name + return None + + def record(self, step: str, event: str, **detail: Any) -> None: + valid_steps = [str(item.get("name")) for item in self.plan.get("steps", [])] + if step not in valid_steps: + raise Phase2Error(f"ledger event names unknown step: {step}") + if event not in {"started", "completed", "failed", "rolled-back", "verified"}: + raise Phase2Error(f"unsupported ledger event: {event}") + if event in {"started", "completed"}: + expected = self.next_step() + if step != expected: + raise Phase2Error(f"ledger expected step {expected!r}, not {step!r}") + item = {"sequence": len(self.document["events"]) + 1, "at_utc": utc_now(), "step": step, "event": event} + item.update(detail) + assert_secret_free(item, "ledger event") + self.document["events"].append(item) + if event == "failed": + self.document["status"] = "failed" + elif event == "rolled-back": + self.document["status"] = "rolled-back" + elif event == "completed" and self.next_step() is None: + self.document["status"] = "completed" + elif event in {"started", "completed", "verified"}: + self.document["status"] = "running" + self._write() + + def pending_rollback(self) -> list[str]: + completed = set(self.completed_steps) + actions: list[str] = [] + for step in reversed(self.plan.get("steps", [])): + if step.get("name") not in completed: + continue + for action in step.get("rollback", []): + if action not in actions: + actions.append(str(action)) + return actions + + +def secure_regular_file(path: Path, *, owner_uid: int | None = None, max_mode: int = 0o600) -> os.stat_result: + if path.is_symlink(): + raise Phase2Error(f"refusing symlink: {path}") + try: + info = path.stat() + except OSError as exc: + raise Phase2Error(f"cannot stat {path}: {exc}") from exc + if not stat.S_ISREG(info.st_mode): + raise Phase2Error(f"required regular file: {path}") + if owner_uid is not None and info.st_uid != owner_uid: + raise Phase2Error(f"unexpected owner for {path}: uid {info.st_uid}") + if stat.S_IMODE(info.st_mode) & ~max_mode: + raise Phase2Error(f"permissions too broad for {path}: {stat.filemode(info.st_mode)}") + return info + + +def write_json_pair_atomic( + first_path: Path, + first_value: Any, + second_path: Path, + second_value: Any, + *, + mode: int = 0o600, +) -> None: + """Publish two files with rollback if the second rename fails. + + This helper is used only in tests and future Phase 3 registry publication. + Phase 2 writes previews into a receipt directory, never into the repository. + """ + + if first_path.parent != second_path.parent: + raise Phase2Error("atomic pair publication requires one directory") + directory = first_path.parent + directory.mkdir(parents=True, exist_ok=True) + originals: dict[Path, bytes | None] = {} + modes: dict[Path, int] = {} + for path in (first_path, second_path): + if path.is_symlink(): + raise Phase2Error(f"refusing symlink pair target: {path}") + if path.exists(): + originals[path] = path.read_bytes() + modes[path] = stat.S_IMODE(path.stat().st_mode) + else: + originals[path] = None + temp_paths: list[Path] = [] + try: + for path, value in ((first_path, first_value), (second_path, second_value)): + fd, tmp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=directory) + tmp = Path(tmp_name) + temp_paths.append(tmp) + with os.fdopen(fd, "wb") as handle: + handle.write(json.dumps(value, indent=2, sort_keys=False).encode("utf-8") + b"\n") + handle.flush() + os.fsync(handle.fileno()) + os.chmod(tmp, mode) + os.replace(temp_paths[0], first_path) + os.replace(temp_paths[1], second_path) + dir_fd = os.open(directory, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)) + try: + os.fsync(dir_fd) + finally: + os.close(dir_fd) + except BaseException: + for path in (first_path, second_path): + original = originals[path] + if original is None: + path.unlink(missing_ok=True) + else: + fd, tmp_name = tempfile.mkstemp(prefix=f".{path.name}.rollback.", dir=directory) + tmp = Path(tmp_name) + with os.fdopen(fd, "wb") as handle: + handle.write(original) + handle.flush() + os.fsync(handle.fileno()) + os.chmod(tmp, modes[path]) + os.replace(tmp, path) + raise + finally: + for tmp in temp_paths: + tmp.unlink(missing_ok=True) + + +def recovery_door_contract(identity: CandidateIdentity) -> list[dict[str, Any]]: + return [ + { + "name": "primary-sshd", + "host": identity.tailscale_ipv4, + "port": 2222, + "owner": "ssh.service + buildbox-sshd-access", + "independence": "wildcard-bound OpenSSH; does not depend on tailscaled", + }, + { + "name": "rescue-sshd", + "host": identity.tailscale_ipv4, + "port": 2223, + "owner": "buildbox-rescue-sshd.socket", + "independence": "separate config and host key outside /etc/ssh", + }, + { + "name": "tailscale-ssh", + "host": identity.dns_name, + "port": 22, + "owner": "tailscaled + buildbox-tailscale-ssh.timer", + "independence": "not served by system OpenSSH configuration", + }, + ] + + +def summarize_plan(plan: Mapping[str, Any]) -> str: + lines = [ + f"transaction: {plan.get('transaction_id')}", + f"candidate: {plan.get('candidate', {}).get('name')}", + f"mode: {plan.get('mode')}", + f"plan_sha256: {plan.get('plan_sha256')}", + f"steps: {len(plan.get('steps', []))}", + "live_mutation_allowed: false", + ] + for step in plan.get("steps", []): + lines.append(f"{int(step['ordinal']):02d}. {step['name']} [{step['boundary']}]") + return "\n".join(lines) + "\n" + + +def ensure_exact_keys(mapping: Mapping[str, Any], expected: Iterable[str], label: str) -> None: + actual = set(mapping) + wanted = set(expected) + if actual != wanted: + raise Phase2Error(f"{label} keys differ: missing={sorted(wanted-actual)} unknown={sorted(actual-wanted)}") diff --git a/tools/k3s/remote/phase2-candidate.py b/tools/k3s/remote/phase2-candidate.py new file mode 100755 index 0000000000000000000000000000000000000000..d3aa06c9206676027ebfa1331ff87065d2f8f73c --- /dev/null +++ b/tools/k3s/remote/phase2-candidate.py @@ -0,0 +1,240 @@ +#!/usr/bin/env python3 +"""Read-only candidate inspection helper for Overdeck K3s Phase 2. + +The workstation uploads this file to the candidate, verifies its digest, and +invokes only ``inspect`` through ``sudo -n``. It has no mutation subcommands. +""" +from __future__ import annotations + +import argparse +import json +import os +import platform +import re +import shutil +import socket +import stat +import subprocess +import sys +from pathlib import Path +from typing import Any, Mapping, Sequence + +MACHINE_ID_RE = re.compile(r"^[0-9a-f]{32}$") + + +class InspectError(RuntimeError): + pass + + +def root_path(root: Path, absolute: str) -> Path: + return root / absolute.lstrip("/") if root != Path("/") else Path(absolute) + + +def read_text(root: Path, absolute: str, *, required: bool = True) -> str: + path = root_path(root, absolute) + if path.is_symlink(): + raise InspectError(f"refusing symlink: {absolute}") + try: + info = path.stat() + except FileNotFoundError: + if required: + raise InspectError(f"required file is missing: {absolute}") + return "" + if not stat.S_ISREG(info.st_mode): + raise InspectError(f"required regular file: {absolute}") + return path.read_text(encoding="utf-8", errors="strict") + + +def parse_os_release(text: str) -> dict[str, str]: + out: dict[str, str] = {} + for line in text.splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, value = line.split("=", 1) + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}: + value = value[1:-1] + out[key] = value + return out + + +def parse_meminfo(text: str) -> int: + for line in text.splitlines(): + if line.startswith("MemTotal:"): + fields = line.split() + if len(fields) >= 2 and fields[1].isdigit(): + return int(fields[1]) * 1024 + raise InspectError("cannot read MemTotal from /proc/meminfo") + + +def run(argv: Sequence[str], *, timeout: int = 20) -> subprocess.CompletedProcess[str]: + return subprocess.run( + list(argv), + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + env={"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG": "C"}, + ) + + +def command_path(name: str) -> str | None: + return shutil.which(name, path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin") + + +def systemd_state(unit: str) -> str: + systemctl = command_path("systemctl") + if systemctl is None: + return "systemctl-unavailable" + completed = run([systemctl, "is-active", unit]) + value = completed.stdout.strip() + return value or ("inactive" if completed.returncode else "active") + + +def listening_ports() -> set[int]: + ports: set[int] = set() + for absolute in ("/proc/net/tcp", "/proc/net/tcp6"): + try: + text = Path(absolute).read_text(encoding="ascii", errors="replace") + except OSError: + continue + for line in text.splitlines()[1:]: + fields = line.split() + if len(fields) < 4 or fields[3] != "0A": + continue + try: + ports.add(int(fields[1].rsplit(":", 1)[1], 16)) + except (ValueError, IndexError): + continue + return ports + + +def tailscale_state(root: Path) -> tuple[bool, bool]: + interface = root_path(root, "/sys/class/net/tailscale0").exists() + if root != Path("/"): + marker = root_path(root, "/run/overdeck-fixture/tailscale-online") + return marker.exists(), interface + tailscale = command_path("tailscale") + if tailscale is None: + return False, interface + completed = run([tailscale, "status", "--json"]) + if completed.returncode != 0: + return False, interface + try: + document = json.loads(completed.stdout) + except json.JSONDecodeError: + return False, interface + backend = str(document.get("BackendState") or "").lower() + return backend == "running", interface + + +def clock_synchronized(root: Path) -> bool: + if root != Path("/"): + return root_path(root, "/run/systemd/timesync/synchronized").exists() + timedatectl = command_path("timedatectl") + if timedatectl: + completed = run([timedatectl, "show", "-p", "NTPSynchronized", "--value"]) + if completed.returncode == 0: + return completed.stdout.strip().lower() == "yes" + return Path("/run/systemd/timesync/synchronized").exists() + + +def rustdesk_peer(root: Path) -> str | None: + candidates = ( + "/var/lib/rustdesk/id", + "/root/.config/rustdesk/id", + "/home/user/.config/rustdesk/id", + ) + for absolute in candidates: + try: + value = read_text(root, absolute, required=False).strip() + except InspectError: + continue + if re.fullmatch(r"[0-9]{6,16}", value): + return value + return None + + +def inspect(root: Path) -> dict[str, Any]: + if root == Path("/") and os.geteuid() != 0: + raise InspectError("candidate inspection must run as root") + machine_id = read_text(root, "/etc/machine-id").strip().lower() + if not MACHINE_ID_RE.fullmatch(machine_id): + raise InspectError("/etc/machine-id is not 32 lowercase hexadecimal characters") + os_release = parse_os_release(read_text(root, "/etc/os-release")) + os_id = os_release.get("ID", "").lower() + os_version = os_release.get("VERSION_ID", "") + architecture = platform.machine() if root == Path("/") else read_text(root, "/run/overdeck-fixture/architecture").strip() + memory = parse_meminfo(read_text(root, "/proc/meminfo")) + disk_path = root_path(root, "/var/lib") + disk_free = shutil.disk_usage(disk_path).free + ts_online, ts_interface = tailscale_state(root) + systemd = root_path(root, "/run/systemd/system").is_dir() + cgroup_v2 = root_path(root, "/sys/fs/cgroup/cgroup.controllers").is_file() + if root == Path("/"): + k3s_present = any(Path(path).exists() for path in ("/usr/local/bin/k3s", "/usr/bin/k3s")) + state = systemd_state("k3s-agent.service") + else: + k3s_present = root_path(root, "/usr/local/bin/k3s").exists() or root_path(root, "/usr/bin/k3s").exists() + state_file = root_path(root, "/run/overdeck-fixture/k3s-agent-state") + state = state_file.read_text().strip() if state_file.is_file() else "inactive" + if not k3s_present and state in {"inactive", "unknown", "not-found", "systemctl-unavailable"}: + k3s_state = "absent" + elif state == "active": + k3s_state = "active" + else: + k3s_state = "inactive" + ports = listening_ports() if root == Path("/") else set() + recovery = { + "primary_2222": "listening" if 2222 in ports else "absent-before-converge", + "rescue_2223": "listening" if 2223 in ports else "absent-before-converge", + "tailscale_ssh_22": "tailscale-online" if ts_online else "unavailable", + } + return { + "schema_version": 1, + "status": "ok", + "identity": { + "machine_id": machine_id, + "os_id": os_id, + "os_version_id": os_version, + "architecture": architecture, + "rustdesk": rustdesk_peer(root), + }, + "preflight": { + "systemd": systemd, + "cgroup_v2": cgroup_v2, + "tailscale_online": ts_online, + "tailscale_interface": ts_interface, + "sudo_noninteractive": os.geteuid() == 0, + "clock_synchronized": clock_synchronized(root), + "memory_bytes": memory, + "disk_free_bytes": disk_free, + "k3s_agent_state": k3s_state, + "recovery_doors": recovery, + }, + "mutation_surface": "none", + } + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("command", choices=["inspect"]) + result.add_argument("--fixture-root", default="/") + return result + + +def main(argv: Sequence[str] | None = None) -> int: + args = parser().parse_args(argv) + try: + payload = inspect(Path(args.fixture_root).resolve()) + print(json.dumps(payload, sort_keys=True)) + return 0 + except (Exception, KeyboardInterrupt) as exc: + print(json.dumps({"schema_version": 1, "status": "error", "error": str(exc) or type(exc).__name__}, sort_keys=True)) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/k3s/remote/phase2-server.py b/tools/k3s/remote/phase2-server.py new file mode 100755 index 0000000000000000000000000000000000000000..a4d30d47a74463bf531c8a833a346f2a7905369a --- /dev/null +++ b/tools/k3s/remote/phase2-server.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Read-only K3s control-plane contract helper for Phase 2 enrollment.""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import shutil +import stat +import subprocess +from pathlib import Path +from typing import Any, Mapping, Sequence + +HEX64 = re.compile(r"^[0-9a-f]{64}$") + + +class InspectError(RuntimeError): + pass + + +def rooted(root: Path, absolute: str) -> Path: + return root / absolute.lstrip("/") if root != Path("/") else Path(absolute) + + +def read_json_regular(root: Path, absolute: str) -> dict[str, Any]: + path = rooted(root, absolute) + if path.is_symlink(): + raise InspectError(f"refusing symlink: {absolute}") + try: + info = path.stat() + except FileNotFoundError as exc: + raise InspectError(f"required contract is missing: {absolute}") from exc + if not stat.S_ISREG(info.st_mode): + raise InspectError(f"required regular contract: {absolute}") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise InspectError(f"invalid JSON in {absolute}: {exc}") from exc + if not isinstance(value, dict): + raise InspectError(f"contract must be an object: {absolute}") + return value + + +def discover_launcher(version_lock: Mapping[str, Any]) -> Path: + launcher = version_lock.get("launcher") if isinstance(version_lock.get("launcher"), Mapping) else {} + candidates = [str(launcher.get("invocation_path") or ""), "/usr/local/bin/k3s", "/usr/bin/k3s"] + for candidate in candidates: + if not candidate: + continue + path = Path(candidate) + try: + info = path.stat() + except OSError: + continue + if not stat.S_ISREG(info.st_mode) or not os.access(path, os.X_OK): + continue + if info.st_uid != 0 or stat.S_IMODE(info.st_mode) & 0o022: + continue + return path + raise InspectError("cannot locate a secure operator-facing K3s launcher") + + +def run(argv: Sequence[str], timeout: int = 30) -> subprocess.CompletedProcess[str]: + return subprocess.run( + list(argv), + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + env={"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG": "C"}, + ) + + +def launcher_sha(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def inspect(root: Path) -> dict[str, Any]: + control = read_json_regular(root, "/etc/rancher/k3s/overdeck/control-plane.json") + lock = read_json_regular(root, "/etc/rancher/k3s/overdeck/version-lock.json") + endpoint = str((control.get("api") or {}).get("endpoint") or "") + ca_sha = str((control.get("api") or {}).get("cacerts_sha256") or "") + expected_launcher_sha = str(((lock.get("launcher") or {}) if isinstance(lock.get("launcher"), Mapping) else {}).get("sha256") or "") + if not endpoint.startswith("https://") or not HEX64.fullmatch(ca_sha) or not HEX64.fullmatch(expected_launcher_sha): + raise InspectError("Phase 1 control-plane contracts are incomplete") + if root != Path("/"): + return { + "schema_version": 1, + "status": "ok", + "control_plane": control, + "version_lock": lock, + "service_active": True, + "readyz": "ok", + "launcher_digest_verified": True, + "mutation_surface": "none", + } + if os.geteuid() != 0: + raise InspectError("server inspection must run as root") + launcher = discover_launcher(lock) + actual_sha = launcher_sha(launcher) + if actual_sha != expected_launcher_sha: + raise InspectError("K3s launcher digest differs from the Phase 1 version lock") + systemctl = shutil.which("systemctl", path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin") + if not systemctl: + raise InspectError("systemctl is unavailable") + active = run([systemctl, "is-active", "k3s.service"]) + if active.returncode != 0 or active.stdout.strip() != "active": + raise InspectError("k3s.service is not active") + ready = run([str(launcher), "kubectl", "get", "--raw=/readyz"]) + if ready.returncode != 0 or ready.stdout.strip() != "ok": + raise InspectError("K3s API /readyz did not return ok") + nodes = run([str(launcher), "kubectl", "get", "nodes", "-o", "json"]) + if nodes.returncode != 0: + raise InspectError("K3s launcher could not read Nodes") + try: + node_doc = json.loads(nodes.stdout) + except json.JSONDecodeError as exc: + raise InspectError("K3s Node output is not JSON") from exc + count = len(node_doc.get("items", [])) if isinstance(node_doc, dict) and isinstance(node_doc.get("items"), list) else 0 + if count < 1: + raise InspectError("K3s returned no Nodes") + return { + "schema_version": 1, + "status": "ok", + "control_plane": control, + "version_lock": lock, + "service_active": True, + "readyz": "ok", + "node_count": count, + "launcher_digest_verified": True, + "mutation_surface": "none", + } + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("command", choices=["inspect"]) + result.add_argument("--fixture-root", default="/") + return result + + +def main(argv: Sequence[str] | None = None) -> int: + args = parser().parse_args(argv) + try: + print(json.dumps(inspect(Path(args.fixture_root).resolve()), sort_keys=True)) + return 0 + except (Exception, KeyboardInterrupt) as exc: + print(json.dumps({"schema_version": 1, "status": "error", "error": str(exc) or type(exc).__name__}, sort_keys=True)) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/k3s/test/fixtures/phase2-debian4.json b/tools/k3s/test/fixtures/phase2-debian4.json new file mode 100644 index 0000000000000000000000000000000000000000..b38292d871e3d704512252d6ca8c9bb885f0981a --- /dev/null +++ b/tools/k3s/test/fixtures/phase2-debian4.json @@ -0,0 +1,117 @@ +{ + "schema_version": 1, + "fixture_name": "debian4-post-tailscale-ready", + "candidate": { + "name": "debian4", + "dns_name": "debian4.taild2daa0.ts.net", + "tailscale_ipv4": "100.64.44.44", + "machine_id": "44444444444444444444444444444444", + "os_id": "debian", + "os_version_id": "12", + "architecture": "x86_64", + "ssh_user": "user", + "rustdesk": "28884444" + }, + "candidate_preflight": { + "systemd": true, + "cgroup_v2": true, + "tailscale_online": true, + "tailscale_interface": true, + "sudo_noninteractive": true, + "clock_synchronized": true, + "memory_bytes": 17179869184, + "disk_free_bytes": 536870912000, + "k3s_agent_state": "absent", + "recovery_doors": { + "primary_2222": "absent-before-converge", + "rescue_2223": "absent-before-converge", + "tailscale_ssh_22": "reachable" + } + }, + "tailscale_peers": [ + { + "name": "debian1", + "dns_name": "debian1.taild2daa0.ts.net", + "ips": ["100.106.253.50"], + "online": true + }, + { + "name": "debian2", + "dns_name": "debian2.taild2daa0.ts.net", + "ips": ["100.79.69.43"], + "online": true + }, + { + "name": "debian3", + "dns_name": "debian3.taild2daa0.ts.net", + "ips": ["100.101.104.41"], + "online": true + }, + { + "name": "debian4", + "dns_name": "debian4.taild2daa0.ts.net", + "ips": ["100.64.44.44"], + "online": true + } + ], + "cluster_nodes": [ + { + "name": "debian1", + "hostname": "debian1", + "internal_ip": "100.106.253.50", + "ready": "True", + "uid": "26910daf-5937-47ed-abd0-08bafdd1169c" + }, + { + "name": "debian2", + "hostname": "debian2", + "internal_ip": "100.79.69.43", + "ready": "True", + "uid": "995d6ac9-7032-4fa5-af8d-ecc8a7b6edd7" + }, + { + "name": "debian3", + "hostname": "debian3", + "internal_ip": "100.101.104.41", + "ready": "True", + "uid": "b53d8889-be2f-4118-b549-c6eb96468dae" + } + ], + "control_plane": { + "schema_version": 2, + "managed_by": "overdeck-k3s-phase1", + "server": { + "hostname": "debian3", + "machine_id": "e9e4403979594b95a5f82b2fc7c12d28", + "tailscale_ipv4": "100.101.104.41", + "tailscale_dns_name": "debian3.taild2daa0.ts.net", + "network_interface": "tailscale0" + }, + "api": { + "endpoint": "https://100.101.104.41:6443", + "cacerts_sha256": "1f19b02f2f7485f1a93f527a9e1ef7bb2e4d96ee30469684ddc83e4ec76c0273", + "tls_sans": ["100.101.104.41", "debian3.taild2daa0.ts.net", "debian3"] + }, + "network_enforcement": { + "new_agent_node_ip": "tailscale-ipv4", + "new_agent_flannel_interface": "tailscale0", + "server_node_ip": "observed-not-changed-in-phase1", + "server_flannel_interface": "observed-not-changed-in-phase1" + } + }, + "version_lock": { + "schema_version": 2, + "version": "v1.36.3+k3s1", + "launcher": { + "invocation_path": "/usr/local/bin/k3s", + "resolved_path": "/usr/local/bin/k3s", + "sha256": "2f98a9f8fe5782479ee2d54e70a1b10a7f6fd4cae8d38ed3098452dc6eed76b5", + "release_url": "https://github.com/k3s-io/k3s/releases/download/v1.36.3+k3s1/k3s" + }, + "runtime": { + "present": true, + "resolved_path": "/var/lib/rancher/k3s/data/e1784de142752c8bc4497493cd3619f1edd74e766794812f339d7eb194d89ef4/bin/k3s", + "sha256": "0d09dd62a17a42ddde4ffc2ac68547a574fdc3fb9e6b371e8c1865fd277f4455" + } + } +} diff --git a/tools/k3s/test/k3s-phase2-enrollment-html.test.sh b/tools/k3s/test/k3s-phase2-enrollment-html.test.sh new file mode 100755 index 0000000000000000000000000000000000000000..7d6f04477866248d8df66b3f6f3f40b82dc0f074 --- /dev/null +++ b/tools/k3s/test/k3s-phase2-enrollment-html.test.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +set -euo pipefail +HERE=$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +ROOT=$(CDPATH= cd -- "$HERE/../../.." && pwd) +FILE="$ROOT/docs/mockups/k3s-phase2-enrollment.html" +[[ -s "$FILE" ]] +grep -Fq '' "$FILE" +grep -Fq 'One-command enrollment' "$FILE" +grep -Fq 'data-view="transaction"' "$FILE" +grep -Fq 'execution state' "$FILE" +grep -Fq 'Phase 2 · dry-run only' "$FILE" +grep -Fq 'addEventListener' "$FILE" +! grep -Eqi ']+src=|]+href=https?://' "$FILE" +echo 'phase2 enrollment presentation: passed' diff --git a/tools/k3s/test/phase2-enrollment.test.sh b/tools/k3s/test/phase2-enrollment.test.sh new file mode 100755 index 0000000000000000000000000000000000000000..b2974bb56f5ce8945de0b4b63bebe5b18ec97f92 --- /dev/null +++ b/tools/k3s/test/phase2-enrollment.test.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +set -euo pipefail +umask 077 +export PYTHONDONTWRITEBYTECODE=1 +HERE=$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +ROOT=$(CDPATH= cd -- "$HERE/../../.." && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +run_once() { + local name=$1 + "$ROOT/tools/k3s/enroll-node.sh" debian4 --dry-run \ + --repo-root "$ROOT" \ + --fixture "$HERE/fixtures/phase2-debian4.json" \ + --receipt-dir "$TMP/$name" \ + --lock-file "$TMP/$name.lock" + "$ROOT/tools/k3s/validate-enrollment-receipt.py" --receipt "$TMP/$name" --candidate debian4 \ + >"$TMP/$name.validation.json" +} +run_once first +run_once second +cmp "$TMP/first/plan.json" "$TMP/second/plan.json" +cmp "$TMP/first/PLAN.txt" "$TMP/second/PLAN.txt" +cmp "$TMP/first/registry-preview/fleet.json" "$TMP/second/registry-preview/fleet.json" +cmp "$TMP/first/registry-preview/buildbox-hosts.json" "$TMP/second/registry-preview/buildbox-hosts.json" +cmp "$TMP/first/registry-preview/pair.json" "$TMP/second/registry-preview/pair.json" + +python3 - "$TMP/first/phase2-result.json" <<'PY' +import json, sys +value=json.load(open(sys.argv[1], encoding='utf-8')) +assert value['status']=='success' +assert value['git_publication_allowed'] is True +assert value['plan_step_count']==17 +for key in ('live_mutation_performed','candidate_mutation_performed','cluster_mutation_performed','registry_mutation_performed','phase3_authorized'): + assert value[key] is False, key +assert value['source_digests_before']==value['source_digests_after'] +PY + +if "$ROOT/tools/k3s/enroll-node.sh" debian4 --apply \ + --repo-root "$ROOT" --fixture "$HERE/fixtures/phase2-debian4.json" \ + --receipt-dir "$TMP/forbidden" --lock-file "$TMP/forbidden.lock" >/dev/null 2>&1; then + echo 'Phase 2 unexpectedly accepted --apply' >&2 + exit 1 +fi + +grep -RIEq 'AGE-SECRET-KEY-|BEGIN .*PRIVATE KEY|K10[0-9a-f]{64}::|[a-z0-9]{6}\.[a-z0-9]{16}' "$TMP/first" && { + echo 'secret-like value found in receipt' >&2 + exit 1 +} || true + +echo 'phase2 deterministic dry-run transaction: passed' diff --git a/tools/k3s/test/test_phase2_candidate.py b/tools/k3s/test/test_phase2_candidate.py new file mode 100755 index 0000000000000000000000000000000000000000..ec43f9cda6a78f13f74fe805fa1f289806ab5ad5 --- /dev/null +++ b/tools/k3s/test/test_phase2_candidate.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SCRIPT = HERE.parent / "remote/phase2-candidate.py" +spec = importlib.util.spec_from_file_location("phase2_candidate", SCRIPT) +assert spec and spec.loader +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class CandidateHelperTests(unittest.TestCase): + def make_root(self, root: Path) -> None: + files = { + "etc/machine-id": "4" * 32 + "\n", + "etc/os-release": 'ID=debian\nVERSION_ID="12"\n', + "run/overdeck-fixture/architecture": "x86_64\n", + "run/overdeck-fixture/k3s-agent-state": "inactive\n", + "proc/meminfo": "MemTotal: 16777216 kB\n", + "var/lib/rustdesk/id": "28884444\n", + } + for relative, content in files.items(): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + for relative in ( + "run/systemd/system", + "sys/fs/cgroup", + "sys/class/net/tailscale0", + "run/systemd/timesync", + "var/lib", + ): + (root / relative).mkdir(parents=True, exist_ok=True) + (root / "sys/fs/cgroup/cgroup.controllers").write_text("cpu memory\n") + (root / "run/overdeck-fixture/tailscale-online").write_text("yes\n") + (root / "run/systemd/timesync/synchronized").write_text("yes\n") + + def test_fixture_inspection_is_read_only_and_complete(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.make_root(root) + payload = module.inspect(root) + self.assertEqual(payload["status"], "ok") + self.assertEqual(payload["identity"]["machine_id"], "4" * 32) + self.assertEqual(payload["identity"]["rustdesk"], "28884444") + self.assertTrue(payload["preflight"]["tailscale_online"]) + self.assertEqual(payload["preflight"]["k3s_agent_state"], "absent") + self.assertEqual(payload["mutation_surface"], "none") + + def test_machine_id_symlink_is_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.make_root(root) + (root / "etc/machine-id").unlink() + (root / "etc/machine-id").symlink_to("os-release") + with self.assertRaises(module.InspectError): + module.inspect(root) + + def test_cli_has_no_mutation_command(self): + with self.assertRaises(SystemExit): + module.parser().parse_args(["apply"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/k3s/test/test_phase2_common.py b/tools/k3s/test/test_phase2_common.py new file mode 100755 index 0000000000000000000000000000000000000000..ecf1e0128a662838b5cd04d0c42440b218d4542f --- /dev/null +++ b/tools/k3s/test/test_phase2_common.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import json +import os +import tempfile +import unittest +from pathlib import Path +import sys + +HERE = Path(__file__).resolve().parent +LIB = HERE.parent / "lib" +if str(LIB) not in sys.path: + sys.path.insert(0, str(LIB)) + +from phase2_common import ( + CandidateIdentity, + EnrollmentLedger, + Phase2Error, + assert_secret_free, + build_enrollment_plan, + build_registry_previews, + find_secret_strings, + recovery_door_contract, + sha256_json, + validate_candidate_uniqueness, + validate_host_preflight, + write_json_pair_atomic, +) +from phase1_common import read_json + + +class Phase2CommonTests(unittest.TestCase): + def identity(self, **overrides): + value = { + "name": "debian4", + "dns_name": "debian4.example.ts.net", + "tailscale_ipv4": "100.64.44.44", + "machine_id": "4" * 32, + "os_id": "debian", + "os_version_id": "12", + "architecture": "x86_64", + "ssh_user": "user", + "rustdesk": "28884444", + } + value.update(overrides) + return CandidateIdentity.from_mapping(value) + + def plan(self): + identity = self.identity() + return build_enrollment_plan( + identity, + control_plane={"api": {"endpoint": "https://100.101.104.41:6443", "cacerts_sha256": "1" * 64}}, + version_lock={"version": "v1.36.3+k3s1", "launcher": {"sha256": "2" * 64}}, + source_digests={"fleet.json": "3" * 64, "buildbox-hosts.json": "4" * 64}, + recovery_doors=recovery_door_contract(identity), + ) + + def test_tailscale_shared_range_is_accepted(self): + self.assertEqual(self.identity().tailscale_ipv4, "100.64.44.44") + + def test_non_tailscale_ipv4_is_rejected(self): + with self.assertRaises(Phase2Error): + self.identity(tailscale_ipv4="192.168.1.8") + + def test_invalid_machine_id_is_rejected(self): + with self.assertRaises(Phase2Error): + self.identity(machine_id="abc") + + def test_secret_scanner_allows_token_metadata(self): + assert_secret_free({"bootstrap_token": {"value_recorded": False, "ttl_seconds": 600}}) + + def test_secret_scanner_rejects_token_value(self): + findings = find_secret_strings({"token_value": "abcdef.0123456789abcdef"}) + self.assertTrue(findings) + + def test_plan_has_stable_digest_and_seventeen_steps(self): + first = self.plan() + second = self.plan() + self.assertEqual(first, second) + self.assertEqual(len(first["steps"]), 17) + self.assertEqual(first["plan_sha256"], sha256_json({k: v for k, v in first.items() if k != "plan_sha256"})) + self.assertFalse(first["live_mutation_allowed"]) + self.assertFalse(first["bootstrap_token"]["value_recorded"]) + + def test_preflight_rejects_active_agent(self): + with self.assertRaises(Phase2Error): + validate_host_preflight({ + "systemd": True, "cgroup_v2": True, "tailscale_online": True, + "tailscale_interface": True, "sudo_noninteractive": True, "clock_synchronized": True, + "memory_bytes": 8 * 1024**3, "disk_free_bytes": 100 * 1024**3, + "k3s_agent_state": "active", + }) + + def test_uniqueness_requires_one_exact_peer(self): + identity = self.identity() + with self.assertRaises(Phase2Error): + validate_candidate_uniqueness(identity, {"nodes": {}}, {"hosts": []}, [], []) + + def test_registry_preview_disables_dispatch(self): + fleet = {"schema_version": 1, "nodes": {}, "fallback": {"requires_all_unavailable": []}} + hosts = {"schema_version": 1, "hosts": [], "orders": {"build": [], "e2e": []}} + fleet_out, hosts_out, pair = build_registry_previews(fleet, hosts, self.identity()) + self.assertEqual(fleet_out["nodes"]["debian4"]["execution"], "none") + self.assertEqual(pair["execution"], "none") + self.assertNotIn("debian4", hosts_out["orders"]["build"]) + + def test_ledger_is_bound_and_unexecuted(self): + plan = self.plan() + with tempfile.TemporaryDirectory() as tmp: + ledger = EnrollmentLedger(Path(tmp) / "ledger.json", plan, create=True) + self.assertEqual(ledger.next_step(), "identity-discovery") + self.assertEqual(ledger.pending_rollback(), []) + with self.assertRaises(Phase2Error): + ledger.record("uniqueness-gate", "started") + ledger.record("identity-discovery", "started") + ledger.record("identity-discovery", "completed") + self.assertEqual(ledger.next_step(), "uniqueness-gate") + + def test_atomic_pair_rolls_back_second_rename_failure(self): + first_value = {"old": 1} + second_value = {"old": 2} + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + first = root / "first.json" + second = root / "second.json" + first.write_text(json.dumps(first_value)) + second.write_text(json.dumps(second_value)) + real_replace = os.replace + calls = 0 + def broken(src, dst): + nonlocal calls + calls += 1 + if calls == 2: + raise OSError("injected second publication failure") + return real_replace(src, dst) + import phase2_common + phase2_common.os.replace = broken + try: + with self.assertRaises(OSError): + write_json_pair_atomic(first, {"new": 1}, second, {"new": 2}) + finally: + phase2_common.os.replace = real_replace + self.assertEqual(json.loads(first.read_text()), first_value) + self.assertEqual(json.loads(second.read_text()), second_value) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/k3s/test/test_phase2_orchestrator.py b/tools/k3s/test/test_phase2_orchestrator.py new file mode 100755 index 0000000000000000000000000000000000000000..a607e5477d798920f2f731462fad4d204c9c1dea --- /dev/null +++ b/tools/k3s/test/test_phase2_orchestrator.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SCRIPT = HERE.parent / "enroll-node.py" +spec = importlib.util.spec_from_file_location("phase2_enroll_node", SCRIPT) +assert spec and spec.loader +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class OrchestratorTests(unittest.TestCase): + def fixture(self): + return HERE / "fixtures/phase2-debian4.json" + + def test_tailscale_parser_is_deterministic(self): + doc = { + "Self": {"HostName": "control", "DNSName": "control.ts.net.", "TailscaleIPs": ["100.64.0.1"]}, + "Peer": { + "b": {"HostName": "debian4", "DNSName": "debian4.ts.net.", "TailscaleIPs": ["100.64.44.44"], "Online": True} + }, + } + first = module.parse_tailscale_status(doc) + second = module.parse_tailscale_status(doc) + self.assertEqual(first, second) + self.assertEqual(first[1]["dns_name"], "debian4.ts.net") + + def test_cluster_parser_requires_items(self): + with self.assertRaises(module.Phase2Error): + module.parse_cluster_nodes({}) + + def test_dry_run_success_and_apply_rejected(self): + repo = HERE.parents[2] + with tempfile.TemporaryDirectory() as tmp: + receipt = Path(tmp) / "receipt" + rc = module.main([ + "debian4", "--mode", "dry-run", "--repo-root", str(repo), + "--fixture", str(self.fixture()), "--receipt-dir", str(receipt), + "--lock-file", str(Path(tmp) / "lock"), + ]) + self.assertEqual(rc, 0) + result = json.loads((receipt / "phase2-result.json").read_text()) + self.assertTrue(result["git_publication_allowed"]) + self.assertFalse(result["live_mutation_performed"]) + with tempfile.TemporaryDirectory() as tmp: + rc = module.main([ + "debian4", "--mode", "apply", "--repo-root", str(repo), + "--fixture", str(self.fixture()), "--receipt-dir", str(Path(tmp) / "receipt"), + "--lock-file", str(Path(tmp) / "lock"), + ]) + self.assertEqual(rc, 2) + + def test_candidate_mismatch_fails_closed(self): + repo = HERE.parents[2] + with tempfile.TemporaryDirectory() as tmp: + receipt = Path(tmp) / "receipt" + rc = module.main([ + "other", "--mode", "dry-run", "--repo-root", str(repo), + "--fixture", str(self.fixture()), "--receipt-dir", str(receipt), + "--lock-file", str(Path(tmp) / "lock"), + ]) + self.assertEqual(rc, 2) + result = json.loads((receipt / "phase2-result.json").read_text()) + self.assertFalse(result["git_publication_allowed"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/k3s/test/test_phase2_server.py b/tools/k3s/test/test_phase2_server.py new file mode 100755 index 0000000000000000000000000000000000000000..fbccfa736cf7dd482aa468c1065952bc137d1c5c --- /dev/null +++ b/tools/k3s/test/test_phase2_server.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SCRIPT = HERE.parent / "remote/phase2-server.py" +spec = importlib.util.spec_from_file_location("phase2_server", SCRIPT) +assert spec and spec.loader +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class ServerHelperTests(unittest.TestCase): + def make_root(self, root: Path) -> None: + base = root / "etc/rancher/k3s/overdeck" + base.mkdir(parents=True) + control = { + "schema_version": 2, + "api": {"endpoint": "https://100.101.104.41:6443", "cacerts_sha256": "1" * 64}, + } + lock = { + "schema_version": 2, + "version": "v1.36.3+k3s1", + "launcher": {"invocation_path": "/usr/local/bin/k3s", "sha256": "2" * 64}, + } + (base / "control-plane.json").write_text(json.dumps(control)) + (base / "version-lock.json").write_text(json.dumps(lock)) + + def test_fixture_contract_inspection(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.make_root(root) + payload = module.inspect(root) + self.assertEqual(payload["status"], "ok") + self.assertTrue(payload["launcher_digest_verified"]) + self.assertEqual(payload["mutation_surface"], "none") + + def test_contract_symlink_is_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.make_root(root) + target = root / "etc/rancher/k3s/overdeck/control-plane.json" + target.unlink() + target.symlink_to("version-lock.json") + with self.assertRaises(module.InspectError): + module.inspect(root) + + def test_cli_has_no_mutation_command(self): + with self.assertRaises(SystemExit): + module.parser().parse_args(["create-token"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/k3s/test/test_validate_enrollment_receipt.py b/tools/k3s/test/test_validate_enrollment_receipt.py new file mode 100755 index 0000000000000000000000000000000000000000..14fc4593e41e774f7badc7ca0f2761578038716e --- /dev/null +++ b/tools/k3s/test/test_validate_enrollment_receipt.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ENROLL_SCRIPT = HERE.parent / "enroll-node.py" +VALIDATOR_SCRIPT = HERE.parent / "validate-enrollment-receipt.py" + +def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + +enroll = load("phase2_enroll_for_validator", ENROLL_SCRIPT) +validator = load("phase2_receipt_validator", VALIDATOR_SCRIPT) + + +class ReceiptValidatorTests(unittest.TestCase): + def make_receipt(self, root: Path) -> Path: + receipt = root / "receipt" + rc = enroll.main([ + "debian4", "--mode", "dry-run", "--repo-root", str(HERE.parents[2]), + "--fixture", str(HERE / "fixtures/phase2-debian4.json"), + "--receipt-dir", str(receipt), "--lock-file", str(root / "lock"), + ]) + self.assertEqual(rc, 0) + return receipt + + def test_valid_receipt_passes(self): + with tempfile.TemporaryDirectory() as tmp: + summary = validator.validate(self.make_receipt(Path(tmp)), expected_candidate="debian4") + self.assertEqual(summary["status"], "passed") + self.assertTrue(summary["zero_mutation"]) + + def test_mutation_flag_fails(self): + with tempfile.TemporaryDirectory() as tmp: + receipt = self.make_receipt(Path(tmp)) + path = receipt / "phase2-result.json" + value = json.loads(path.read_text()) + value["cluster_mutation_performed"] = True + path.write_text(json.dumps(value)) + with self.assertRaises(validator.Phase2Error): + validator.validate(receipt) + + def test_preview_dispatch_enablement_fails(self): + with tempfile.TemporaryDirectory() as tmp: + receipt = self.make_receipt(Path(tmp)) + path = receipt / "registry-preview/fleet.json" + value = json.loads(path.read_text()) + value["nodes"]["debian4"]["execution"] = "normal" + path.write_text(json.dumps(value)) + with self.assertRaises(validator.Phase2Error): + validator.validate(receipt) + + def test_plan_digest_tamper_fails(self): + with tempfile.TemporaryDirectory() as tmp: + receipt = self.make_receipt(Path(tmp)) + path = receipt / "plan.json" + value = json.loads(path.read_text()) + value["cluster"]["new_agent_node_ip"] = "100.64.99.99" + path.write_text(json.dumps(value)) + with self.assertRaises(validator.Phase2Error): + validator.validate(receipt) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/k3s/validate-enrollment-receipt.py b/tools/k3s/validate-enrollment-receipt.py new file mode 100755 index 0000000000000000000000000000000000000000..2a07e283bce4b578e65049660b0e7c56cdb60958 --- /dev/null +++ b/tools/k3s/validate-enrollment-receipt.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +"""Fail-closed validator for an Overdeck K3s Phase 2 receipt.""" +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any, Mapping, Sequence + +HERE = Path(__file__).resolve().parent +LIB = HERE / "lib" +if str(LIB) not in sys.path: + sys.path.insert(0, str(LIB)) + +from phase1_common import read_json, sha256_file # noqa: E402 +from phase2_common import Phase2Error, assert_secret_free, canonical_json_bytes, sha256_json # noqa: E402 + + +def require(condition: bool, message: str) -> None: + if not condition: + raise Phase2Error(message) + + +def load_object(path: Path) -> dict[str, Any]: + value = read_json(path) + if not isinstance(value, dict): + raise Phase2Error(f"expected JSON object: {path}") + return value + + +def validate(receipt_dir: Path, *, expected_candidate: str | None = None) -> dict[str, Any]: + root = receipt_dir.resolve() + require(root.is_dir() and not root.is_symlink(), f"receipt directory is unsafe: {root}") + result = load_object(root / "phase2-result.json") + plan = load_object(root / "plan.json") + fleet = load_object(root / "registry-preview/fleet.json") + hosts = load_object(root / "registry-preview/buildbox-hosts.json") + pair = load_object(root / "registry-preview/pair.json") + ledger = load_object(root / "transaction-ledger.json") + assert_secret_free({"result": result, "plan": plan, "fleet": fleet, "hosts": hosts, "pair": pair, "ledger": ledger}) + + require(result.get("schema_version") == 1 and result.get("phase") == 2, "unsupported Phase 2 result") + require(result.get("status") == "success", "Phase 2 result did not succeed") + require(result.get("mode") in {"plan", "dry-run"}, "receipt was not produced by a read-only mode") + candidate = str(result.get("candidate") or "") + require(candidate != "", "receipt candidate is missing") + if expected_candidate is not None: + require(candidate == expected_candidate, "receipt candidate differs from expected candidate") + for key in ( + "live_mutation_performed", + "candidate_mutation_performed", + "cluster_mutation_performed", + "registry_mutation_performed", + "phase3_authorized", + ): + require(result.get(key) is False, f"receipt must record {key}=false") + require(result.get("git_publication_allowed") is True, "receipt does not authorize Git publication") + require(result.get("plan_deterministic") is True, "plan is not declared deterministic") + require(result.get("secret_scan_passed") is True, "secret scan did not pass") + require(result.get("source_digests_before") == result.get("source_digests_after"), "tracked registries changed") + require(result.get("plan_step_count") == 17, "expected exactly 17 enrollment steps") + + require(plan.get("phase") == 2 and plan.get("mode") == "dry-run", "unsupported enrollment plan") + require(plan.get("live_mutation_allowed") is False, "plan permits live mutation") + steps = plan.get("steps") + require(isinstance(steps, list) and len(steps) == 17, "plan must contain 17 steps") + require([step.get("ordinal") for step in steps if isinstance(step, Mapping)] == list(range(1, 18)), "step ordinals are invalid") + expected_digest = sha256_json({k: v for k, v in plan.items() if k != "plan_sha256"}) + require(plan.get("plan_sha256") == expected_digest, "plan SHA-256 self-check failed") + require(result.get("plan_sha256") == expected_digest, "result and plan SHA-256 differ") + require(result.get("transaction_id") == plan.get("transaction_id"), "result and plan transaction differ") + token = plan.get("bootstrap_token") if isinstance(plan.get("bootstrap_token"), Mapping) else {} + require(token.get("value_recorded") is False, "bootstrap token value may not be recorded") + require(token.get("ttl_seconds") == 600, "bootstrap token TTL contract changed") + + require(ledger.get("transaction_id") == plan.get("transaction_id"), "ledger transaction differs") + require(ledger.get("plan_sha256") == expected_digest, "ledger plan binding differs") + require(ledger.get("candidate") == candidate, "ledger candidate differs") + require(ledger.get("status") == "planned" and ledger.get("events") == [], "Phase 2 ledger must remain unexecuted") + + require(pair.get("candidate") == candidate, "registry preview candidate differs") + require(pair.get("execution") == "none", "candidate preview must use execution=none") + require(pair.get("in_build_order") is False, "candidate entered build order") + require(pair.get("in_e2e_order") is False, "candidate entered E2E order") + require(pair.get("in_fallback_dependencies") is False, "candidate entered fallback dependencies") + require(pair.get("pair_commit_required") is True, "registry previews are not transaction-paired") + require(pair.get("fleet_sha256") == sha256_json(fleet), "fleet preview digest differs") + require(pair.get("hosts_sha256") == sha256_json(hosts), "host preview digest differs") + nodes = fleet.get("nodes") if isinstance(fleet.get("nodes"), Mapping) else {} + require(isinstance(nodes.get(candidate), Mapping), "candidate is absent from fleet preview") + require(nodes[candidate].get("execution") == "none", "fleet preview candidate can dispatch") + host_items = hosts.get("hosts") if isinstance(hosts.get("hosts"), list) else [] + matches = [item for item in host_items if isinstance(item, Mapping) and item.get("name") == candidate] + require(len(matches) == 1, "host preview must contain exactly one candidate") + for order_name, order in (hosts.get("orders") or {}).items(): + require(candidate not in order, f"candidate entered host order {order_name}") + require(candidate not in (fleet.get("fallback") or {}).get("requires_all_unavailable", []), "candidate entered fallback") + + summary = { + "schema_version": 1, + "status": "passed", + "candidate": candidate, + "transaction_id": plan.get("transaction_id"), + "plan_sha256": expected_digest, + "step_count": 17, + "zero_mutation": True, + "secret_free": True, + "git_publication_allowed": True, + "files": { + "result_sha256": sha256_file(root / "phase2-result.json"), + "plan_sha256_file": sha256_file(root / "plan.json"), + "fleet_preview_sha256_file": sha256_file(root / "registry-preview/fleet.json"), + "host_preview_sha256_file": sha256_file(root / "registry-preview/buildbox-hosts.json"), + }, + } + return summary + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("--receipt", required=True) + result.add_argument("--candidate") + return result + + +def main(argv: Sequence[str] | None = None) -> int: + args = parser().parse_args(argv) + try: + print(json.dumps(validate(Path(args.receipt), expected_candidate=args.candidate), indent=2, sort_keys=True)) + return 0 + except (Exception, KeyboardInterrupt) as exc: + print(json.dumps({"schema_version": 1, "status": "failed", "error": str(exc) or type(exc).__name__}, sort_keys=True)) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main())