# K3s Phase 1 — qualification-first control-plane baseline, backup, and recovery

Audience: owner, migration agents, and reviewers
Status: rederived implementation prepared; live execution pending
Supersedes: Phase 1 v1, v2, and v3 candidates

## 1. Decision summary

Phase 1 establishes a reproducible recovery baseline around the existing K3s server without changing the K3s version, datastore type, server node IP, current flannel interface, workload scheduling, or existing installer arguments.

The phase owns these durable surfaces:

1. `/etc/rancher/k3s/config.yaml.d/90-overdeck-control-plane.yaml`
2. `/etc/rancher/k3s/overdeck/control-plane.json`
3. `/etc/rancher/k3s/overdeck/version-lock.json`
4. encrypted pre-change and post-change backups on the trusted workstation
5. a hardened recurring workstation-side backup service and timer

The critical change after three safe live stops is ordering. A complete qualification and verified pre-change restore proof are mandatory before the first managed K3s file can be published. Repository commit and push are later still: the external package launcher may publish Git only after the live receipt explicitly authorizes it.

No Overdeck product UI changes in Phase 1. No primitive or Astryx component is added.

## 2. Goals

- Identify the real outer K3s command launcher independently of the running extracted executable.
- Pin the current K3s version and separate launcher/runtime digests without upgrading either.
- Establish the canonical API endpoint, TLS SAN set, server identity, and future-agent network contract.
- Preserve existing installer arguments and non-owned configuration while drift-locking their non-secret shape.
- Qualify `/readyz`, Ready Node inventory, kubeconfig, peer API CA, and serving-certificate identity.
- Support SQLite and embedded-etcd recovery archives.
- Pair every datastore backup with the original server token and a safe agent-token restoration contract.
- Encrypt backups off-host before retention.
- Verify every archive and materialize a complete isolated restore tree without modifying the live server.
- Create a verified pre-change backup before configuration publication and a verified post-change backup before finalization.
- Install an idempotent, locked, hardened recurring backup timer.
- Roll back server and workstation-managed state automatically before the commit point.
- Produce a redacted, machine-readable receipt that can gate Git publication and Phase 2.

## 3. Non-goals

- No K3s version change or binary replacement.
- No datastore conversion.
- No cluster reset or live restore.
- No server reboot.
- No node enrollment.
- No workload migration.
- No namespace, RBAC, quota, network-policy, or Pod Security rollout.
- No removal or translation of unknown installer arguments.
- No Tailscale policy change.
- No direct push or merge to `main`.
- No automatic external escrow of the age private key.

## 4. Trust model

### 4.1 Trusted workstation

The workstation controls the transaction. It holds:

- the repository and host registry;
- SSH recovery-door configuration;
- the kubeconfig used to derive and prove the API endpoint;
- the age private key;
- encrypted off-server recovery archives;
- recurring backup units and configuration;
- qualification, transaction, and package receipts.

The workstation does not persist a decrypted TAR. Decryption and restore materialization occur under a private temporary directory and are removed after verification.

### 4.2 Existing K3s server

The server executes a checksum-verified, stdlib-only Python helper through non-interactive sudo. The helper is uploaded to a random `0700` file under `/tmp` for each invocation and removed at exit. Qualification does not install a persistent privileged helper.

The helper may:

- inspect non-secret service, executable, network, datastore, and file metadata;
- create root-only backup staging under `/var/lib/overdeck/k3s-phase1/backups`;
- create rollback transactions under `/var/lib/overdeck/k3s-phase1/transactions`;
- publish exactly three managed files;
- restart only `k3s.service`, and only when the managed YAML changes;
- remove only phase-owned staging and rollback material.

It never prints token values, token hashes, kubeconfig contents, datastore contents, raw environment values, or unredacted secret-bearing arguments.

### 4.3 Other registry-reachable hosts

Every host declared `reachable` in `buildbox-hosts.json` runs a read-only proof through its registered SSH door:

1. fetch `<canonical-endpoint>/cacerts`;
2. compare the CA bundle SHA-256 with the server helper observation;
3. use that CA to perform hostname/IP-verified TLS against the API endpoint.

A reachable host is not silently skipped. Hosts explicitly declared unreachable are outside the Phase 1 peer gate.

## 5. K3s executable roles

K3s installations may expose two related but non-interchangeable executables:

### 5.1 Outer command launcher

Typical path: `/usr/local/bin/k3s`.

Used for:

- `k3s --version`;
- `k3s kubectl ...`;
- `k3s etcd-snapshot ...`;
- release-artifact digest lock and future upgrade candidate verification.

Discovery order:

1. structured systemd `ExecStart` executable path;
2. canonical K3s installer paths;
3. a deterministic approved root executable path.

The running process executable is never a launcher candidate.

### 5.2 Running executable evidence

Typical path: `/var/lib/rancher/k3s/data/<digest>/bin/k3s`.

Used only to record:

- current running executable path;
- SHA-256;
- trusted ownership/mode/directory-chain status;
- continuity across Phase 1 convergence.

It is not called with nested `kubectl` or `etcd-snapshot` commands.

### 5.3 Privileged executable validation

When the helper runs as root, an executable is accepted only when:

- its invocation path is absolute and named `k3s`;
- the resolved target is a regular executable file;
- the target is root-owned;
- the target is not group/world writable;
- both the invocation and resolved directory chains are root-owned and not group/world writable.

A discovery failure returns sanitized service state and attempted paths. It does not trigger an installation or upgrade.

## 6. Qualification transaction

Qualification is a complete compatibility gate, not a shallow dry run.

### 6.1 Independent checks

The orchestrator records each check as `passed`, `failed`, or `blocked`. Independent checks continue after a failure so one owner run produces the full known compatibility matrix.

Checks:

1. **Age identity and tools**
   - install `age` only when explicitly allowed by the package launcher;
   - create or load a regular non-symlink identity owned by the current user with mode `0600`;
   - derive the public recipient without recording the private value.

2. **Recurring backup preflight**
   - validate every source file;
   - verify user-systemd availability;
   - render service/timer units in a temporary directory;
   - run `systemd-analyze --user verify` when available;
   - inspect linger and non-interactive sudo requirements;
   - do not install any unit.

3. **Server inspection**
   - verify registered SSH identity;
   - upload/checksum the ephemeral helper;
   - discover launcher and runtime separately;
   - require active K3s service;
   - retry outer-launcher `/readyz` for a bounded 30 seconds;
   - require a successful, non-empty Node query with at least one Ready Node;
   - classify datastore;
   - classify server and agent token layouts;
   - record public CA hash and serving certificate facts;
   - reject unfinished control-plane transactions.

4. **Control-plane plan**
   - derive endpoint from kubeconfig, remapping only a loopback host to the verified server Tailscale address while preserving the port;
   - derive deterministic SANs;
   - generate desired managed YAML, contract, and version lock;
   - compare contents without publication;
   - reject external configuration drift against an existing canonical contract.

5. **Workstation API proof**
   - run kubeconfig `/readyz` and require `ok`.

6. **Registry-host API/CA/TLS proof**
   - run the peer proof from every reachable host and require all to match.

7. **Pre-change backup and restore proof**
   - create a datastore/config/token archive on the server;
   - stream it into `age` on the workstation;
   - compare plaintext size and SHA-256 with server metadata;
   - decrypt under a private temporary directory;
   - verify the TAR, manifest, datastore, token contract, and file set;
   - materialize an isolated restore root;
   - remove remote plaintext staging;
   - retain the verified encrypted archive and metadata.

### 6.2 Qualification mutation boundary

Qualification may create:

- the workstation age identity;
- local receipt files;
- a verified encrypted pre-change backup;
- temporary remote helper and backup staging, both removed before completion.

Qualification does not:

- publish K3s configuration;
- restart K3s;
- install the recurring timer;
- create a Git commit;
- push a branch;
- open a pull request.

## 7. Configuration ownership

### 7.1 Managed YAML drop-in

`90-overdeck-control-plane.yaml` contains only:

- `write-kubeconfig-mode: "0600"`;
- deterministic `tls-san+` values.

The drop-in is mode `0600`, root-owned, and atomically replaced. K3s restarts only when its bytes change.

### 7.2 Control-plane contract

`control-plane.json` records:

- hostname, machine ID, Tailscale IPv4/DNS, and `tailscale0`;
- canonical API endpoint and SAN list;
- public CA SHA-256;
- future-agent rule: Tailscale IPv4 plus `tailscale0`;
- explicit decision not to rewrite current server node-IP/flannel settings;
- a non-secret external configuration baseline;
- qualification requirements and observed agent-token layout.

The baseline includes hashes and metadata for non-secret configuration, redacted process arguments, environment key names, and metadata-only records for secret-bearing environment files. Exact files exist only in the encrypted backup.

### 7.3 Version lock schema 2

`version-lock.json` separates:

- `launcher`: invocation path, resolved path, SHA-256, and release URL;
- `runtime`: presence, resolved path, SHA-256, and discovery source;
- K3s version;
- install-script reference;
- explicit version-and-launcher-digest upgrade policy.

A schema-1 lock that conflates launcher and runtime is rejected. Installed version/launcher drift is never silently adopted.

## 8. Token backup model

### 8.1 Server token

The canonical server token must be a root-owned, mode-restricted regular file. Its value and content hash never enter receipts or manifests.

### 8.2 Agent token

Supported forms:

- absent;
- root-owned, mode-restricted regular file;
- symlink resolving exactly to the canonical server token.

A safe symlink is represented in the manifest as:

```json
{
  "classification": "symlink-to-server-token",
  "restore": "symlink-to-token",
  "link_target": "token",
  "payload_path": null
}
```

The TAR contains no symlink. The verifier reconstructs only the relative link `agent-token -> token` inside the isolated restore root.

## 9. Backup and restore contract

### 9.1 SQLite

The helper uses Python’s online SQLite backup API against `state.db`, excludes live WAL/SHM files, and requires `PRAGMA integrity_check = ok` on the copied database.

### 9.2 Embedded etcd

The helper invokes a uniquely named compressed snapshot through the outer launcher, requires the snapshot to appear in `etcd-snapshot ls`, copies it into the phase payload, and removes only the uniquely created source snapshot after TAR staging.

### 9.3 Manifest

Schema 2 records:

- purpose: `prechange`, `postchange`, or `scheduled`;
- server identity;
- K3s version;
- launcher and runtime facts separately;
- public CA hash;
- datastore metadata;
- server-token path;
- agent-token restoration descriptor;
- systemd source paths;
- every payload file’s path, size, SHA-256, mode, and source uid/gid.

It records no secret values or token content hashes.

### 9.4 Canonical file rules

- A `prechange` archive may contain none of the three Phase 1 canonical files.
- If any canonical file is present, all three must be present and consistent.
- `postchange` and `scheduled` archives require the complete canonical set.

### 9.5 Archive validation

The verifier rejects:

- absolute or traversal paths;
- symlinks and hard links;
- devices and FIFOs;
- duplicate members;
- unlisted payload files;
- missing manifest files;
- size, SHA-256, or mode mismatches;
- unsafe token permissions/contracts;
- unsupported datastore types;
- incomplete canonical configuration;
- version-lock/manifest identity mismatch.

### 9.6 Isolated materialization

The restore root must be empty, non-live, and non-symlinked. Materialization reconstructs:

- `/etc/rancher/k3s` configuration;
- server and agent token layout;
- SQLite DB tree or etcd snapshot path;
- captured systemd files;
- `OVERDECK_RESTORE_PLAN.json`.

It does not stop K3s, replace live files, apply ownership, run cluster reset, or activate a restore.

## 10. Server configuration transaction

Changed convergence uses states:

```text
prepared -> files-published -> converged -> finalizing -> finalized
```

Rollback states:

```text
prepared/files-published/converged -> rolled-back
```

A failed rollback becomes `rollback-failed` and blocks later mutation.

### 10.1 Prepare

- refuse another unfinished transaction;
- record each managed path as absent, regular file, or symlink;
- copy prior files without following links;
- atomically publish transaction metadata and rollback material.

### 10.2 Publish and prove

- atomically publish only changed managed files;
- set root ownership and mode `0600`;
- restart K3s only when YAML changed;
- require outer-launcher `/readyz`;
- require successful Ready Node inventory;
- require unchanged launcher and running-runtime digests;
- require every managed SAN in the serving certificate.

### 10.3 Rollback

Before finalization:

- restore exact prior bytes/type/mode/ownership or prior absence;
- restart K3s when required;
- require readiness;
- retain transaction evidence.

### 10.4 Finalization

- mark `finalizing`;
- remove rollback material;
- mark `finalized`;
- make finalization idempotent;
- prohibit rollback after rollback authority may have been removed.

## 11. Apply sequence

A passing qualification returns endpoint, SANs, recipient, plan, server observation, peer results, and verified pre-change artifact.

Apply then executes:

1. transactional managed configuration convergence;
2. fresh server inspect with readiness and Nodes;
3. workstation `/readyz`;
4. peer API/CA/TLS proof;
5. verified encrypted post-change backup and isolated restore proof;
6. recurring backup timer installation;
7. server transaction finalization;
8. backup retention pruning;
9. success receipt with `git_publication_allowed: true`.

A failure returns `git_publication_allowed: false` and attempts timer/config rollback.

## 12. Recurring backup timer

The user-systemd installation copies a self-contained runtime under `~/.local/lib/overdeck/k3s`, including a snapshot of the host registry. The service runs `--mode backup` with:

- an exclusive server-specific lock;
- private config, state, and backup paths;
- `UMask=0077`;
- `NoNewPrivileges=true`;
- `PrivateTmp=true`;
- `ProtectSystem=strict`;
- explicit writable paths;
- bounded runtime and randomized timer delay.

Timer publication is transactional. Previous files, symlinks, enabled state, active state, and phase-enabled linger are restored on pre-finalization failure.

Retention deletes only verified archive/metadata pairs beyond the configured count. Unpaired or malformed evidence is retained for review.

## 13. Git publication contract

The external package launcher:

1. verifies ZIP checksums and launcher coupling;
2. creates an isolated worktree from current `origin/main`;
3. applies stable paths with a three-way patch;
4. semantically updates the K3s plan-index row;
5. runs all required repository and fake-cluster gates locally;
6. obtains local sudo authorization when required;
7. invokes one live `--mode apply` transaction;
8. requires `status=success` and `git_publication_allowed=true`;
9. stages exactly declared paths;
10. commits and optionally pushes a candidate branch and opens a draft PR.

No repository commit exists before live success. Qualification-only mode never stages, commits, pushes, or opens a PR.

## 14. Failure and recovery policy

### Before configuration publication

- no K3s rollback is needed;
- verified pre-change backup remains retained;
- temporary remote helper/staging is removed when reachable;
- no Git publication occurs.

### After configuration publication but before finalization

- restore previous managed paths;
- restart K3s if needed;
- verify readiness;
- restore timer files/state;
- retain pre-change backup and failure receipt;
- do not commit/push.

### Ambiguous or incomplete transaction

The next run refuses mutation and reports the transaction ID. The owner uses the explicit recovery mode:

```bash
bash tools/k3s/phase1-control-plane.sh \
  --mode recover \
  --server debian3 \
  --transaction <id> \
  --recovery-action rollback
```

`finalize` is used only when the prior success evidence is complete and rollback material should be intentionally discarded.

## 15. Acceptance criteria

Phase 1 is complete only when one live receipt proves:

- full qualification passed;
- outer launcher and runtime are separate and trusted;
- K3s service, `/readyz`, and Ready Node inventory passed;
- endpoint, CA, and TLS identity passed from workstation and every reachable registry host;
- pre-change encrypted backup passed offline restore materialization before convergence;
- post-change encrypted backup passed after convergence;
- server/agent token restoration contracts passed without exposing values;
- canonical files are deterministic and drift-locked;
- recurring timer is installed and active;
- no rollback is pending;
- `git_publication_allowed=true`;
- candidate Git commit contains only declared paths;
- branch/PR remains unmerged until receipt review.

## 16. References

- Rederivation report: `docs/reports/2026-08-11-k3s-phase1-rederivation.md`
- Backup/restore runbook: `docs/runbooks/k3s-control-plane-backup-restore.md`
- Agent instructions: `docs/runbooks/k3s-phase1-agent-instructions.md`
- Migration plan: `docs/plans/2026-08-10-k3s-migration-execution.md`
- Entrypoint: `tools/k3s/phase1-control-plane.sh`