# K3s Phase 1 rederivation after three safe live stops

Date: 2026-08-11
Audience: owner, migration agents, reviewers
Status: replacement implementation complete; qualification-first live execution pending

## Outcome

Phase 1 was rederived rather than patched a fourth time. The replacement treats the three owner runs as live compatibility evidence and changes the transaction boundary accordingly:

1. qualify the complete apply path;
2. create, encrypt, verify, and materialize a pre-change recovery archive;
3. only then publish control-plane configuration;
4. verify the converged control plane and create a second recovery archive;
5. install the recurring backup timer;
6. only after the live transaction succeeds may the package launcher commit, push, or open a pull request.

The previous Phase 1 candidates remain unmerged and are not inputs to this implementation.

## Evidence from the three runs

### Run 1 — shared plan index conflict

The first package was generated from an older `main` and attempted to patch `docs/plans/INDEX.md` textually. Newer coordination changes on `main` caused a conflict before tests or live actions.

Correction retained in the replacement:

- stable files use a three-way Git patch;
- the K3s row in `docs/plans/INDEX.md` is updated semantically;
- unrelated rows, receipts, status changes, formatting, and line endings are preserved;
- a missing, duplicate, malformed, symlinked, or non-UTF-8 target fails closed.

### Run 2 — privileged `PATH` assumption

The second package passed repository gates but the remote root helper searched for `k3s` only through the non-interactive sudo environment. `/usr/local/bin` was not visible there.

Correction retained in the replacement:

- child commands use a deterministic root-controlled executable path;
- the outer K3s launcher is discovered from structured systemd `ExecStart`, canonical installer paths, and the approved path;
- every privileged executable and its directory chain are verified;
- no missing executable is repaired by downloading, reinstalling, or upgrading K3s.

### Run 3 — launcher/runtime conflation

The third package found both:

- outer command launcher: `/usr/local/bin/k3s`;
- running executable: `/var/lib/rancher/k3s/data/<digest>/bin/k3s`.

It selected the running executable and called:

```text
<runtime-app>/k3s kubectl get --raw=/readyz
```

That applet rejected nested `kubectl` dispatch. The resulting message claimed K3s was not ready even though `k3s.service` was active.

Correction in the replacement:

- launcher and runtime are separate typed inventory objects;
- only the outer launcher is used for `kubectl`, `etcd-snapshot`, and `--version`;
- the running executable is evidence only;
- readiness failures distinguish invalid launcher dispatch, API reachability, authentication, and service state;
- server inspection retries readiness for a bounded 30-second window instead of a single five-second attempt.

## Additional defect found before another live run

The latest receipt reported:

```text
/var/lib/rancher/k3s/server/agent-token
  -> /var/lib/rancher/k3s/server/token
```

The prior backup implementation rejected every symlink and would therefore have failed after the readiness defect was fixed.

The replacement classifies agent-token layout explicitly:

- absent;
- independent root-owned mode-restricted regular file;
- symlink resolving exactly to the canonical server token.

A supported symlink is not archived as a TAR link. The manifest records a constrained restoration descriptor, and the offline restore proof reconstructs only `agent-token -> token` inside the isolated restore tree. Broken, cyclic, escaping, or unrelated symlinks fail closed.

## Version-lock correction

The old lock associated one digest with both the downloaded K3s release and the extracted runtime executable. Those are not guaranteed to be the same artifact.

The replacement schema records:

```json
{
  "version": "v1.36.3+k3s1",
  "launcher": {
    "invocation_path": "/usr/local/bin/k3s",
    "resolved_path": "/usr/local/bin/k3s",
    "sha256": "...",
    "release_url": "..."
  },
  "runtime": {
    "present": true,
    "resolved_path": "/var/lib/rancher/k3s/data/.../bin/k3s",
    "sha256": "..."
  }
}
```

Only the launcher digest is tied to the release URL and future upgrade candidate. A legacy ambiguous lock is rejected rather than silently migrated.

## Qualification boundary

The replacement qualification matrix runs these checks before control-plane configuration can change:

1. `age` tools and private identity are available with safe ownership and permissions.
2. The recurring user-systemd backup installation is preflighted in a temporary directory.
3. The server is reached through the registered recovery door.
4. The ephemeral root helper is checksum verified.
5. The outer launcher and running executable are inventoried separately.
6. `k3s.service`, `/readyz`, and non-empty Ready Node inventory pass.
7. The datastore is classified as SQLite or embedded etcd.
8. Server-token and agent-token layouts are supported.
9. The desired endpoint, SAN set, configuration, contract, and version lock are planned without publication.
10. Workstation kubeconfig `/readyz` passes.
11. Every registry-reachable host proves the same API CA and hostname/IP-verified TLS.
12. A pre-change datastore/config/token archive is created on the server.
13. The TAR is streamed directly into `age` off-host with plaintext size and SHA-256 comparison.
14. The encrypted archive is decrypted temporarily, verified file by file, and materialized into an isolated restore tree.
15. Remote plaintext backup staging is removed.

Independent checks continue after a failure so the receipt reports all known failures and blocked dependencies instead of stopping at the first defect.

Qualification may create the workstation age identity and a verified encrypted pre-change backup. It does not publish K3s configuration, restart K3s, install the recurring timer, or publish Git changes.

## Apply boundary

Only a passing qualification can enter apply:

1. publish the three managed control-plane files transactionally;
2. restart K3s only if the deterministic YAML drop-in changed;
3. prove server readiness, Ready Node inventory, launcher/runtime digests, and serving-certificate SANs;
4. prove workstation and peer API/CA/TLS access again;
5. create and verify a post-change encrypted recovery archive;
6. install and verify the recurring backup timer;
7. finalize the server transaction and remove rollback authority;
8. retain both verified encrypted backups and their metadata.

A failure before finalization restores the prior managed files or prior absence, restarts K3s when required, verifies readiness, rolls back timer files, retains recovery evidence, and prevents Git publication.

## Git publication boundary

The package launcher now applies repository files and runs tests in an isolated worktree but does not commit before live success.

The exact order is:

```text
package verification
  -> repository patch
  -> deterministic tests
  -> local sudo qualification
  -> live qualification + pre-change backup
  -> live apply + post-change backup
  -> inspect phase1-result.json
  -> stage declared paths
  -> commit
  -> push candidate branch
  -> draft pull request
```

The orchestrator must return both:

```json
{
  "status": "success",
  "git_publication_allowed": true
}
```

before the launcher stages or commits any repository path. Qualification-only mode never commits or pushes.

## Test rederivation

The replacement test suite includes:

- 55 Python unit and failure-injection tests;
- a sanitized fixture derived from the real `debian3` receipt;
- distinct outer-launcher and extracted-runtime behavior;
- the runtime-app nested-`kubectl` failure that stopped run 3;
- agent-token absent, regular, safe symlink, unrelated symlink, broken symlink, and unsafe permissions;
- non-empty Node inventory and explicit probe-failure classification;
- pre-change backup without not-yet-published canonical files;
- post-change backup requiring a complete canonical set;
- separate launcher/runtime version-lock fields;
- safe TAR extraction and exact manifest file-set validation;
- SQLite integrity verification;
- isolated restore materialization and agent-token symlink reconstruction;
- a full fake-cluster transaction proving:
  - qualification and pre-change backup precede convergence;
  - post-change backup follows convergence;
  - timer installation follows post-change proof;
  - finalization is last;
  - Git publication is authorized only on success.

The fake-cluster fixture now models K3s v1.36.3+k3s1, `/usr/local/bin/k3s`, an extracted runtime path, SQLite, three Ready Nodes, and `agent-token -> token`.

## Safety status of prior candidates

The three reported owner runs stopped before persistent K3s mutation. Their isolated worktrees and candidate commits are evidence only. They must not be merged or used as the base for the replacement.

The replacement package is a complete delta from the merged Phase 0 base and can be applied to a newer `origin/main` through three-way patching plus the semantic plan-index update.

## UI boundary

This phase changes no product UI, no design-system code, and no UI primitive. Astryx is not required. The owner rule against creating a primitive without permission is unaffected.