# K3s control-plane backup and restore runbook Audience: owner and trusted recovery agents Applies to: Phase 1 qualification-first tooling Safety rule: verification and materialization are allowed; live restore activation is always a separate owner-approved incident operation ## 1. Artifacts created by a successful Phase 1 run Default workstation paths: ```text ~/.config/overdeck/k3s-backup.agekey ~/.config/overdeck/k3s-backup.json ~/.local/lib/overdeck/k3s/ ~/.local/state/overdeck/k3s-backups/debian3/ ~/.local/state/overdeck/k3s-phase1-debian3.lock ~/.config/systemd/user/overdeck-k3s-backup.service ~/.config/systemd/user/overdeck-k3s-backup.timer ``` A successful apply retains at least two verified pairs: ```text overdeck-k3s-debian3-prechange-.tar.age overdeck-k3s-debian3-prechange-.json overdeck-k3s-debian3-postchange-.tar.age overdeck-k3s-debian3-postchange-.json ``` The `.json` file is non-secret verification metadata. The `.tar.age` file contains encrypted datastore, token, configuration, and service recovery material. ## 2. Critical custody rule The recovery archive and age private key must not have their only durable copies on the same workstation. After Phase 1 succeeds: 1. copy the encrypted `.tar.age` files to an owner-controlled off-workstation location; 2. copy the age private key to a separate secure secret store or offline medium; 3. preserve mode `0600` when stored on a filesystem; 4. never upload the private key in a phase receipt; 5. record the external locations in the owner’s private operations inventory, not in Git. The encrypted backup is useful without secrecy only if the private key remains protected. The private key alone is not a backup. ## 3. Inspect recurring backup status ```bash systemctl --user status overdeck-k3s-backup.timer --no-pager systemctl --user list-timers overdeck-k3s-backup.timer --all systemctl --user status overdeck-k3s-backup.service --no-pager ``` Read recent logs: ```bash journalctl --user -u overdeck-k3s-backup.service -n 200 --no-pager ``` List verified pairs: ```bash find "$HOME/.local/state/overdeck/k3s-backups/debian3" \ -maxdepth 1 -type f \ \( -name '*.tar.age' -o -name '*.json' \) \ -printf '%TY-%Tm-%Td %TH:%TM:%TS %m %s %p\n' | sort ``` Do not delete unpaired or failure-qualified archives automatically. They may be the only evidence from an interrupted transaction. ## 4. Run an additional backup manually Use the installed runtime after Phase 1 succeeds: ```bash bash "$HOME/.local/lib/overdeck/k3s/phase1-control-plane.sh" \ --config "$HOME/.config/overdeck/k3s-backup.json" \ --mode backup \ --non-interactive ``` This operation: - uploads a fresh ephemeral helper; - verifies K3s health and identity; - creates a `scheduled` archive; - streams and encrypts it off-host; - materializes an isolated restore proof; - checks workstation and peer API access; - removes remote plaintext staging; - prunes only old verified archive/metadata pairs. ## 5. Verify an encrypted archive without live changes Choose an empty private directory: ```bash BACKUP="$HOME/.local/state/overdeck/k3s-backups/debian3/.tar.age" RESTORE_ROOT="$HOME/k3s-restore-proof-$(date -u +%Y%m%dT%H%M%SZ)" bash tools/k3s/restore-control-plane.sh \ --encrypted "$BACKUP" \ --identity "$HOME/.config/overdeck/k3s-backup.agekey" \ --restore-root "$RESTORE_ROOT" ``` The wrapper decrypts to a private temporary TAR, validates it, and materializes the restore tree. It does not stop K3s or copy anything into `/etc` or `/var/lib`. Expected top-level output: ```text $RESTORE_ROOT/ OVERDECK_RESTORE_PLAN.json etc/rancher/k3s/ var/lib/rancher/k3s/server/token var/lib/rancher/k3s/server/agent-token var/lib/rancher/k3s/server/db/... etc/systemd/system/... ``` On the current `debian3` layout, `agent-token` is expected to be reconstructed as the relative link: ```text agent-token -> token ``` The encrypted TAR itself never contains that symlink. ## 6. Interpret verification metadata A verified metadata file includes: - `status: verified`; - purpose (`prechange`, `postchange`, or `scheduled`); - encrypted file size and SHA-256; - server identity; - K3s version plus separate launcher/runtime facts; - datastore type and integrity result; - agent-token restoration classification; - canonical configuration presence/consistency; - `secret_values_recorded: false`. A pre-change backup may report: ```json { "canonical_config": { "present": false, "purpose": "prechange", "reason": "prechange-baseline" } } ``` That is expected when the archive predates Phase 1 publication. Post-change and scheduled backups must report `canonical_config.present: true`. ## 7. Qualification-only execution The package launcher supports a qualification-only mode. It is useful after a topology or workstation change: ```bash ./apply-overdeck-k3s-phase1-qualified.sh \ /home/user/Projects/overdeck \ --qualification-only ``` It may create an age identity and verified encrypted pre-change backup, but it must not: - publish K3s configuration; - restart K3s; - install the timer; - stage or commit repository files; - push or open a pull request. Return the generated receipt for review. ## 8. Unfinished configuration transaction A normal rerun refuses to mutate when the server reports an unfinished transaction. Find the transaction ID in: ```text /live/remote-inspect-error.json /live/server-inspect.json /live/phase1-result.json ``` Do not delete `/var/lib/overdeck/k3s-phase1/transactions` manually. ### Roll back Use when the prior apply did not complete or its evidence is incomplete: ```bash bash tools/k3s/phase1-control-plane.sh \ --mode recover \ --repo-root /home/user/Projects/overdeck \ --server debian3 \ --transaction \ --recovery-action rollback ``` A successful rollback restores prior managed files/absence and requires K3s readiness. ### Finalize Use only when the prior live result proves convergence, both recovery archives, timer installation, and peer checks succeeded, but the final response was lost: ```bash bash tools/k3s/phase1-control-plane.sh \ --mode recover \ --repo-root /home/user/Projects/overdeck \ --server debian3 \ --transaction \ --recovery-action finalize ``` Finalization discards rollback authority. Never use it merely to make a blocked rerun proceed. ## 9. Live restore activation — incident procedure The Phase 1 tools deliberately stop before live activation. A real restore is destructive and must be adapted to the incident, current K3s version, surviving control-plane state, and datastore type. Before activation: 1. preserve the failed live filesystem and logs; 2. verify the selected encrypted archive again; 3. compare server machine identity and intended target; 4. confirm exact K3s version and outer launcher digest; 5. confirm console or equivalent recovery access; 6. stop workload publication and prevent concurrent operators; 7. prepare rollback media for the current failed state; 8. choose SQLite filesystem restoration or embedded-etcd snapshot restoration explicitly. ### SQLite outline With K3s stopped and after preserving the current DB/token/config: - restore the isolated `/etc/rancher/k3s` tree; - restore server/agent token layout; - replace the K3s DB tree from the verified restore root; - apply root ownership and restrictive modes; - start K3s; - prove `/readyz`, Nodes, CA, serving SANs, workloads, and peer access. ### Embedded-etcd outline With a verified snapshot and matching token/version: - follow the current K3s cluster-reset snapshot-restore procedure; - use the snapshot copied into the isolated restore tree; - restore the matching server token and configuration; - verify cluster identity before admitting agents/workloads. Do not derive a live command blindly from this outline. The incident agent must inspect the installed K3s version’s official recovery procedure and produce an owner-reviewed command sequence. ## 10. Backup failure handling If an encrypted archive has a failure suffix such as: ```text .unverified-phase-failed.tar.age .verified-phase-failed.tar.age ``` retain it with its metadata and receipt. Do not treat it as the newest valid recovery point unless `verify-backup.py` succeeds independently. The server-side plaintext archive should be absent after cleanup. A retained remote path under `/var/lib/overdeck/k3s-phase1/backups` indicates an interrupted cleanup. Remove it only through a reviewed helper transaction or after confirming its transaction ID and corresponding off-host archive. ## 11. Upgrade boundary `upgrade-control-plane.sh` is plan-only. It rejects apply requests. A future K3s upgrade must: - use an explicit version; - verify the outer launcher candidate SHA-256; - retain pre-upgrade backup and rollback capability; - preserve datastore/token compatibility; - update launcher and runtime lock fields only after post-upgrade proof. Phase 1 never upgrades K3s to repair discovery, readiness, or backup failures. ## 12. Security notes Never include in tickets, chat, or Git: - age private-key contents; - server or agent token contents; - kubeconfig contents; - decrypted datastore files; - raw `k3s.service` environment values; - unredacted secret-bearing process arguments. Safe evidence includes: - SHA-256 of encrypted archives; - public age recipient; - public CA hash; - executable digests; - file metadata without content hashes for token files; - transaction IDs and redacted structured receipts.