# K3s control-plane backup and restore runbook

Audience: owner and trusted recovery agents
Applies to: Phase 1 qualification-first tooling
Safety rule: verification and materialization are allowed; live restore activation is always a separate owner-approved incident operation

## 1. Artifacts created by a successful Phase 1 run

Default workstation paths:

```text
~/.config/overdeck/k3s-backup.agekey
~/.config/overdeck/k3s-backup.json
~/.local/lib/overdeck/k3s/
~/.local/state/overdeck/k3s-backups/debian3/
~/.local/state/overdeck/k3s-phase1-debian3.lock
~/.config/systemd/user/overdeck-k3s-backup.service
~/.config/systemd/user/overdeck-k3s-backup.timer
```

A successful apply retains at least two verified pairs:

```text
overdeck-k3s-debian3-prechange-<stamp>.tar.age
overdeck-k3s-debian3-prechange-<stamp>.json
overdeck-k3s-debian3-postchange-<stamp>.tar.age
overdeck-k3s-debian3-postchange-<stamp>.json
```

The `.json` file is non-secret verification metadata. The `.tar.age` file contains encrypted datastore, token, configuration, and service recovery material.

## 2. Critical custody rule

The recovery archive and age private key must not have their only durable copies on the same workstation.

After Phase 1 succeeds:

1. copy the encrypted `.tar.age` files to an owner-controlled off-workstation location;
2. copy the age private key to a separate secure secret store or offline medium;
3. preserve mode `0600` when stored on a filesystem;
4. never upload the private key in a phase receipt;
5. record the external locations in the owner’s private operations inventory, not in Git.

The encrypted backup is useful without secrecy only if the private key remains protected. The private key alone is not a backup.

## 3. Inspect recurring backup status

```bash
systemctl --user status overdeck-k3s-backup.timer --no-pager
systemctl --user list-timers overdeck-k3s-backup.timer --all
systemctl --user status overdeck-k3s-backup.service --no-pager
```

Read recent logs:

```bash
journalctl --user -u overdeck-k3s-backup.service -n 200 --no-pager
```

List verified pairs:

```bash
find "$HOME/.local/state/overdeck/k3s-backups/debian3" \
  -maxdepth 1 -type f \
  \( -name '*.tar.age' -o -name '*.json' \) \
  -printf '%TY-%Tm-%Td %TH:%TM:%TS %m %s %p\n' | sort
```

Do not delete unpaired or failure-qualified archives automatically. They may be the only evidence from an interrupted transaction.

## 4. Run an additional backup manually

Use the installed runtime after Phase 1 succeeds:

```bash
bash "$HOME/.local/lib/overdeck/k3s/phase1-control-plane.sh" \
  --config "$HOME/.config/overdeck/k3s-backup.json" \
  --mode backup \
  --non-interactive
```

This operation:

- uploads a fresh ephemeral helper;
- verifies K3s health and identity;
- creates a `scheduled` archive;
- streams and encrypts it off-host;
- materializes an isolated restore proof;
- checks workstation and peer API access;
- removes remote plaintext staging;
- prunes only old verified archive/metadata pairs.

## 5. Verify an encrypted archive without live changes

Choose an empty private directory:

```bash
BACKUP="$HOME/.local/state/overdeck/k3s-backups/debian3/<archive>.tar.age"
RESTORE_ROOT="$HOME/k3s-restore-proof-$(date -u +%Y%m%dT%H%M%SZ)"

bash tools/k3s/restore-control-plane.sh \
  --encrypted "$BACKUP" \
  --identity "$HOME/.config/overdeck/k3s-backup.agekey" \
  --restore-root "$RESTORE_ROOT"
```

The wrapper decrypts to a private temporary TAR, validates it, and materializes the restore tree. It does not stop K3s or copy anything into `/etc` or `/var/lib`.

Expected top-level output:

```text
$RESTORE_ROOT/
  OVERDECK_RESTORE_PLAN.json
  etc/rancher/k3s/
  var/lib/rancher/k3s/server/token
  var/lib/rancher/k3s/server/agent-token
  var/lib/rancher/k3s/server/db/...
  etc/systemd/system/...
```

On the current `debian3` layout, `agent-token` is expected to be reconstructed as the relative link:

```text
agent-token -> token
```

The encrypted TAR itself never contains that symlink.

## 6. Interpret verification metadata

A verified metadata file includes:

- `status: verified`;
- purpose (`prechange`, `postchange`, or `scheduled`);
- encrypted file size and SHA-256;
- server identity;
- K3s version plus separate launcher/runtime facts;
- datastore type and integrity result;
- agent-token restoration classification;
- canonical configuration presence/consistency;
- `secret_values_recorded: false`.

A pre-change backup may report:

```json
{
  "canonical_config": {
    "present": false,
    "purpose": "prechange",
    "reason": "prechange-baseline"
  }
}
```

That is expected when the archive predates Phase 1 publication.

Post-change and scheduled backups must report `canonical_config.present: true`.

## 7. Qualification-only execution

The package launcher supports a qualification-only mode. It is useful after a topology or workstation change:

```bash
./apply-overdeck-k3s-phase1-qualified.sh \
  /home/user/Projects/overdeck \
  --qualification-only
```

It may create an age identity and verified encrypted pre-change backup, but it must not:

- publish K3s configuration;
- restart K3s;
- install the timer;
- stage or commit repository files;
- push or open a pull request.

Return the generated receipt for review.

## 8. Unfinished configuration transaction

A normal rerun refuses to mutate when the server reports an unfinished transaction.

Find the transaction ID in:

```text
<phase-result>/live/remote-inspect-error.json
<phase-result>/live/server-inspect.json
<phase-result>/live/phase1-result.json
```

Do not delete `/var/lib/overdeck/k3s-phase1/transactions` manually.

### Roll back

Use when the prior apply did not complete or its evidence is incomplete:

```bash
bash tools/k3s/phase1-control-plane.sh \
  --mode recover \
  --repo-root /home/user/Projects/overdeck \
  --server debian3 \
  --transaction <transaction-id> \
  --recovery-action rollback
```

A successful rollback restores prior managed files/absence and requires K3s readiness.

### Finalize

Use only when the prior live result proves convergence, both recovery archives, timer installation, and peer checks succeeded, but the final response was lost:

```bash
bash tools/k3s/phase1-control-plane.sh \
  --mode recover \
  --repo-root /home/user/Projects/overdeck \
  --server debian3 \
  --transaction <transaction-id> \
  --recovery-action finalize
```

Finalization discards rollback authority. Never use it merely to make a blocked rerun proceed.

## 9. Live restore activation — incident procedure

The Phase 1 tools deliberately stop before live activation. A real restore is destructive and must be adapted to the incident, current K3s version, surviving control-plane state, and datastore type.

Before activation:

1. preserve the failed live filesystem and logs;
2. verify the selected encrypted archive again;
3. compare server machine identity and intended target;
4. confirm exact K3s version and outer launcher digest;
5. confirm console or equivalent recovery access;
6. stop workload publication and prevent concurrent operators;
7. prepare rollback media for the current failed state;
8. choose SQLite filesystem restoration or embedded-etcd snapshot restoration explicitly.

### SQLite outline

With K3s stopped and after preserving the current DB/token/config:

- restore the isolated `/etc/rancher/k3s` tree;
- restore server/agent token layout;
- replace the K3s DB tree from the verified restore root;
- apply root ownership and restrictive modes;
- start K3s;
- prove `/readyz`, Nodes, CA, serving SANs, workloads, and peer access.

### Embedded-etcd outline

With a verified snapshot and matching token/version:

- follow the current K3s cluster-reset snapshot-restore procedure;
- use the snapshot copied into the isolated restore tree;
- restore the matching server token and configuration;
- verify cluster identity before admitting agents/workloads.

Do not derive a live command blindly from this outline. The incident agent must inspect the installed K3s version’s official recovery procedure and produce an owner-reviewed command sequence.

## 10. Backup failure handling

If an encrypted archive has a failure suffix such as:

```text
.unverified-phase-failed.tar.age
.verified-phase-failed.tar.age
```

retain it with its metadata and receipt. Do not treat it as the newest valid recovery point unless `verify-backup.py` succeeds independently.

The server-side plaintext archive should be absent after cleanup. A retained remote path under `/var/lib/overdeck/k3s-phase1/backups` indicates an interrupted cleanup. Remove it only through a reviewed helper transaction or after confirming its transaction ID and corresponding off-host archive.

## 11. Upgrade boundary

`upgrade-control-plane.sh` is plan-only. It rejects apply requests. A future K3s upgrade must:

- use an explicit version;
- verify the outer launcher candidate SHA-256;
- retain pre-upgrade backup and rollback capability;
- preserve datastore/token compatibility;
- update launcher and runtime lock fields only after post-upgrade proof.

Phase 1 never upgrades K3s to repair discovery, readiness, or backup failures.

## 12. Security notes

Never include in tickets, chat, or Git:

- age private-key contents;
- server or agent token contents;
- kubeconfig contents;
- decrypted datastore files;
- raw `k3s.service` environment values;
- unredacted secret-bearing process arguments.

Safe evidence includes:

- SHA-256 of encrypted archives;
- public age recipient;
- public CA hash;
- executable digests;
- file metadata without content hashes for token files;
- transaction IDs and redacted structured receipts.
