# I7a Machine Onboarding Runbook

**Status:** implementation runbook for the owner-authorized I7a worker enrollment lane
**Cluster control-plane:** `debian3` / `https://100.101.104.41:6443`
**Current K3s version:** `v1.36.3+k3s1` (discovered at deployment/runtime; not hard-coded by product logic)
**Enrollment SSH key fingerprint:** `SHA256:BwI3jmoAnAvw7/h40iK0a1HEP93mFkFVBsZNFYYbBVE`
**Runtime activation:** READY — landed and deployed. Use `https://debian3.taild2daa0.ts.net/cluster`.

## Before the onboarding day

The persistent enrollment identity is already provisioned on `debian3` at:

- `~/.config/awp-dogfood/machine-enrollment-ssh-key` — private, mode 600; never copy into Git, logs, chat, or PostgreSQL.
- `~/.config/awp-dogfood/machine-enrollment-ssh-key.pub` — public key to authorize on a new worker.
- `~/.config/awp-dogfood/k3s-node-token` — cluster join credential, mode 600; projected only by the trusted enrollment boundary.
- `~/.config/awp-dogfood/machine-known-hosts` — dedicated SSH host-key trust file.

The public key may be displayed safely with:

```sh
ssh debian3 'cat ~/.config/awp-dogfood/machine-enrollment-ssh-key.pub'
```

Verify its fingerprint before authorizing it on a target:

```sh
ssh debian3 'ssh-keygen -lf ~/.config/awp-dogfood/machine-enrollment-ssh-key.pub'
```

Expected fingerprint: `SHA256:BwI3jmoAnAvw7/h40iK0a1HEP93mFkFVBsZNFYYbBVE`.

## Target prerequisites

I7a intentionally supports a narrow worker profile. The target must satisfy all of these before AWP will mutate it:

1. Debian or Ubuntu.
2. `x86_64`, `aarch64`, or `arm64`.
3. At least 2 CPU cores.
4. At least 2 GiB RAM.
5. At least 10 GiB free space on `/`.
6. `systemd` present.
7. `curl` present.
8. Tailscale connected, with `tailscale0` present and a stable Tailscale IPv4 address.
9. The target can reach `https://100.101.104.41:6443/cacerts`.
10. Normal OpenSSH is reachable on the dogfood enrollment port **2222**. Port 22 may remain owned by Tailscale SSH; AWP deliberately bypasses it.
11. The chosen SSH user can log in on port 2222 using the AWP enrollment public key.
12. The chosen SSH user has passwordless sudo (`sudo -n true`).
13. The target either has no K3s installation or is already an agent of this exact AWP cluster.

I7a deliberately refuses mutation for a K3s server, an agent belonging to another cluster, an ambiguous/partial K3s installation, an unsupported host, or an unreachable host.

## Authorize the target

On the new target, add the AWP enrollment **public** key to the chosen user's `~/.ssh/authorized_keys` and enforce normal SSH permissions:

```sh
umask 077
mkdir -p ~/.ssh
chmod 700 ~/.ssh
# Append the single public-key line obtained from debian3.
chmod 600 ~/.ssh/authorized_keys
```

Do not copy the private key to the target.

Before opening AWP, verify from the target:

```sh
sudo -n true
curl -kfsS --connect-timeout 4 --max-time 6 https://100.101.104.41:6443/cacerts >/dev/null
tailscale ip -4
```

## AWP enrollment flow

1. Open AWP and choose **Cluster** in the Operations navigation (`/cluster`).
2. In **Add Machine**, enter:
   - a stable machine name (normally the host name);
   - the Tailscale IPv4 address or a stable SSH-resolvable name;
   - the SSH user authorized above;
   - optional `key=value` labels separated by commas.
3. Select **Run preflight**. This step is read-only.
4. Review the durable preflight result. For a fresh machine it must be `clean`. For a node already in this exact cluster it may be `already-enrolled`.
5. If the result is `clean`, select **Enroll worker**. This is the explicit mutation boundary.
6. AWP rechecks state, installs a K3s **agent** pinned to the running server K3s version, sets the node IP to the Tailscale IPv4 address, and uses `tailscale0` for the flannel interface.
7. For `already-enrolled`, AWP does not reinstall K3s; **Adopt and verify worker** proceeds directly to Kubernetes verification.
8. AWP waits for the Kubernetes Node to register and become `Ready`.
9. AWP persists Machine capabilities (architecture, OS, runtime/Kubelet versions, capacity/allocatable values and AWP labels) and marks the Machine `ready`.

## Result verification

From `debian3`:

```sh
k3s kubectl get nodes -o wide
```

The new node must be `Ready`, have the expected Tailscale internal IP, and appear as a worker (not control-plane). The AWP Cluster page must show the Machine as `ready`.

## Recovery behavior

- **unreachable / unsupported:** fix the reported prerequisite and run preflight again. No K3s mutation was attempted.
- **foreign:** do not force enrollment. AWP refuses mutation because the node belongs to another cluster or runs a K3s server. Decide ownership outside I7a.
- **partial:** inspect the existing K3s installation manually. I7a does not automatically uninstall or overwrite ambiguous state.
- **install/join failure:** the enrollment becomes `failed`. Run preflight again. If the prior attempt actually joined successfully, state reconciles to `already-enrolled`; otherwise the next preflight identifies the remaining clean/partial condition.
- **node registered but not Ready:** the enrollment remains failed/degraded with the Kubernetes verification error. Diagnose node/network state, then rerun preflight before applying again.

The operation ID makes request creation idempotent. Repeated application after `completed` is a no-op, and every retry reconciles observed host state before mutation.

## Manual fallback

The existing operator-managed K3s worker onboarding remains a supported fallback after I7a landing and deployment until a separate owner-approved retirement decision removes it. I7a readiness by itself does not authorize removing or weakening that fallback. Never place the K3s node token in documentation or command history when using the manual path.

## Live evidence already recorded

`docs/progress/2026-08-24-i7a-live-preflight-evidence.json` records a read-only observation of existing worker `debian1`. It proves the target profile, cluster API reachability, same-cluster agent detection and exact K3s version without capturing secret material or invoking K3s mutation.