# Remote reliability ladder — six slices to a proven stack

outcome: remote execution failures name themselves, transport survives quiet channels, a standing canary guards the path, gates run contained with pinned toolchains — then k3s is pure orchestration on a proven stack.
status: ACTIVE — slices distributed across lanes 2026-08-16; slice 2 dispatched to codex (gpt-5.6-terra/low, owner-directed launcher).
source request: owner 2026-08-16 /brainstorm, verbatim ladder. Ordering principle (owner, pinned): "diagnosability and transport before capacity, because every hour spent on a silent-failure layer is paid back tenfold once failures name themselves." Each slice ends in evidence the owner can SEE; each proven before the next starts.

## Slices

- [ ] **S1 — Keepalives** (IN FLIGHT, lane running): discriminating experiment — silent 400s command per remote path dies without keepalives, survives with them; one shared ssh-args helper adopted by every seam (remote-build.mjs, pi_remote_dispatch, cdx offload, e2e-remote). Evidence: quiet-channel probe green on all three boxes; config key visible in build-remote.json. First because this class killed every run on 2026-08-15/16 and travels to k3s unchanged.
- [ ] **S2 — Standing canary** (dispatched to codex terra/low): a trivial gate run through the full remote path a few times daily, reporting to the requests board with the loud-failure envelope naming any death. Evidence: a canary row on the board with consecutive green runs and timestamps. Converts future transport regressions from "my work died mysteriously" into "the canary failed with a named cause"; continuously verifies S3–S5 for free.
- [ ] **S3 — One green factory run** (acceptance already assigned to the keepalive lane): full adw_plan_build_test_quality surviving build + remote gate. Evidence: the adw id green on /factory. After the canary by design: canary proves transport, this proves the whole stack.
- [ ] **S4 — Fleet pinning**: pin the boxes' package-manager versions (the mise drift that broke debian3), enforced by remote-doctor going RED on drift. Evidence: `local-gate --remote-doctor` showing a version-parity check green on all three boxes. The second travels-to-k3s class.
- [ ] **S5 — Podman P1 flip** (P1 built, ticket in queue — docs/plans/2026-08-16-podman-default.md): flip after one real ticket's gate log names the container executor while build.slice shows no gate processes. Evidence: that log line + empty build.slice. Kills the contention class permanently.
- [ ] **S6 — Tooling manifests for the gate path** (design: docs/specs/2026-08-15-split-config-rendering-design.md per-project tooling section): per-project pinned toolchains consumed by the container/remote gate — binary resolution stops depending on PATH luck (the astro class, fixed structurally). Evidence: a gate log showing tools resolved from the manifest; an undeclared project refused visibly.

Then k3s: scheduling, placement, restart-on-death only. The podman plan's interim-scheduler retirement clause applies.

## Cross-references

S1/S3: keepalive lane (running). S2: this dispatch. S5: podman-default plan P1 (ticket 7d005685). S6: split-config spec. Canary + notifications: the factory notify contract (one start/finish/crash) and edge-triggered doctrine apply to canary reporting — a green canary is SILENT on the channel (board row only); only a red canary notifies, once per crossing.

current receipt: ladder registered 2026-08-16; S1 lane running, S2 dispatched, S5 P1 queued.
next executable action: S2 lands → S1 evidence → S3 button-press.