# CI estate containment — design

audience: AI coding agents first. Implementer: codex (`-m gpt-5.6-terra -c model_reasoning_effort=medium`).
Owner intent: CI jobs on the buildboxes must never go red because a co-located agent
workload spiked the box ("slow box" reds). Overdeck owns the whole lifecycle, so the fix
lives here, not in each project's repo.

## Diagnosis this encodes (measured 2026-08-16)

debian1/2/3 are identical (8 cores, 31G). "debian2 is slow" was false — it was the
quietest box at measurement time. The reds come from two uncoordinated workload planes:
GitHub Actions runners (≥10 runner installs across the boxes, multiple projects) and
overdeck agent work (cdx offload, sandbox image builds, e2e). Runner jobs are pinned to
named hosts, so they cannot move when an agent workload lands on the same box; wall-clock
timeouts then read as flake. Half of multideal's runners were also offline, funnelling
everything to one box.

## Contract

Everything ships through `modules/buildbox/host-config/apply.sh` (already fleet-applied
via `deckctl fleet`) plus one new module file per item. Boxes: debian1/2/3.

1. **`ci.slice` on every box** — new systemd user slice unit installed by host-config:
   `CPUQuota=500%` (5 of 8 cores), `MemoryHigh=20G`, `MemoryMax=24G`, `IOWeight=50`,
   `CPUWeight=80`. NEVER a transient per-run cap (standing doctrine: transient
   MemoryHigh/Max caps have killed runs); this is one persistent slice bounding the
   aggregate.
2. **Runner services join the slice.** `apply.sh` already iterates
   `$HOME/actions-runner-*` (line ~39). Extend it: for each runner's systemd unit, drop in
   `Slice=ci.slice` (user-level drop-in, `systemctl --user daemon-reload`, restart only
   units whose drop-in changed). Runners not run as systemd units are reported by name,
   never silently skipped.
3. **Agent plane keeps its existing caps** (sandbox-run `--cpus/--memory`) — out of scope
   here except one gap: `sandbox-image-build` on a box runs uncontained. Wrap its podman
   build in `systemd-run --user --slice=ci.slice` on the box side (reuse the same slice:
   the point is one bounded aggregate for non-interactive load).
4. **Runner liveness surfaces as an incident.** `bb-status` already lists runner
   online/offline. Add an edge-triggered check to the existing box health poll seam (find
   it in `modules/buildbox/` or the systray health path): a runner transitioning
   online→offline fires ONE `od-requests fire` with signature
   `gh-runner:<runner-name>:offline` (dedup via the existing fire claim). No cron, no
   repeat nagging (standing rule: notifications are edge-triggered only).
5. **Idle-scheduling guard.** A conformance check script (wired into the overdeck gate
   chain, not orphaned) that greps every workflow file in repos overdeck deploys runners
   for — start with the repos named by the runner install dirs on the boxes — and fails
   naming file+line when a self-hosted job uses the cpu-limit wrapper without
   `BUILD_SCHED_IDLE=0`. Where the workflow lives outside overdeck, the check emits an
   advisory list instead of failing (we cannot gate another repo's tree from here).

## Explicit non-goals

- No shared-label re-registration of runners in this slice — that changes other repos'
  workflow `runs-on` values and needs a per-repo pass; record it as follow-up, don't do it.
- No k3s. Direction of record is podman + tmux (laptop-as-terminal plan); k3s Phase 3+
  is parked.
- No hand-rolled scheduler, no load-sensing dispatch for CI.

## Acceptance

- `deckctl fleet` apply on one box, then all three; `systemctl --user show <runner unit>
  -p Slice` reports `ci.slice` for every runner unit on every box.
- Slice pressure proof: with a synthetic 8-core burn inside `ci.slice` on one box, an
  interactive `ssh` command and a `cdx exec` probe (model `gpt-5.3-codex-spark` ONLY)
  still complete; the burn is throttled to ~5 cores (`systemd-cgtop` sample attached to
  the receipt).
- Kill one runner service → exactly one board fire appears with the runner's name;
  restart it → no second fire until the next offline transition.
- The conformance check fails RED on a synthetic workflow missing `BUILD_SCHED_IDLE=0`,
  passes GREEN after fixing it (both runs shown).
- All host-config tests: `bash modules/buildbox/test/*.test.sh` (or the module's runner)
  green.