# Fleet configuration management — one engine, one inventory, drift as an alert

outcome: every managed file on the laptop and the three buildboxes converges from ONE git source through ONE engine; direct edits to managed paths are refused; drift is a dashboard alert, never archaeology.
status: ACTIVE — owner ruled go via relayed dispatch 2026-08-15 (see source request); S1 delivered same day with a real per-role apply gate, not just check-mode.
source request: owner 2026-08-15 — "the trap is that we have config drift constantly... we are using some weird patchy system that is not industry standard compliant at all... what is the industry standard for a solution to a problem like ours? What are we doing different? and how would you recommend for our sized situation to proceed?"

**Owner re-scope ruling, same day, relayed mid-S1 (verbatim intent, recorded here since it is now
standing policy the systemd role's lint enforces, not just a one-off instruction):** the two-week
observation window is cancelled as conservatism. The per-role reviewed diff is the safety floor and
is non-negotiable. New contract: collapse S1+S2+S3 into one gated pass, per role — write role → run
`--check --diff` → an agent reads the whole diff → if it matches expectations, apply immediately.
Role order, owner-fixed, lowest risk first: (1) PATH bins, (2) settings templates, (3) systemd units
last. The diff-read must catch: (a) a wrong source applied fleet-wide is drift weaponized; (b)
fold-forward — a diff that would revert a live hand-fix is an automatic STOP-and-adopt, source
absorbs the live value, never the reverse; (c) headless boxes have one door (sshd:2222) — ssh/sshd
stays permanently out of scope, and any first apply to a buildbox needs an armed deadman (backup +
scheduled revert, cancelled after the box is confirmed healthy).

**CPU/memory/PIDs doctrine (owner doctrine, born from 2026-08-15's incidents), enforced by the
systemd role's `policy_lint.py`:** agent-work slices carry `CPUWeight` only — `CPUQuota` on any
agent slice is forbidden (a hard quota starved every agent shell under contention, nr_throttled=
2.88M, ~300h stalled). `MemoryHigh`/`MemoryMax` on `agent.slice` stay forbidden (froze the box once,
8.7M reclaim events); memory pressure stays stall-based (`ManagedOOMMemoryPressure`). `pids.max`
(`TasksMax=` in systemd) is required on `agent.slice`; forkbombs are handled by pids-guard's
victim-share floor (>=10%, commit `0fe056c2d`), not by the slice. The declared normal is
`CPUWeight=50` (per the `50-no-starvation.conf` drop-in that predated this fix); any live
`CPUWeight=10` seen during incident response was a temporary fire-response override, not the
correct steady state.

## Diagnosis (from 2026-08-15's incidents, all in one day)

- SEVEN partial convergence mechanisms, each covering a slice with its own semantics:
  per-dispatch `seat-creds.mjs`, deploy-time `install-buildbox-parity.sh`,
  `buildbox-parity.timer`, `buildbox bootstrap` (devtools.json), `deckctl sync apply`,
  the `~/.claude` symlink farm, `adopt-owner-edits`.
- No inventory: 13 config homes discovered by sweep, 5 previously unknown; a seat
  template silently armed a fleet-wide plugin; retention had to be hand-set in 13 places.
- Two-writers conflict on box `~/.claude` is a recorded standing defect.
- Live config is agent-writable everywhere; 50 agents = 50 hands editing prod files.
- Config, runtime state, and caches live mixed in the same trees.

The disease has a name — unmanaged configuration — and a 20-year-old cure: declared
desired state + continuous convergence + single ownership. Drift is eliminated as a
CATEGORY, not cleaned as a chore.

## Decision (recommended)

**Ansible, pull mode.** Agentless, YAML, SSH — the boring industry standard at 4-machine
scale. Rejected for now: Nix/home-manager (drift-immune by construction, but a heavy
migration that fights the fast-moving bespoke tooling; remains the candidate end-state,
this plan does not foreclose it). Rejected: chezmoi (dotfiles-only scope; estate includes
systemd units, PATH tools, podman, root units).

Ground rules the engine enforces:
- One source of truth: `fleet/` in this repo — inventory (laptop + debian1/2/3) + roles.
- One owner per managed file: exactly one role writes it. Two-writers dies by construction.
- Three-way split declared per path: CONFIG (managed), RUNTIME STATE (never touched),
  CACHE (rebuildable, never managed).
- KEEP `seat-creds.mjs` — per-dispatch credential fast-path is sound and out of scope.

## Slices

**S1 — inventory + check-mode + gated per-role apply, collapsed with what was S2/S3's
apply mechanic (owner re-scope, same day — see ruling above).** `fleet/` with all four
hosts and roles transcribing what parity/bootstrap/deckctl already converge. Delivered
2026-08-15: `path_bins` role applied for real on the laptop (0 drift, verified empty
post-apply diff); `settings_template` and `systemd_units` built and check-mode verified
against all reachable hosts but NOT applied — both hit a named fold-forward stop (see
Receipt). Old mechanisms untouched. Evidence: `fleet/reports/drift-20260815-124720.md` +
per-role raw diffs, naming real differences on real boxes.
**S2 — drift on the board.** Check-mode timer on all four machines; results surface in
Overdeck in owner language. NOT built this pass (systemd role is check-only pending the
deadman + laptop-session-safety call — see Receipt); the timer unit itself is design-ready
(`fleet/check.sh` is the entrypoint it would call) but not installed or enabled.
**S3 — converge.** Per-role apply now exists (`fleet/apply.sh`, gated fail-closed on a
fresh read diff) rather than waiting for a separate slice; remaining work is finishing the
two deferred roles (buildbox-correct `path_bins`, reconciled `settings_template`) and
building the buildbox deadman, then applying them the same way `path_bins`/laptop was
applied this pass. Retire `install-buildbox-parity.sh`, `buildbox-parity.timer`, and
bootstrap's file-install half only once their managed paths are provably covered here.
**S4 — managed paths refuse hand edits.** Extend the existing deny-gate/shim registry:
agent writes to managed paths are refused with "change fleet/<role>, land it, it
converges." Owner's own hands are never gated. Evidence: an agent edit attempt on a
managed file returns the refusal; the fleet change lands and converges instead.
**S5 — absorb the remainder.** deckctl sync file-pushes and bootstrap vendor installs
folded into roles or explicitly declared out-of-scope with reasons. Evidence: the
mechanism list in this plan's Diagnosis section is each marked retired / kept / absorbed.

## Constraints

- Never converge credentials/tokens through the new engine (seat-creds owns that plane;
  od-auth Rule 0 stands).
- Buildboxes are headless, sshd:2222 only — a bad convergence must not sever access:
  ssh/sshd config stays OUT of managed scope until last, and every run keeps a deadman.
- Check mode precedes apply mode on every new role, always.
- Transcripts and session state are RUNTIME STATE — the engine never touches them
  (transcript unification is its own landed system).

## Receipt

2026-08-15 (morning): recommendation delivered with same-day incident evidence; plan
registered PROPOSED.

2026-08-15 (afternoon), S1 delivered in `wt/fleet-s1`:
- `fleet/inventory.yml` (regenerated from `~/.claude/buildbox-hosts.json` by
  `fleet/gen-inventory.sh`, never hand-edited), `fleet/site.yml`, `fleet/check.sh`
  (check-mode only, fail-closed against explicit `--check`/`-C`), `fleet/apply.sh`
  (one role at a time, gated on a `<4h`-old check-diff artifact in `fleet/reports/`).
- Three roles: `path_bins`, `settings_template`, `systemd_units` (+ `policy_lint.py`
  enforcing the CPU/memory/PIDs doctrine against source, currently clean).
- Real check-mode run against laptop + debian1 + debian2 + debian3 (all reachable):
  `path_bins` — laptop 0 drift; buildboxes: `buildbox-scratch-bind`/`cdx` absent, `pi` →
  mise shim not the workstation formula, `~/.claude/{bin,lib,hooks}` are real directories
  not symlinks (a different, not-yet-modelled buildbox install mechanism — role is
  read-only there this pass). `settings_template` — 1 changed on all 4 hosts; diff is
  large and goes well beyond the three named 2026-08-15 fixes (retention, security-
  guidance, its env flag — all three now correctly reflected in the repo template after
  cherry-picking commit `082b4a478` into this branch); the remainder is real
  pre-existing live customization (different `model`, several hook entries) that a blind
  full-file copy would silently revert — **not applied anywhere**, named fold-forward
  stop. `systemd_units` — laptop 1 changed (base `agent.slice` had `CPUQuota=150%` +
  `CPUWeight=1`, the exact shape that starved agent shells on 2026-08-15, while live
  already ran a hand override); fixed in the repo source this pass (fold-forward) but
  **not applied to the live laptop** (daemon-reload of this session's own cgroup
  mid-session judged unnecessary risk for same-day work; no live regression from leaving
  it unapplied since the live drop-in already carries the same effective values) and
  **not run against buildboxes at all** (no `agent.slice` deployed there today).
- Real apply, once: `fleet/apply.sh path_bins --limit laptop` — 0 changes (already
  converged), post-apply re-check confirms empty diff (receipt:
  `fleet/reports/postapply-20260815-124720-path_bins-laptop.txt`).
- Full drift report: `fleet/reports/drift-20260815-124720.md` + raw per-role diffs in
  the same directory, committed as evidence.
- Did NOT verify: buildbox PATH-bin/version-pin correctness against `devtools.json`;
  config homes beyond the one `~/.claude/settings.json` per host (plan's "13 config
  homes" figure is not fully enumerated yet); anything ssh/sshd (out of scope by design).

Next executable action: (1) reconcile `settings_template`'s repo/live divergence
field-by-field (not a blind copy) so it can be applied without regressing live
customization; (2) read `buildbox-parity.sh`/`devtools.json` install logic and model
the buildbox-correct `path_bins` desired state; (3) build and prove the buildbox deadman
mechanism before any role targets a buildbox with `fleet/apply.sh`; (4) once those land,
apply `systemd_units` to the laptop at a moment outside an active agent session, then
build the S2 check-mode timer unit (install dark, enable later per the plan's delivery
discipline).