# Disk Hygiene and Build Admission

<!-- markdownlint-disable MD013 -->

Audience: AI coding agents first.

Status: Phase 1 implemented in this repository. Remaining waves are contracts for
follow-up work.

## Objective

Keep root filesystem below 85% during automated builds. Prevent queue time from consuming execution timeout. Delete only reproducible artifacts proven idle.

## Evidence baseline — 2026-07-15

- Root filesystem before cleanup: 819/870 GiB used; 6.8 GiB available; 100% reported.
- Root filesystem after cleanup: 709/870 GiB used; 117 GiB available; 86% reported.
- Inodes: 39% used. Block exhaustion caused incident; inode exhaustion did not.
- Rootless Podman storage: 135.7 GiB. Nine containers were stopped for 2–4 months. Unused-image cleanup reclaimed about 110 GiB. Named volumes remained.
- Concurrent `platform` and `multideal` Vitest/Astro/TypeScript builds were active across worktrees.
- Build-slot wait reached 270 seconds before test execution. Wrapper counted queue delay against execution timeout, retried work, then quarantined task.
- Large secondary consumers: `~/.cache` 58.8 GiB (`uv` 29.6 GiB), `~/.local/share/pnpm` 18.4 GiB, `trance-shop-israel/tmp` 20.5 GiB, `multideal/tmp` 5.7 GiB, `~/tmp/rp-iso` 9.0 GiB.
- `uv cache prune` was refused because cache lock was active. NEVER force cache cleanup while writer exists.

## Root causes

1. Unbounded rootless Podman image-layer retention consumed 15.6% of filesystem.
2. Multiple runplan marathons admitted concurrently on one workstation. Shared build semaphore serialized work after dispatch, too late to prevent retry amplification.
3. Timeout model conflated queue wait with execution time. Healthy queued jobs became false failures and duplicated build output.
4. Worktrees duplicate `node_modules`, build output, logs, and temporary isolation trees without terminal-state garbage collection.
5. No capacity admission gate, retention budgets, growth alert, or idle-only cleanup timer existed.

## Wave plan

| Wave | Change                                                  | Dependency       |
| ---- | ------------------------------------------------------- | ---------------- |
| 1    | Instrument disk, writer, run, cache, worktree inventory | None             |
| 2    | Add global run admission and timeout separation         | Wave 1 metrics   |
| 3    | Add terminal-state artifact garbage collection          | Run registry     |
| 4    | Add idle-only cache/container maintenance               | Writer detection |
| 5    | Add alerts and incident tests                           | Waves 1–4        |

## Wave 1 — observability

Implemented baseline:

- `netdata/plugins/disk_guard.plugin` emits root block usage, available GiB,
  reserved GiB, and inode usage every 30 seconds.
- `netdata/health.d/disk_fill.conf` warns at 75% or <100 GiB and becomes critical
  at 85% or <50 GiB.
- `slices/bin/run-build` rejects new builds at >=85% or <100 GiB before creating
  `builds.slice` scope.

- [ ] Create one host inventory command returning JSON: filesystem bytes/inodes, top growth deltas, active runplans, active build PIDs, Podman usage, cache usage, worktree path/status/age.
- [ ] Store daily snapshots for 30 days under `~/.local/state/disk-guard/`; cap directory at 100 MiB.
- [ ] Emit Prometheus textfile metrics or structured journal events.
- [ ] Acceptance: snapshot completes within 60 seconds and never traverses dependency trees synchronously; cached/indexed size inventory supplies deep totals.

## Wave 2 — admission control

- [ ] Add host-global run lock. Default maximum: one runplan marathon per workstation. Require explicit operator override for concurrent marathons.
- [ ] Reject new run when root usage is `>=85%` or available space is `<100 GiB`.
- [ ] Reserve projected space before launch: `max(20 GiB, 2 * historical peak delta for plan/repo)`.
- [ ] Keep build-slot queue timeout separate from execution timeout. Start execution clock only after slot acquisition.
- [ ] NEVER retry or quarantine solely because build-slot queue wait exceeded execution timeout. Report `RESOURCE_WAIT` with owner PID/run/age.
- [ ] Acceptance: two synthetic runs serialize; second run performs zero build writes before admission; 10-minute slot wait does not decrement execution budget.

## Wave 3 — lifecycle garbage collection

- [ ] Register every created worktree and temp isolation tree with owning run ID, creation time, terminal state, and cleanup policy.
- [ ] On successful terminal state: remove build outputs and dependency directories immediately; remove clean worktree after 24-hour debug window.
- [ ] On failed/quarantined state: retain logs and diff; remove reproducible build outputs; retain worktree seven days.
- [ ] NEVER remove dirty worktree, unpushed commit, active-process CWD, open-file target, mounted path, or non-terminal run path.
- [ ] Prune stale Git metadata only after filesystem path validation: `git worktree prune --expire=now`.
- [ ] Acceptance: fixtures cover active, dirty, unpushed, terminal-clean, and orphaned worktrees; only terminal-clean/orphaned fixtures delete.

## Wave 4 — bounded maintenance

- [ ] Run Podman cleanup weekly when no Podman build/container is active: prune stopped containers and unused images; NEVER pass `--volumes`.
- [ ] Fail and alert on Podman metadata errors. Preserve volume data until explicit backup and owner approval.
- [ ] Run `uv cache prune` only when cache lock is free. NEVER use `--force` from scheduled maintenance.
- [ ] Run `pnpm store prune` only when no `pnpm`, Node build, or runplan process is active.
- [ ] Apply size budgets: Podman 40 GiB, `uv` 20 GiB, pnpm store 20 GiB, agent temp/cache 10 GiB, per-run temp 10 GiB.
- [ ] Configure age-based cleanup for `/tmp` and `~/tmp`; exclude registered active run paths.
- [ ] Acceptance: maintenance exits without mutation while matching writer exists; idle run returns reclaimed bytes and post-clean sizes.

## Wave 5 — alerts and failure tests

- [ ] Alert at 75% usage; block new runs at 85%; stop dispatching new tasks at 92%; preserve active task and logs.
- [ ] Alert on >20 GiB/day growth by owner directory, build-slot wait >120 seconds, orphan worktree >7 days, or cleanup failure.
- [ ] Add disk-pressure integration test using bounded loopback filesystem. Verify admission rejection before ENOSPC.
- [ ] Add concurrency test reproducing two marathons, 270-second queue wait, wrapper timeout, and retry amplification.
- [ ] Acceptance: test proves one real execution, zero retry from queue delay, zero quarantine, and cleanup leaves dirty/unpushed worktrees intact.

## Operating rules

- Measure before deletion; record before/after bytes and deleted category.
- Delete reproducible cache/output first. Preserve source, commits, databases, volumes, user media, and active-run evidence.
- Stop cleanup at first ownership ambiguity. Emit exact path, size, PID/run owner, and required decision.
- Treat cleanup error as failed maintenance; NEVER report green with partial prune.