# Build-box scratch garbage collection

Audience: AI coding agents first.

Slug: `buildbox-scratch-gc`. Repo root: `/home/user/Projects/overdeck`. Every path is relative to it unless absolute.

## Problem

`local-gate` offloads gated builds to debian1/debian2. Each build rsyncs the local project root to a remote mirror at `${remote_root}/${mirrorName(root)}` where `mirrorName(root) = basename(root)-sha256(root)[0..12]`. Mirrors are ~1.1G.

Land operations create an **ephemeral** local worktree — `finish-branch.sh:441`, `mktemp -d "$cand_base/candidate-XXXXXX"`. Every one of those is a distinct local root, so every one mints a distinct permanent remote mirror. The worktree is removed at end of land; its mirror is not. Observed 2026-08-05: 24 mirrors, 41G, debian1 at 0 bytes free, `local-gate` retrying the remote push forever with no surfaced error (tool timeout 24h ⇒ the agent goes silent).

A GC exists and did not help. `lib/remote-build-gc.mjs`, called from `tryRemoteBuild` (`lib/remote-build.mjs:673`), stamps `~/.claude/run/local-gate/gc-<host>.stamp`. It last ran 2026-08-04 22:41 and reclaimed nothing; the box hit zero at 03:06. Four defects, all structural:

1. **Throttle is 24h** (`gc_interval_hours: 24`). Ephemeral mirrors arrive at ~65/day.
2. **Retention is 7 days** (`gc_retention_days`). A candidate mirror is dead the instant its land finishes.
3. **The floor pass cannot evict a young mirror.** `gcScript` requires `now - ts > 86400` before a mirror is eligible for floor eviction. On the day that fills the box, every mirror is younger than that, the loop finds no candidate, `break`s, and the box stays full. This is precisely the observed failure.
4. **Policy is age-only.** Nothing in the system knows a run finished. There is no ownership signal, so GC cannot distinguish a dead candidate mirror from the live project-reuse cache and therefore treats both by clock.

The box-side `~/.local/bin/ci-scratch-prune.sh` (hourly timer on debian1) is not a fix: it prunes `~/builds` at `-mmin +1440` — which both misses the same-day flood *and* deletes long-lived reuse mirrors that are merely idle — and its disk-floor alarm only calls `logger`, never reclaims. Its journal shows it firing four times in 24h with no effect. It is also hand-placed, not version-controlled, so the two boxes cannot be proven identical.

## Goal

Reclaim mirror scratch on **run completion**, not on a clock. Keep a crash-safe backstop. Never destroy the incremental-sync reuse cache.

Non-goal: reducing mirror size, changing rsync strategy, or touching job scheduling.

## Invariants

These bind every layer. A violation is a defect, not a tuning question.

- **A job dir with no `rc` and an mtime older than `gc_job_stale_hours` is abandoned, not live.** Measured on debian1: 47 job dirs carry no `rc`, the oldest 190 hours old — crashed jobs that never wrote a return code. A liveness rule of "no `rc` ⇒ live" would let each of them pin its mirror permanently, which is the same immortality failure this project exists to close, moved one level down. An abandoned job dir is itself garbage and is reaped with the mirror.
- **Never delete a mirror with a live job.** Live = some `~/.rb/jobs/<id>/meta.json` has `mirror` equal to this mirror path and the job dir has no `rc` file. `rc` presence is the existing completion signal `fenceOlderJobs` already relies on; reuse it, do not invent a second one.
- **Never delete a mirror whose local root still exists.** That is the reuse cache. Only the disk-floor layer may override this, and only when below floor.
- **Never delete outside `remote_root`.** Refuse to act when `remote_root` is empty, `/`, or `$HOME`. Never touch `.repos/` except under its own explicit retention.
- **Fail closed.** Unparseable owner record, unreadable job meta, ssh failure, ambiguous state ⇒ skip that entry and report it. Never guess, never force.
- **Never fail the caller.** GC is opportunistic. A failed sweep logs and returns; it never fails a land, a build, or a gate.
- **Deterministic and idempotent.** Running any layer twice on the same state removes the same set the first time and nothing the second.

## Architecture

Four layers, strongest signal first. Each is independently useful; later layers only catch what earlier ones missed.

```dot
digraph gc {
  rankdir=LR; node [shape=box];
  "L0 owner record\n(syncPush)" -> "L1 explicit release\n(worktree teardown)";
  "L0 owner record\n(syncPush)" -> "L2 orphan sweep\n(before each push)";
  "L2 orphan sweep\n(before each push)" -> "L3 floor reclaim\n(client, live-job-guarded)";
  "L3 floor reclaim\n(client, live-job-guarded)" -> "L4 host backstop\n(buildbox-gc timer)";
}
```

### L0 — ownership record

The enabling primitive. Without it the remote cannot tell a dead candidate from the reuse cache, which is defect 4.

`syncPush` writes, on every push, a single-line JSON file **inside the mirror** at `${mirrorPath}/.rb-owner`:

```json
{"root":"/abs/local/root","client":"<os.hostname()>","updated":<epochSeconds>}
```

Living inside the mirror means it is created, replaced, and destroyed with the mirror — no second index to keep consistent.

Seams:

- `ownerRecord(root, now): string` — pure, one line, no trailing newline ambiguity. Exported for test.
- `.rb-owner` MUST be added to the rsync push exclude set and to `pull_excludes` in `DEFAULT_REMOTE_CONFIG` (`lib/remote-build.mjs:22`), beside `.rb-epoch`. Omitting either lets `rsync -az --delete` erase the record on the next push or drag it into the local tree on pull.

### L1 — explicit release at run completion

The primary mechanism. User contract: cleanup happens when the run finishes, not after an interval.

`releaseMirror({ cfg, root, ssh, log }): { host, status, detail }[]` — new export in `lib/remote-build-gc.mjs`. For each configured host, one ssh running a script that:

1. Resolves `mirrorPath = ${remote_root}/${mirrorName(root)}`; exits `gc-absent` if it does not exist.
2. Scans `~/.rb/jobs/*/meta.json`; if any entry matches this mirror, has no `rc`, **and** has a job-dir mtime newer than `gc_job_stale_hours`, prints `gc-busy <mirror>` and exits without deleting. A matching job with no `rc` and an older mtime is **abandoned** — it does not block, and it is reaped in step 3.
3. Otherwise `rm -rf` the mirror, then `rm -rf` every job dir whose meta matches this mirror (each is finished or abandoned by step 2), printing `gc-released <mirror> jobs=<n>`.

Status values: `released | busy | absent | error`. `busy` is not an error — the mirror is left for L2/L3.

CLI: `bin/remote-mirror` with subcommand `release <root>` (and `sweep`, `status`), installed alongside the other `~/.claude/bin` tools. Exit 0 on `released|absent|busy`; non-zero only on a usage error.

Call sites, both non-fatal (`|| true`, never in an `&&` chain that gates the land):

- `workflows/lib/finish-branch.sh` — immediately after the `candidate-*` worktree is removed, for that worktree's path. This is the site that closes the 65-mirrors/day leak at its source.
- `bin/wt-reaper.sh` — for each worktree it reaps.

### L2 — orphan sweep at allocation

The crash backstop the user asked for: a run that dies before L1 runs leaves an owned mirror whose local root is gone.

`sweepOrphanMirrors({ cfg, stateDir, ssh, log, now }): { removed, busy, skipped }` — new export. Runs **inside `tryRemoteBuild`, before `syncPush`**, i.e. exactly when space is about to be needed. No timer, no daemon, no new failure surface.

1. One ssh emits every `${remote_root}/*/.rb-owner` as `<mirror>\t<record>` lines.
2. Locally: keep records where `client === os.hostname()` and `existsSync(root) === false`. A record from another client machine is never actionable here — skip it, it is L4's problem. An unparseable record is skipped and counted.
3. Release each survivor through the same host-side script as L1 (busy guard included).
4. The mirror for the build about to run is passed as `protect` and is never a candidate.

Throttle: `gc_sweep_interval_minutes`, default **10**, stamped per host in `stateDir`. This replaces `gc_interval_hours: 24` as the pacing for orphan and floor work; the 24h stamp is removed, not merely lowered, because it is the wrong unit for a per-run mechanism.

`sweepOrphanMirrors` takes `force?: boolean`, which bypasses the throttle (the stamp is still written). `bin/remote-mirror sweep --force` exposes it. This is what makes an orphan sweep observable on demand — verification and operator use both need a run that is not silently throttled into a no-op.

### L3 — floor reclaim, fixed

Last-resort space recovery on the client path. Same throttle and same ssh round as L2.

`gcScript` is rewritten around the live-job guard instead of the clock:

- **Removed:** the `now - ts > 86400` eligibility test in the floor loop (defect 3) and the `retentionDays` age sweep over owned, root-alive mirrors (defect 2). Age is no longer evidence of garbage.
- **Rewritten:** the grace sweep now keys on **`.rb-owner` absent, regardless of `.rb-epoch`**, at `gc_unowned_grace_hours`, ordered and aged off `.rb-epoch` mtime when present and dir mtime otherwise, with the live-job guard applied. Unowned means *unattributable*, which is a different evidence class from *stale*: no local root can ever be resolved for it, so no later pass will ever claim it. Without this predicate every pre-upgrade mirror — including the 9 `candidate-*` orphans on debian1, which do carry `.rb-epoch` — is immortal above the floor, and the new policy would be strictly worse than the one it replaces. This does not weaken "age is never eligibility for an **owned** mirror"; owned mirrors remain exempt from every age rule.
- **Kept:** `.repos/` retention at `gc_repo_retention_days`, unchanged.
- **Floor loop:** while free space under `remote_root` is below `gc_min_free_gb`, evict the mirror with the oldest `.rb-epoch` mtime among those that are (a) not live-job-guarded, (b) not `protect`, (c) not `.repos`. Age is now an *ordering* key, never an eligibility gate. Stop when no candidate remains, printing `gc-floor-exhausted <freeGb>` so an unreclaimable box is visible instead of silent.

The floor loop is the only place an owned, root-alive mirror may be deleted, and only under demonstrated pressure. Cost of a wrong eviction is one full re-sync; cost of not evicting is the observed 24h silent hang.

### L4 — host-side backstop

What the client cannot do: reclaim when no client is running, or when the owning client machine is off.

Ship a version-controlled `buildbox-gc` (script + systemd user timer, hourly) through `modules/buildbox`, so both hosts converge from one source and `buildbox audit` can prove it. It performs only host-local work:

- Reap job dirs under `~/.rb/jobs/` that have an `rc` file older than 1 day, **and** job dirs with no `rc` whose mtime is older than `gc_job_stale_hours` (abandoned). Measured: 5,510 job dirs on debian1 and 10,293 on debian2, of which 47 are abandoned with no `rc`. Nothing reaps this directory today.
- Reclaim to the disk floor using the same live-job-guarded oldest-first policy as L3.
- Reap mirrors with **no `.rb-owner`** (regardless of `.rb-epoch`) older than `gc_unowned_grace_hours`, same predicate as L3's grace sweep.
- It MUST NOT delete an owned, root-alive-unknown mirror above the floor. It cannot see local roots; ownership is the client's to resolve.

`modules/buildbox/lib/buildbox-checks.sh` gains `item_scratch_gc` in the existing declarative `item_*` / `ok|bad|fixed` / `MODE=audit|bootstrap` form: audit reports drift when the unit or timer is missing, inactive, or stale; bootstrap installs and enables it.

`~/.local/bin/ci-scratch-prune.sh` loses its `~/builds` prune entirely — that responsibility moves to `buildbox-gc`, which is the only component that knows the liveness rules. Its `/tmp` prune and floor alarm stay, and the alarm now fires **after** `buildbox-gc` has had its chance, so an alarm means genuinely unreclaimable, not merely unattended.

## Configuration

Added to `build-remote.json`, all optional with defaults in `GC_DEFAULTS`:

| Key | Default | Meaning |
|---|---|---|
| `gc_sweep_interval_minutes` | `10` | pacing for L2 + L3 |
| `gc_min_free_gb` | `15` | floor (unchanged) |
| `gc_repo_retention_days` | `30` | `.repos/` retention (unchanged) |
| `gc_unowned_grace_hours` | `24` | grace for unowned/unmarked dirs |
| `gc_job_stale_hours` | `6` | past this, an `rc`-less job dir is abandoned, not live |

Removed: `gc_interval_hours`, `gc_retention_days`. Both encode the age-based policy this design replaces; leaving them would let a stale config silently restore the broken behavior.

## Error handling

| Condition | Behavior |
|---|---|
| ssh fails / times out | log, return `{ok:false}`, caller proceeds |
| `remote_root` empty, `/`, or `$HOME` | refuse, log `gc-refused-root`, no ssh |
| owner record unparseable | skip that mirror, count in `skipped` |
| job meta unreadable | treat mirror as **live** (fail closed), skip |
| mirror busy | report `busy`, leave for a later pass |
| floor unreachable | print `gc-floor-exhausted <freeGb>` and stop; never loop |

## Testing

Extend `tests/remote-build-gc.test.mjs` (existing harness; it already drives `gcScript` against a real temp dir via `spawnSync`). Every case below is required:

1. `ownerRecord` is a single line, stable for identical input, and round-trips through the sweep parser.
2. Release deletes a mirror with no live job, and its finished job dirs.
3. Release refuses a mirror with a job lacking `rc` (`busy`), and the mirror survives.
4. Release of a nonexistent mirror is `absent`, exit 0.
5. Sweep selects only records whose `client` matches and whose `root` is gone; a live root and a foreign client are both left alone.
6. Sweep never selects `protect`.
7. Unparseable owner record is skipped, counted, and does not abort the sweep.
8. **Floor evicts a same-day idle mirror** — the exact regression that wedged the box. Fabricate mirrors all with `.rb-epoch` mtime of "now", set the floor unreachable, assert eviction happens in oldest-first order and that a live-job mirror and `protect` are never touched.
9. Floor stops at `gc-floor-exhausted` instead of spinning when only guarded mirrors remain.
10. `.repos/` is untouched by every pass except its own retention.
11. Refusal on `remote_root` of `/`, `""`, `$HOME` — no ssh invoked.
12. ssh failure returns cleanly and the caller still proceeds.
13. `.rb-owner` appears in both the push exclude set and `pull_excludes`.
14. The grace sweep reclaims a mirror that has `.rb-epoch` but **no** `.rb-owner` once past `gc_unowned_grace_hours`, and leaves one inside the grace window. This is the pre-upgrade-orphan class.
15. **An abandoned job does not pin its mirror.** A job dir with no `rc` and an mtime older than `gc_job_stale_hours` must not make release report `busy`; the mirror and that job dir are both reclaimed. A job dir with no `rc` and a fresh mtime must still report `busy`. Without case 15 a single crashed job makes its mirror immortal.

**Every test MUST target a fabricated `remote_root` under the test's own temp dir. NEVER the configured `remote_root`, and never a path under `~/builds` on any host.** The live box already carries `gcbusy-f5af4982c4b9` and `gcfloor-9168994962b1` — mirrors of roots named `gcbusy`/`gcfloor`, i.e. a prior GC test that reached the real machine. A deletion test pointed at the real root can eat live reuse mirrors.

**Required disposition of the three existing cases — they assert the behavior being removed. Do not leave them to be "made green".**

| Existing case | Disposition |
|---|---|
| `tests/remote-build-gc.test.mjs:68-69` — *"floor … but never same-day ones"* | **Invert.** That clause IS defect 3. It becomes case 8 above. |
| `:31` — retention case (stale marked mirror dies, fresh lives) | **Delete.** `gc_retention_days` is removed; there is no age retention for owned mirrors. |
| `:43` — unmarked-mirror grace | **Rewrite** to the new predicate: keyed on `.rb-owner` absence, not `.rb-epoch` absence. |

A red pre-existing test is not a signal to restore the old guard. Restoring `now - ts > 86400` in the floor loop reintroduces the wedge and is a rejection, however green the gate goes.

Shell-side, following `tests/remote-build-integration.test.sh`: `bin/remote-mirror release` end-to-end against a fabricated `remote_root` over loopback ssh, asserting exit codes and printed status tokens.

`modules/buildbox`: `buildbox audit` reports drift on a host with the timer removed; `buildbox bootstrap` installs it and a second audit is clean.

## Live verification — required before this is considered done

Unit tests do not prove the leak is closed. On debian1 and debian2:

1. Record baseline: `ls -d ~/builds/*/ | wc -l`, `du -sh ~/builds`, `df -h ~`.
2. Run a real land from a throwaway worktree so a `candidate-*` mirror is minted; confirm it exists remotely.
3. Complete the land. Assert the `candidate-*` mirror is gone within that operation (L1), with **no** wait interval.
4. Kill a land mid-flight so L1 never runs; confirm the orphan mirror exists; run `bin/remote-mirror sweep --force`; assert the orphan is gone (L2) and the project reuse mirror (`invariantum-*`, `md-*`) still exists. `--force` is required here: the 10-minute throttle would otherwise make this step pass or fail depending on whether a gated build happened to run in the preceding 10 minutes.
5. Assert the long-lived reuse mirrors survived all of the above — a GC that eats the cache is a regression, not a fix.
6. Report before/after counts and free space for both hosts.

The current 9 `candidate-*` mirrors on debian1 are pre-existing orphans; step 4's sweep MUST reclaim them, and that reclamation is the acceptance evidence.

## Architecture Decisions

- **Ownership record inside the mirror, not a side index.** A `${remote_root}/.rb-owners/` directory would need its own consistency handling on every delete path. Inside-the-mirror is destroyed atomically with what it describes. Cost: it must be excluded from rsync both directions.
- **L2 runs at allocation, not on a timer.** Deletion test: remove L2 and the complexity moves to a new user timer, a new unit to converge, and a new failure mode when it is not installed. Sweeping immediately before the push runs it exactly when space is about to be consumed and adds no daemon.
- **L1 and L2 are not collapsed.** They answer different questions — "this run just ended" versus "this run never ended". L1 alone leaks on every crash; L2 alone reintroduces the delay the user explicitly rejected.
- **Client and host layers both retained.** L3 covers a live client on a wedged box; L4 covers a box with no client attached. Neither subsumes the other.
- **Rejected: delete the mirror after every build.** Destroys incremental-rsync reuse and forces a full re-clone per build. The correct boundary is the local root's lifetime, not the build's.
- **Rejected: lowering `gc_interval_hours` to minutes.** Leaves defects 2 and 3 intact — a young mirror still cannot be evicted under pressure, so the observed wedge recurs unchanged.