# pids-guard — kill, don't wedge audience: AI coding agents first. **A cgroup at its `pids.max` denies `fork()` to everything inside it, including whatever would fix it. NEVER raise a cap to escape that. Kill the runaway.** ## Rules - **NEVER write `pids.max` / `TasksMax` from a command line.** A PreToolUse hook (`pids-cap-guard`) denies it; there is no `# raw-ok` escape. Caps live in `modules/monitor/systemd/user/*.slice` and `modules/workstation/claude/lib/confine.sh` — change them in a commit. - **NEVER `cgroup.kill` anything by hand.** `pids-guard` kills agent scopes automatically; `human-kill-guard` denies the manual form. - Wedged scope with a session inside it → `pids-rescue --keep-comm claude,codex,cursor-agent`. - Exhaustion-class tests run in the danger lab ONLY: `modules/monitor/tests/*.dangerlab.sh`, per `Projects/0 DOCS/dangerlab.md`. They refuse to run on any host but `dangerlab`. ## What the watchdog does `pids-guard --watch` (unit `pids-guard.service`, slice `guard.slice`) samples every 100ms and writes `cgroup.kill` on any **agent-owned scope** at or past **60% of its effective cap** (tightest `pids.max` on its path). Kill path = pure file I/O, no subprocess: it still runs when `fork()` is denied machine-wide. `guard.slice` is uncapped and `ManagedOOMPreference=omit`, so the watchdog never shares a limit with what it watches. The diagnostic arm (pressure warnings, desktop-placement self-heal, `notify-send`) runs every 150th tick and is exception-isolated — a failure there can never delay a kill. ## Killable — the whole list `is_killable()` in `modules/monitor/lib/pids_cgroup.py`, one source of truth for guard and rescue. ALL must hold: 1. path under `/user.slice/user-.slice/user@.service/` 2. first component ∈ `agent.slice`, `agent-seat.slice`, `build.slice`, `unsafe.slice` 3. NO component matching `human.slice`, `app.slice`, `init.scope`, `session-N.scope`, `vte-spawn-*.scope` 4. leaf named by a known agent launcher: `confine-{agent,build}--.scope`, `agent-seat-.scope`, `unsafe---.scope`, `run-.scope` Fail-closed: anything unrecognised is NOT killed. A slice is never a target (killing it kills every sibling agent). Deepest qualifying scope wins — a nested build scope dies, its parent agent session lives. **Slice saturation is also a trigger.** Six scopes at 512 of a 1024 cap fill a 3072 `agent.slice`: no scope is over its own threshold, yet all six are about to be fork-denied. A killable slice past 60% of ITS cap therefore contributes its largest scope as a target. The slice is still never killed. ## Caps | cgroup | TasksMax | measured peak of real work | |---|---:|---:| | `agent.slice` | 3072 | — | | `unsafe.slice` | 2048 | — | | `build.slice` | 1024 | 150 (all-time slice peak) | | per-launch `confine-agent-*.scope` | 1024 | 276 | | per-launch `confine-build-*.scope` | 512 | 150 | | `agent-seat.slice` | 512 | — | | `app.slice` | 24000 | watched, NEVER killed — holds the terminal scopes | An `unsafe-*` hatch scope declares no cap of its own, so its effective cap is `unsafe.slice`'s 2048 and the guard arms at ~1229 tasks. The hatches are owner-launched, and killing one ends that session — but a wedged hatch ends it too AND takes the slice with it. To keep the session, evacuate first: `pids-rescue --keep-comm claude,codex,cursor-agent`. Kill threshold 60% ⇒ the guard arms at ~615 tasks in a 1024 scope — 2.2× the measured peak, so real work is never a target. It arms 409 tasks below the cap, but a bomb forking faster than ~4000/s still saturates for up to one 100ms tick before dying. **The guarantee is containment, not never-touching-the-cap**: measured in the lab, `agent.slice` recorded zero denied forks and a sibling agent scope forked 200/200 while the bomb died. ## Stall arm — kill a wedge, never a size A second arm in the same daemon. **No memory ceiling kills anything any more** — `confine.sh`, `build.slice.d/90-ceiling.conf` and `git-hook-confine.sh` set no `MemoryMax`/`MemoryHigh`, so a legitimate 35 GB build runs. What dies instead is a cgroup making **no forward progress at all**. Sampled every `PIDS_STALL_SAMPLE_S` (30s), never on the 100ms path, exception-isolated exactly like the diagnostic arm. Detector: `modules/monitor/lib/stall_detect.py`, shared with `stall-guard`. Three signals, all read per cgroup, accumulated as **positive rises only** (raw counters fall when a child exits): | signal | source | |---|---| | `cpu` | `cpu.stat` `usage_usec` | | `io` | `rchar + wchar` summed over `/proc//io` for `cgroup.procs` — no `io` controller is delegated, so `io.stat` does not exist | | `pids` | `pids.current` | **`io` is absent, not zero, when the kernel refuses it.** `/proc//io` is ptrace-gated and a bwrap-confined agent tree denies it to its own uid (mode `0400`, same uid, `yama/ptrace_scope=0` — `open()` still returns `EACCES`). Swallowing that error left `io` a constant `0`, and a constant counter is always flat, so it could never dissent: the three-signal AND had silently degraded to `cpu AND pids`. `_read_proc_io` now returns `None` when a cgroup has tasks and not one was readable, and `classify_cgroup` refuses to answer while any signal is absent. The arm says so — `STALL-BLIND unreadable=io`, at most once per `PIDS_STALL_LOG_EVERY` (300s) — rather than degrading into a narrower rule that still fires. A kill needs ALL THREE flat for the full `PIDS_STALL_IDLE_WINDOW_S` (1800s) **and** the machine to be actually short of memory. Flatness alone is not a runaway: an agent blocked on a socket read waiting for a remote build has no CPU, no IO and no forks, and it is indistinguishable from a wedge by any local activity signal. It is also taking nothing from anyone. The scarcity term is `/proc/pressure/memory` `some avg60 >= PIDS_STALL_PRESSURE_PCT` (10) — the share of the last minute in which some task stalled waiting on memory. Unreadable pressure never kills (`STALL-SKIP-NO-PRESSURE`). When several stalled cgroups qualify, only the **largest memory holder** is killed per pass: killing a small stalled scope costs the owner work without returning the memory the machine is short of. Two eligibility gates, evaluated before anything is sampled: `memory.current >= PIDS_STALL_FLOOR_FRACTION` of `MemTotal` (5%) and `pids.current >= PIDS_STALL_MIN_PIDS` (2). `is_killable()` above still decides what may be touched at all. **The floor is relative to the machine, not a byte count, and it gates observation only.** `bash+sleep`, `bash+tail+timeout` and `ssh`-only scopes are genuinely flat on the readable signals; they are excluded because they sit far below it. It never triggers a kill — raising it only makes the arm sample less. History is dropped, never treated as flatness, when a counter cannot be read, when the wall-clock gap between samples exceeds 3× the sample interval (a stopped daemon must not wake up and see 1800s of flatness), or when the cgroup is gone. **Owner veto**, evaluated lazily only for an already-classified stall: any pid in `cgroup.procs` matching a live ledger record (`~/.local/state/agent-sessions/sessions/*.json`, `finishedAt` null) with `launchedBy == "user"` cancels the kill and logs `STALL-SKIP-OWNER`. This is a *positive owner veto*, not a fail-closed enrolment check: measured over 74 live records, `launchedBy` is null on 59 and `"agent"` on none, so vetoing on "not agent" would veto every target. Absence or null does not veto — `is_killable()` carries safety there. Every scope holding a `claude` process read `"user"`. **The ledger must be able to answer at all.** An owner session lives in the same `agent.slice` this arm kills from, so the ledger is the only thing that marks it. If the sessions directory is unreadable or holds no live parseable record, no cgroup can be vouched for: the arm logs `STALL-SKIP-NO-LEDGER` (at most once per kill cooldown) and kills nothing that pass. Sampling continues, so history is intact when the ledger recovers. A *healthy* ledger holding no record for a given cgroup still allows the kill — that is the positive-veto rule above, unchanged. `pids-guard --stall-once` runs one sample+classify pass without the 100ms kill loop, persisting history to `stall-state.json`. ### What this does NOT catch - **A runaway burning CPU while leaking memory.** Not separable from a legitimate 35 GB build by any cgroup-level signal. `ManagedOOMMemoryPressure=kill` at 60% on `build.slice`/`agent.slice` remains the backstop. No new threshold is introduced under a new name. - **A process forking below the pid cap while burning CPU.** - **A tree legitimately blocked a full 30 minutes** on a remote build while holding more than the floor — the real false-positive class. - **A long-lived idle helper parked in a killable slice.** `rootlessport` (rootless podman) was observed in `agent.slice` with no ledger entry; flat by nature, under the floor today. ## Arm / re-arm `modules/monitor/install.sh` arms this and is idempotent: ```bash deck-sudo sh /home/user/.local/share/overdeck/deploy/modules/monitor/install.sh ``` Run it from the deploy clone, never a worktree — the units and `~/.local/bin` entries are symlinked to `$REPO_ROOT`, so a worktree run leaves the daemon executing a tree that disappears at land. `deploy_desktop_survivability` runs FIRST: under `set -eu` anything ahead of it could abort the run and leave the machine unprotected. `verify_guards` runs last and exits non-zero unless `pids-guard.service` is active, `guard.slice` has `TasksMax=infinity`, and all four links resolve under `$REPO_ROOT`. A stale `~/.config/systemd/user.control/.slice.d/50-TasksMax.conf` from an old `systemctl set-property` outranks the repo unit. Delete that one file and `daemon-reload` — the repo cap then applies. ## Recovery — `pids-rescue` ```bash pids-rescue --keep-comm claude,codex,cursor-agent # evacuate, then kill pids-rescue --dry-run # report only ``` Run it from a HEALTHY cgroup (a terminal in `human.slice` can still fork) — it cannot start inside the wedge. It touches the wedged cgroup through file I/O only: writes the named pids into a fresh leaf under `human.slice`, reads BOTH `cgroup.procs` back to confirm the move, then writes `cgroup.kill`. Unconfirmed move ⇒ refuses to kill. Non-agent target ⇒ exit 3. ## Verify ```bash python3 -m pytest modules/monitor/tests # decision logic, synthetic inputs bash modules/workstation/claude/tests/pids-cap-guard.test.sh # the cap-write gate # danger lab (debian1), never the workstation: /home/user/dangerlab/dangerlab-run --payload /modules/monitor --timeout 240 \ -- bash monitor/tests/pids-guard-kill.dangerlab.sh ```