# stall-guard

Audience: AI coding agents first.

Turns silent multi-hour stalls into immediate loud failures. `bin/stall-guard`.

## Rule

**Never infer liveness from process existence.** A crash-looping or socket-blocked child holds a live PID for its entire timeout while doing nothing. Judge PROGRESS.

Four orthogonal monotonic signals, sampled across the whole process tree:

| Signal | Source | Catches |
|---|---|---|
| `out` | captured log size | runs that print |
| `cpu` | `/proc/<pid>/stat` utime+stime, recursive | silent long compile — NOT a hang |
| `io` | `/proc/<pid>/io` **rchar+wchar**, recursive | network waits (`read_bytes` would miss sockets) |
| `beat` | mtime of a heartbeat file | declared legitimate waits |

Two independent kill rules:

1. **every tracked signal flat for the full idle window** — catches PASSIVE hangs (blocked socket, deadlock, `sleep`).
2. **`out` alone flat for the output ceiling**, whatever cpu/io do — catches ACTIVE hangs (spin loop, retry loop). Rule 1 ANDs the signals, so one perpetually-moving signal would otherwise veto detection forever.

Deltas are accumulated as **positive rises only**. Raw counters are NOT monotonic — summed cpu/io fall when a child exits, `out` falls when a log is truncated — and comparing raw endpoints reads a decrease as progress, which silently disables the detector. Two real `cdx` traces contained a cpu decrease.

### The residual class this CANNOT catch

A process emitting **novel** output forever (`echo waiting for lock...` in a loop) is indistinguishable from progress by any content-agnostic signal. No progress rule fires on it. It gets a **non-killing** advisory (`--wall-notice`) instead, because the only rule that would kill it is a wall-clock cap — the one rule with a real false-positive cost. Verified: the chatty-poll trace stays quiet under both kill rules and does raise the advisory.

## Usage

```bash
# observe only — records a trace, never kills (DEFAULT)
stall-guard run --key ci-journeys --idle-window 900 -- <command>

# arm it
stall-guard run --key ci-journeys --idle-window 900 --enforce -- <command>

# a legitimate long wait declares itself alive
stall-guard run --key gate --heartbeat /run/user/1000/gate.beat -- <command>

# warn loudly at 3h without killing; kill only on the progress rules
stall-guard run --key gate --idle-window 1800 --output-ceiling 3600 --wall-notice 10800 --enforce -- <command>
```

For `cdx` this is wired in `modules/systray/stall_supervisor.py`; every knob has an env override (`CDX_STALL_IDLE_WINDOW`, `CDX_STALL_OUTPUT_CEILING`, `CDX_STALL_WALL_NOTICE`, `CDX_STALL_HEARTBEAT`, `CDX_STALL_ENFORCE=0`).

Stall exit code: **91**. Every run writes a JSONL trace to `~/.cache/stall-guard/`.

Desktop notifications are opt-IN: a verdict goes to stderr unless `STALL_GUARD_NOTIFY=1`. Only the `cdx`/`cld`/`claudex` CLI entry points set it, so every other caller — test suites, gates, worktrees, CI — is silent by default instead of raising critical popups.

## Before arming enforcement anywhere

Both arms MUST pass — a detector that never fires also passes the no-false-positive arm alone.

```bash
stall-guard replay ~/.cache/stall-guard/*.jsonl --idle-window 900 --expect quiet   # completed runs
stall-guard replay <known-stuck traces> --idle-window 900 --expect fired           # real stalls
```

## Picking the idle window

Measured from 5918 real `cdx` runs: p50 5.7m, p90 19.5m, p99 57m, p99.9 175m, max 531m. **Wall-clock alone is unusable** — a 30m cap false-positives on 4.56% of legitimate runs. That is why the progress signals are primary and wall-clock never kills by default.

Reasoning models at high effort go long stretches with no output. `cpu` is what keeps them alive; do not drop it. A real 11m47s `cdx` reasoning run went **195s with zero output** while cpu and io never stopped moving: a naive output-only watchdog at 120s would have killed it, and all-flat measured 0s.

Armed for `cdx`: idle window 1800s, output ceiling 3600s (18× the measured healthy output-flat max), advisory at 10800s ≈ p99.9.

## Declared waits

Any legitimate unbounded wait must touch a heartbeat, or it reads as stuck. Known ones: the `flock` in `apps/web/scripts/cpu-limit.sh`, `local-gate` slot/remote wait, the factory-DB barrier in `journey-correctness.yml`.

## Tests

`tests/test_stall_guard.py` — detector is a pure function over samples, so both arms are directly testable, plus end-to-end runs on real processes (stalled child killed, working child untouched, spin loop killed on the ceiling, setsid grandchild killed, slow-but-progressing run warned not killed, observe mode reports without killing).

The end-to-end tests are load-bearing. They caught an unsatisfiable window guard that made the detector unable to fire at all; unit tests over the pure function alone did not. The grandchild test was verified to FAIL against a `killpg`-only kill before being trusted — a test that never had a chance to fail proves nothing.

Killing signals the enumerated tree AND the process group, then **waits for the tree to actually die** before `SIGKILL`. `killpg` alone misses grandchildren that called `setsid`, and an unconditional sleep before `SIGKILL` lets a recycled PGID absorb the second signal.

Sampling or detector failure inside the supervisor reports **loudly** (stderr + notification) that the run is now UNGUARDED. Silently returning restores exactly the multi-hour blindness the tool exists to remove.