# system-monitor — build spec

audience: AI coding agents first (Codex executes this). Optimize for activation, not prose.

## 0. Mission

Host all custom system alerts ON an existing extensible monitoring platform (netdata), NOT as a standalone platform. Build ONLY the checks nothing off-the-shelf ships. Deliver an idempotent, reversible, tested infra-as-code repo at `~/Projects/system-monitor/`.

Single sentence: netdata = metrics + alarms + dashboard host; a thin `agent-guard` sidecar = event-stream checks + interactive desktop notifications with action buttons; a few root-cause fixes stop two failure modes at source.

## 1. Machine facts (pinned — do NOT re-probe, do NOT assume beyond these)

- Debian 14, ThinkPad e14, 61G RAM, kernel 7.0.12, encrypted LVM, single user `user` (uid 1000).
- systemd 261, cgroup v2 unified (`cgroup2fs`). Slices + per-unit resource control available.
- Desktop: X-Cinnamon, X11. `notify-send` AND `gdbus` present at `/usr/bin`. D-Bus user session live.
- `python3` = 3.13. Standard-library only. NO pip deps, NO Go, NO Rust.
- netdata NOT installed and NOT in Debian apt (candidate: none) → install via netdata's SIGNED APT REPO (native `.deb`), NEVER `curl | sh`.
- Active daemons to BRIDGE (netdata reads them; do NOT reimplement): `smartmontools.service` (smartd), `thermal-watch.service`, `systemd-oomd.service`.
- `/tmp` = tmpfs 30.8G, 18.6G already consumed by stray-agent junk (junk eats RAM here).
- Root memory posture: `scripts/mem-guard-root.sh` is the ONLY source of truth. Apply = `deck-sudo bash scripts/mem-guard-root.sh`; audit = `--audit` (also runs in `health.sh`). NEVER re-enable `ManagedOOMSwap=kill`, a `SwapUsedLimit` below 100%, or a `user-.slice` `MemoryHigh`/`MemoryMax` — each has been measured to SIGKILL live agent sessions or freeze the box.
- Already-running user notifiers to RETIRE (this repo supersedes them): `~/.local/bin/mem-pressure-notify` (+ its `--user` service, currently active), `~/.local/bin/disk-fill-notify`. Port their proven logic; then disable+remove them in `install.sh` to prevent double-alerting.
- `~/.ssh` exists; `crontab` at `/usr/bin/crontab`.

## 2. Non-goals (do NOT build — reject on sight)

- NO custom dashboard UI. netdata's local dashboard IS the dashboard. `// reject:` any web server, HTML, JS.
- NO SQLite event store. netdata stores metrics; structured journald stores events. Sidecar keeps ONLY a small JSON state file (baselines + cooldowns).
- NO plugin-registry / entry-point machinery. ~5 checks in one repo → an explicit list IS the registry.
- NO polling of logs. Event-stream checks consume streams (journald follow, inotify), NOT a poll loop over log files.
- NO killing by a PID captured earlier. Kill only by RE-SCANNING at action time and matching cgroup + process start-time (PID-reuse safety).
- NO `curl | sh` installs. Signed apt repo only.
- Codex MUST NOT itself install netdata, write to `/etc`, `/usr`, run `sudo`, or add apt repos. ALL privileged steps live in `install.sh`, run SEPARATELY by the user via gui-askpass sudo. Codex builds files + the installer + non-root tests only.

## 3. Repo layout (create exactly)

```
~/Projects/system-monitor/
  README.md                 # what it is, install/uninstall, how each check maps to netdata
  SPEC.md                   # this file (leave as-is)
  install.sh                # idempotent, root; the ONLY privileged actor
  uninstall.sh              # idempotent reverse; restores pre-install state
  Makefile                  # `make test` (non-root unit), `make check` (lint/syntax)
  netdata/
    conf/netdata.conf                       # bind 127.0.0.1 only, telemetry off
    conf/stream-and-cloud-disabled.conf     # cloud/registry off
    plugins/mem_trajectory.plugin           # external plugin (python3), exec bit
    plugins/tmpfs_guard.plugin
    plugins/cpu_runaway.plugin
    plugins/proc_fd.plugin
    health.d/mem_trajectory.conf
    health.d/tmpfs_guard.conf
    health.d/cpu_runaway.conf
    health.d/proc_fd.conf
    health.d/disk_fill.conf                 # thresholds over netdata STOCK disk metrics
    notify/health_alarm_notify.conf         # route custom recipient -> agent-guard bridge
  agent-guard/
    pyproject.toml                          # src-layout, stdlib-only, no deps
    src/agent_guard/__init__.py
    src/agent_guard/daemon.py               # queue core: scheduler + stream producer threads
    src/agent_guard/events/journald.py      # journal follower matcher
    src/agent_guard/events/fswatch.py       # inotify on sensitive paths
    src/agent_guard/events/ports.py         # listening-port baseline diff
    src/agent_guard/notifier.py             # gdbus notify + action buttons + remediation
    src/agent_guard/culprit.py              # live re-scan: fastest-growing exe group + PIDs
    src/agent_guard/state.py                # JSON baselines + per-key cooldowns
    src/agent_guard/cli.py                  # `agentctl` entrypoint (netdata exec calls this)
    src/agent_guard/config.py               # load TOML (tomllib), defaults baked
    config/agent-guard.toml                 # thresholds, allowlists, watched paths
    systemd/agent-guard.service             # --user unit (self-survival hardened)
    tests/                                  # fault-injection + synthetic /proc unit tests
  slices/
    systemd/agents.slice systemd/builds.slice   # cgroup-fact CPU attribution
    bin/run-agent bin/run-build                 # wrappers: systemd-run --slice=...
  fixes/
    tmpfiles.d/system-monitor-tmp.conf      # age-out /tmp junk at source
```

## 4. netdata as host — install + config contract

`install.sh` MUST, idempotently:

1. Add netdata signed apt repo (fetch GPG key to `/usr/share/keyrings/`, write `/etc/apt/sources.list.d/netdata.list` with `signed-by=`), `apt-get update`, `apt-get install -y netdata`. If already installed at required version → skip. NEVER `curl | sh`.
2. Deploy `netdata/conf/*` into `/etc/netdata/` (netdata's config dir), `netdata/plugins/*.plugin` into netdata's custom-plugins dir with mode 0755 + owner root, `netdata/health.d/*` into `/etc/netdata/health.d/`.
3. Harden: bind web to `127.0.0.1` only; disable netdata cloud, registry, anonymous telemetry. netdata runs as its own `netdata` user.
4. Add the `netdata` user to whatever group is needed to read `smartd`/sensors IF required for the bridge collectors; enable stock collectors: `smartd_log` (bridge smartd), `sensors`/thermal (bridge thermal-watch), `systemdunits` (unit-failed), disk-space + disk-fill prediction, processes/fd. Do NOT reimplement these.
5. `systemctl enable --now netdata`; verify `http://127.0.0.1:19999` answers.

### 4.1 Custom collector contract (external plugins)

Each `*.plugin` = a python3 script that speaks netdata's external-plugin line protocol on stdout in a loop at its update interval (Codex: implement the real `CHART`/`DIMENSION`/`BEGIN`/`SET`/`END` protocol; consult netdata docs for exact syntax). Each MUST:

- Emit only NUMERIC metrics. Attribution/PIDs are NOT emitted here (that happens live in the sidecar at action time).
- Read `/proc` + cgroup files with stdlib; tolerate missing files (`try/except`, skip).
- Be pure-measurement, side-effect-free (never kill, never write outside its own stdout).

Metric contracts (chart family `system_monitor.*`):

| plugin | chart | dimensions | update |
|---|---|---|---|
| mem_trajectory | `mem_runway` | `runway_pct` (MemAvailable+SwapFree as % of MemTotal+SwapTotal), `eta_seconds` (projected s to FLOOR at recent slope; emit a large sentinel e.g. 999999 when not falling), `psi_full_avg10` | 5s |
| tmpfs_guard | `tmpfs_tmp` | `used_pct` (bytes), `inodes_pct`, `stale_junk_mb` (user-owned files older than config age) | 30s |
| cpu_runaway | `cpu_runaway` | `nonbuild_busy_pct` (sustained CPU of the busiest process group NOT classified as build), `build_busy_pct` | 10s |
| proc_fd | `proc_fd` | `open_fds_pct` (of `file-nr`/`file-max`), `proc_count`, `inotify_watch_pct` (of `max_user_watches`) | 15s |

mem_trajectory ETA math = port existing `mem-pressure-notify` exhaustion-trajectory logic (runway + recent-slope → seconds-to-floor; FLOOR = 8% of RAM+swap). Reference file: `~/.local/bin/mem-pressure-notify`.

cpu_runaway build-classification (build-aware, MANDATORY — a naive load alarm = pure false-positive noise on this box):
- Group processes by exe basename. A group is BUILD if EITHER: (a) its processes live under `builds.slice` (see §6, primary/robust signal), OR (b) exe/cmdline matches the build allowlist in `agent-guard.toml` (fallback for unwrapped strays): `tsc, vite, esbuild, rollup, webpack, turbo, pnpm, npm, yarn, node`(only when argv indicates a build/dev script), `cargo, rustc, cc, clang, gcc, ld, make, ninja, go`(build), `bun`(build).
- `nonbuild_busy_pct` reflects ONLY non-build groups. Alarm (in health.d) fires only on SUSTAINED (multi-minute) high non-build CPU → real runaway, not a build.

### 4.2 Health alarm contract

Each `health.d/*.conf` = netdata health config with hysteresis + cooldown (`delay`, `hysteresis`) on the custom metrics above (and STOCK metrics for `disk_fill.conf`). On CRITICAL, `exec` the notify bridge (§4.3). Alarm set (thresholds live in the conf, tunable):

- `mem_eta_critical`: `eta_seconds < 90` OR `psi_full_avg10 >= 20` OR `runway_pct < 8` → critical.
- `mem_eta_warn`: `eta_seconds < 360` → warning.
- `tmpfs_fill`: `used_pct > 85` OR `inodes_pct > 85` → warning; `> 95` critical.
- `disk_fill`: netdata stock out-of-space prediction < 1h → warning, < 15m critical (use netdata's built-in disk fill-rate metric; do NOT hand-roll disk ETA).
- `cpu_runaway`: `nonbuild_busy_pct` sustained (e.g. `> 90` for `> 300s` via `delay`/lookup window) → warning.
- `proc_fd`: `open_fds_pct > 80` OR `inotify_watch_pct > 80` → warning.

### 4.3 netdata → sidecar notification bridge

netdata alarm `exec` runs headless as the `netdata` user with NO desktop session → CANNOT show a user notification directly. Bridge:
- `notify/health_alarm_notify.conf` defines a custom recipient whose command calls the sidecar CLI: `agentctl alarm --name <alarm> --status <status> --value <v>` (exact invocation Codex wires per netdata's `health_alarm_notify.sh` custom-exec contract).
- `agentctl` runs as the `user` session (reached via the running `--user` `agent-guard` service's local socket OR a `systemd-run --user`/`busctl` handoff — Codex picks the robust mechanism; MUST deliver the notification into uid-1000's D-Bus session).
- For a mem alarm, the sidecar RE-SCANS live (`culprit.py`) to name the current fastest-growing exe group + its exact PIDs, then shows the notification WITH the kill button (§5.2). Never trust a PID from alarm time.

## 5. agent-guard sidecar — contract

Thin `--user` service. Handles ONLY what netdata does poorly: event streams + interactive desktop notifications + remediation actions.

### 5.1 Core (`daemon.py`)
- One main loop draining ONE queue. Producers: (a) a journald follower thread, (b) an inotify thread, (c) a low-frequency ports/baseline tick. NO asyncio. NO thread-per-check.
- Each producer emits a common `Event(tier, source, reason, culprit?, actions?)`. Main loop applies per-key cooldown (`state.py`) then dispatches to `notifier.py`.

### 5.2 Notifier (`notifier.py`)
- Persistent user-session desktop notification via `gdbus` calling `org.freedesktop.Notifications` (NOT repeated `notify-send` fork per event; `notify-send --wait` blocks — use gdbus and handle the `ActionInvoked` signal).
- Action buttons per tier. Mem-critical → `Kill <group> (N procs)` + `Dismiss`. tmpfs/disk → `Clean /tmp (safe)` + `Dismiss`.
- Remediation:
  - Kill: TERM the re-scanned PIDs, wait 3s, KILL survivors. MUST verify each PID's cgroup + start-time still matches the culprit before signalling (PID-reuse guard). NEVER signal a PID in the PROTECT allowlist (`config`).
  - Clean /tmp: delete ONLY files under `/tmp` owned by uid 1000 AND older than config age. NEVER `rm -rf /tmp`. NEVER touch non-owned or fresh files.
- Log every notification + action to structured journald (`journal.send` fields: `SM_TIER`, `SM_SOURCE`, `SM_ACTION`, `SM_TARGET`). journald IS the event store.

### 5.3 Event sources
- `journald.py`: follow the system journal; fire on OOM-kill (kernel + systemd-oomd), `EXT4-fs error`, `I/O error`, MCE, `hung_task`, `oops`/`BUG`, `remount .* read-only`. Reason = the matched line (trimmed). These also serve the separate hardware-hang investigation → include a stable `SM_SOURCE=kernel` tag.
- `fswatch.py`: inotify on `~/.ssh` (any write/create), the user + system crontab dirs, `~/.config/systemd/user/` and `/etc/systemd/system/` (new/changed unit). Fire tier=notice, with the changed path. Alert-fatigue guard: coalesce bursts, per-path cooldown.
- `ports.py`: diff current LISTEN sockets (`/proc/net/tcp*`, stdlib) vs a learned baseline in `state.py`. New listening port → notice. Alert-fatigue guard: `agent-guard.toml` `dev_port_allowlist` (ranges dev servers use, e.g. 3000-9999, 5173, 4321, 6006) suppressed by default; MUST ship with a sane dev-port allowlist or it self-DoSes on every vite launch.

### 5.4 Config (`agent-guard.toml`)
Tunables ONLY (logic in code). Keys: thresholds mirror §4.2, `protect` (never-kill exe list — seed from the existing `mem-pressure-notify` PROTECT set), build-allowlist, watched paths, `tmp_stale_hours`, `dev_port_allowlist`, cooldowns.

### 5.5 Self-survival (`agent-guard.service`, --user)
The guard MUST outlive the incident it watches:
```
[Service]
MemoryMax=200M
ManagedOOMPreference=avoid
WatchdogSec=30
Restart=always
RestartSec=5
```
`daemon.py` MUST `sd_notify` WATCHDOG pings (stdlib socket to `$NOTIFY_SOCKET`; no dep).

## 6. Root-cause fixes (stop failure modes at source, not just observe)

### 6.1 CPU attribution by cgroup fact (primary signal for §4.1 cpu_runaway)
- `slices/systemd/{agents,builds}.slice`: define with `CPUWeight` + `MemoryHigh` caps.
- `slices/bin/{run-agent,run-build}`: wrappers = `systemd-run --user --slice=<agents|builds>.slice -- "$@"`. Ship them; README instructs the user to launch heavy builds/agents through them. Unwrapped strays fall back to the cmdline allowlist. Document the tradeoff (wrapper = fact; allowlist = heuristic).

### 6.2 /tmp junk at source
- `fixes/tmpfiles.d/system-monitor-tmp.conf`: systemd-tmpfiles age policy aging out `/tmp` entries older than `tmp_stale_hours` (default 72h). `install.sh` deploys to `/etc/tmpfiles.d/` and runs `systemd-tmpfiles --clean`. This is the durable fix; the interactive "Clean /tmp" button is the manual escape hatch.

## 7. install.sh / uninstall.sh contract

- Idempotent: re-run = no-op if already in target state. Every step guarded.
- Fail-closed: unexpected state → error + detail + abort, NEVER force/guess.
- `install.sh` (root): apt repo + netdata install (§4), deploy all configs/plugins/tmpfiles/slices, enable netdata, enable the `--user` agent-guard service for uid 1000 (`systemctl --user` via the user's manager, or `loginctl enable-linger` + user-unit enable), disable+remove the two retired notifiers (§1), run `systemd-tmpfiles --clean`. Print a summary of every file written (for reversal).
- `uninstall.sh`: reverse all of the above; restore the retired notifiers ONLY if the user asks (default: leave removed, since superseded). Leave the root oomd/caps/swappiness prevention untouched.
- Runs SEPARATELY by the user with gui-askpass sudo. Codex does NOT execute it.

## 8. Tests (fable-mandated — parsing tests are NOT detection tests)

Two tiers. `make test` runs tier-1 (no root, Codex runs it to prove the build):

- Tier 1 (unit, non-root, synthetic /proc + fixtures): each collector emits correct protocol + correct metric given a crafted `/proc` snapshot (mem near-floor → small eta; plateau → sentinel; a build-tool group → excluded from nonbuild_busy; a non-build hog → counted). Sidecar: journald matcher flags a synthetic OOM line; ports diff flags a new non-allowlisted port; culprit picks the fastest grower; kill-guard refuses a PROTECT name and a start-time mismatch.
- Tier 2 (fault-injection integration, needs install; document in README, do NOT auto-run): actually drive MemAvailable down in a scratch cgroup and assert `mem_eta_critical` + notification fire; actually fill a scratch tmpfs and assert `tmpfs_fill`; actually crashloop a dummy `--user` unit and assert netdata `systemdunits` + alarm; write a file in a temp watched dir and assert `fswatch` fires. Assert the NOTIFICATION fires end-to-end, not just that a metric moved.

## 9. Acceptance checklist (Codex: satisfy every box)

- [ ] Repo built exactly per §3. stdlib-only, no pip/Go/Rust deps.
- [ ] `install.sh` idempotent, fail-closed, the ONLY privileged actor; Codex ran NO sudo/apt/`/etc` writes itself.
- [ ] netdata configs bind 127.0.0.1, cloud+telemetry off, bridge smartd/thermal/systemdunits/disk-fill (stock, not reimplemented).
- [ ] 4 custom collectors emit the §4.1 metric contracts; cpu_runaway is build-aware (slice-fact + allowlist-fallback).
- [ ] health.d alarms per §4.2 with hysteresis/cooldown; mem+tmpfs+disk criticals exec the bridge.
- [ ] netdata→sidecar bridge delivers into uid-1000 D-Bus session; mem alarm re-scans live for culprit+PIDs.
- [ ] Sidecar: queue core (no asyncio/thread-per-check), journald+inotify+ports sources, gdbus notifier with action buttons, kill-by-rescan with cgroup+start-time guard + PROTECT, safe /tmp clean, journald as event store.
- [ ] Self-survival: MemoryMax=200M + ManagedOOMPreference=avoid + WatchdogSec + sd_notify pings.
- [ ] Root-cause: slices+wrappers shipped; tmpfiles /tmp age policy shipped.
- [ ] Retired notifiers disabled+removed by install.sh; root prevention untouched.
- [ ] Alert-fatigue guards: dev-port allowlist + inotify burst coalescing present.
- [ ] `make test` (tier-1) green; tier-2 fault-injection documented.
- [ ] README maps every check → its netdata mechanism or sidecar source, + install/uninstall.
```