# system-monitor — Grafana/Prometheus provisioning spec

audience: AI coding agents first (Codex executes). Optimize for activation. Register: caveman on prose; normative keywords + identifiers + PromQL verbatim.

## 0. Mission

Provision a clean, good-looking Grafana dashboard + a full set of alert rules for the agreed system signals, backed by Prometheus + node_exporter. Reuse metric logic ALREADY in this repo. Deliver reproducible provisioning + one idempotent installer. Codex builds files ONLY; the USER runs the installer via gui-askpass sudo.

## 1. Machine facts (pinned — do NOT re-probe)

- Stack live: node_exporter @127.0.0.1:9100, prometheus v3.13 @127.0.0.1:9090, grafana OSS 13.1 @127.0.0.1:3000. Prometheus scrapes node+prometheus jobs, both `up`.
- node_exporter (Debian pkg `prometheus-node-exporter`) ALREADY exposes: PSI `node_pressure_{memory,cpu,io}_{waiting,stalled}_seconds_total`, `node_systemd_unit_state`, `node_hwmon_temp_celsius`, `node_thermal_zone_temp`, `node_filefd_{allocated,maximum}`, `node_filesystem_{avail,size,files,files_free}_bytes`, `node_network_*`, `node_vmstat_oom_kill`, `node_textfile_scrape_error`, full meminfo.
- MISSING and MUST be enabled: `--collector.processes` (gives `node_processes_*` threads/procs). Textfile collector loaded but NO directory set → MUST add `--collector.textfile.directory=/var/lib/node_exporter/textfile`.
- node_exporter args live in `/etc/default/prometheus-node-exporter` (`ARGS=...`), currently `--web.listen-address=127.0.0.1:9100`.
- Grafana: `/etc/grafana/grafana.ini` (localhost + all phone-home OFF — KEEP), provisioning under `/etc/grafana/provisioning/{datasources,dashboards,alerting}/`. Prometheus already default datasource. Login admin/admin.
- Existing repo logic to REUSE (do NOT rewrite the math): `netdata/plugins/mem_trajectory.plugin` (runway+recent-slope→exhaustion ETA seconds + PSI), `netdata/plugins/cpu_runaway.plugin` (build-aware classification: exe basename grouped, BUILD if under builds.slice OR cmdline matches build allowlist; emits nonbuild busy), `netdata/plugins/tmpfs_guard.plugin` (/tmp used/inodes/stale-owned-bytes), `agent-guard/src/agent_guard/*` for PROTECT set + build allowlist. These emit netdata protocol; the producers below emit Prometheus textfile format instead — SAME numbers, different output writer.

## 2. Non-goals (reject on sight)

- NO event-log/desktop-notification alerts here (kernel EXT4/MCE/hung_task/oops, fs-remount-ro, ssh-writes, new-port, one-click kill/clean). Those are the agent-guard SIDECAR, a separate pass. Do NOT build them here.
- NO exposing any port beyond 127.0.0.1. NO re-enabling grafana cloud/phone-home.
- NO Loki / no new daemons. Custom metrics arrive ONLY via node_exporter textfile `.prom` files written by short-lived timer units.
- Codex runs NO sudo/apt/`/etc` writes; all privileged steps live in the installer.

## 3. Deliverables (repo layout)

```
~/Projects/system-monitor/
  grafana/
    install-dashboards.sh              # idempotent, root; the only privileged actor for THIS pass
    node_exporter/textfile-producers/  # .prom writers (python3 stdlib), reuse repo logic
      mem_exhaustion.py                # -> node_mem_exhaustion_eta_seconds, node_mem_psi_* (if not native-enough)
      cpu_nonbuild.py                  # -> node_cpu_nonbuild_busy_percent, node_cpu_build_busy_percent
      tmpfs_junk.py                    # -> node_tmp_stale_bytes, node_tmp_used_percent, node_tmp_inodes_percent
      backup_age.py                    # -> node_backup_age_seconds (guarded: emit only if a backup path/config exists; else no-op)
      smartmon.sh                      # node_exporter upstream smartmon example (SMART -> textfile); bridges smartd
    systemd/                           # timers that run each producer into the textfile dir
      node-textfile@.service node-textfile@.timer   # templated: %i = producer name
      (or one .service+.timer per producer — Codex picks the cleaner idempotent form)
    prometheus/rules/system-monitor.rules.yml        # recording rules if useful (e.g. fill predict); optional
    provisioning/
      dashboards/system-monitor.yaml   # dashboard provider -> points at dashboards dir
      dashboards/json/overview.json    # THE dashboard (see §5)
      alerting/system-monitor.yaml     # provisioned unified-alerting rule group (see §4)
      alerting/contactpoints.yaml      # a contact point (default; desktop-notify wiring deferred to sidecar pass)
  README-grafana.md                    # what each panel/alert is, how to tune, how to import
```

Producers are pure-measurement, stdlib-only python3, side-effect-free, `try/except` tolerant of missing files, write ATOMICALLY (`*.prom.$$` then rename) into `/var/lib/node_exporter/textfile/`. Each writes a `# HELP`/`# TYPE` header per metric. Timer cadence: mem/cpu 15s, tmpfs 30s, smartmon 5m, backup 1h.

## 4. Alert rules (provisioned unified alerting; thresholds tunable in the yaml)

Each rule: expr (PromQL), `for` duration, severity label, summary naming the offending instance/unit/device. Group eval interval 30s.

| uid | expr | for | sev |
|---|---|---|---|
| mem_avail_low | `node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 8` | 1m | crit (`<15` warn as 2nd rule) |
| mem_exhaustion_eta | `node_mem_exhaustion_eta_seconds < 90` | 0s | crit (`<360` warn) |
| mem_psi_high | `rate(node_pressure_memory_stalled_seconds_total[1m]) > 0.20` | 1m | crit |
| io_pressure_thrash | `rate(node_pressure_io_stalled_seconds_total[1m]) > 0.30` | 1m | warn |
| swap_near_full | `node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes * 100 < 10` | 2m | warn |
| disk_fill_predict | `predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay|squashfs"}[1h], 3600) < 0` | 5m | crit (4h-horizon `[1h],14400` warn) |
| disk_used_high | `(1 - node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 > 90` (same fstype filter) | 5m | warn |
| inodes_low | `node_filesystem_files_free / node_filesystem_files * 100 < 10` (same filter, files>0) | 5m | warn |
| tmpfs_junk | `node_tmp_used_percent > 85 or node_tmp_inodes_percent > 85` | 1m | warn (`>95` crit) |
| cpu_runaway_nonbuild | `node_cpu_nonbuild_busy_percent > 85` | 5m | warn |
| fd_exhaustion | `node_filefd_allocated / node_filefd_maximum * 100 > 80` | 2m | warn |
| proc_explosion | `sum(node_processes_threads) > 12000` | 2m | warn (tune) |
| unit_failed | `node_systemd_unit_state{state="failed"} == 1` | 0s | warn (label = unit name) |
| oom_kill | `increase(node_vmstat_oom_kill[5m]) > 0` | 0s | crit |
| temp_high | `node_hwmon_temp_celsius > 90 or node_thermal_zone_temp > 90` | 1m | warn (`>95` crit) |
| smart_unhealthy | `node_smartmon_device_smart_healthy == 0` | 0s | crit |
| smart_realloc | `node_smartmon_reallocated_sector_ct_raw_value > 0` | 0s | warn |
| backup_stale | `node_backup_age_seconds > 604800` | 0s | warn (only fires if metric present) |

Verify every custom metric name in a rule matches EXACTLY what the §3 producers emit. `node_smartmon_*` come from the upstream smartmon script — use its real metric names; if they differ, correct the rule expr to match.

## 5. Dashboard (`overview.json`) — clean, at-a-glance layout

Grafana schema, dark, refresh 30s, time range last 3h. Rows top→bottom, each a collapsible row:

1. **OVERVIEW** — a row of `stat` panels w/ color thresholds (green/yellow/red): Mem available %, Swap used %, CPU busy %, Load1, Root FS free %, /tmp used %, Max temp °C, Failed units (count), Uptime. Big, glanceable, is-the-box-ok in one look.
2. **MEMORY & PRESSURE** — timeseries: mem used/avail/cached + swap used; PSI mem/cpu/io as % (`rate(...stalled...[1m])*100`); a `stat`/timeseries for exhaustion ETA (seconds, with the 999999 sentinel HIDDEN or clamped so autoscale isn't destroyed).
3. **CPU** — per-core busy (`irate(node_cpu_seconds_total{mode!="idle"}[1m])`) + total; a timeseries overlaying total busy vs `node_cpu_nonbuild_busy_percent` vs build-busy; load 1/5/15.
4. **DISK & FILESYSTEM** — per-mount used % (bar/timeseries), inodes used %, disk fill prediction, disk IO (`node_disk_io_time_seconds_total` rate), read/write bytes.
5. **tmpfs /tmp** — used %, inodes %, stale-junk bytes timeseries.
6. **PROCESSES & FD** — total procs, threads, `node_filefd_allocated` vs maximum (%).
7. **NETWORK** — rx/tx throughput per iface, errors + drops.
8. **HEALTH** — temps (all hwmon+thermal sensors), a `table` of SMART per device (health + reallocated + power-on-hours), a `table` of systemd units in `failed` state, OOM-kill counter (`node_vmstat_oom_kill`).

Every panel: correct unit (bytes→bytes, percent→percent 0-100, °C), sensible legend, threshold coloring on the danger metrics. No panel may reference a metric that does not exist (native list in §1 + custom in §3). Set `uid` stable so re-provision updates in place.

## 6. install-dashboards.sh contract (root, idempotent, fail-closed)

1. Add `--collector.processes` + `--collector.textfile.directory=/var/lib/node_exporter/textfile` to `/etc/default/prometheus-node-exporter` ARGS (idempotent: don't duplicate flags). `mkdir -p` the textfile dir, owned so node_exporter (user `prometheus-node-exporter` or `node-exp`—detect actual) can read and the producers can write (dir writable by producers' user). Restart node_exporter.
2. Install producers to `/usr/local/lib/system-monitor/`, timers+services to `/etc/systemd/system/`, enable+start the timers. First run must populate `.prom` files (start the services once).
3. If SMART: ensure `smartmontools` present (it is — smartd active); install smartmon.sh; its timer writes SMART textfile.
4. Deploy Prometheus rules (if any) to `/etc/prometheus/` + reload prometheus (`--web.enable-lifecycle` is on: POST `http://127.0.0.1:9090/-/reload`).
5. Deploy grafana provisioning (`dashboards/`, `alerting/`, dashboard JSON) into `/etc/grafana/provisioning/...`; `chown -R grafana:grafana`; restart grafana-server.
6. Verify: node_exporter serves each custom metric (`curl -s 127.0.0.1:9100/metrics | grep node_mem_exhaustion_eta_seconds` etc.), prometheus rules loaded (`/api/v1/rules`), grafana dashboard present (`/api/dashboards/uid/<uid>` → 200), no provisioning error in `journalctl -u grafana-server`. Print a checklist.
7. Runs SEPARATELY by the user via gui-askpass sudo. Codex executes nothing privileged.

## 7. Acceptance (Codex satisfy all)

- [ ] node_exporter reconfigured (processes + textfile dir), restarted, still 127.0.0.1 only.
- [ ] 4 custom producers + smartmon on timers, writing atomic `.prom`; every metric named in a §4 rule / §5 panel is actually emitted (or native). Producers REUSE existing repo logic, not rewritten math.
- [ ] `overview.json` provisions clean, all 8 rows, correct units + thresholds, no dangling metric refs, stable uid.
- [ ] All §4 alert rules provisioned, expr valid PromQL against real metric names, severity + naming labels set.
- [ ] install-dashboards.sh idempotent, fail-closed, only privileged actor; verify-step present.
- [ ] grafana.ini localhost + phone-home-off untouched; nothing exposed beyond 127.0.0.1.
- [ ] README-grafana.md maps every panel + alert to its metric + how to tune.
- [ ] Python producers: `python3 -m py_compile` clean; a dry-run of each prints valid textfile format to stdout.
```