# Grafana/Prometheus Provisioning

Run privileged install separately:

```bash
sudo -A bash grafana/install-dashboards.sh
```

The installer keeps node_exporter, Prometheus, and Grafana bound to 127.0.0.1. It adds `--collector.processes`, configures the textfile directory, installs producers/timers, deploys recording rules, provisions Grafana, then verifies metrics, rules, dashboard UID, and Grafana provisioning logs. It also fails if any textfile `.prom` reaching Prometheus is older than 2h — a stale file still exports an mtime, so presence alone would not catch a producer that stopped writing.

Retention is owned by `install-prometheus-grafana.sh`, not by this installer: `--storage.tsdb.retention.time=30d` with `--storage.tsdb.retention.size=2GiB` as the disk-fill ceiling (measured 2026-08-08: 26 MB/day, so 30d ≈ 780 MB). Both live in `systemd/system/prometheus.service`; `deckctl` copies that unit but only `daemon-reload`s, so the flags apply on the restart `install-prometheus-grafana.sh` performs, and it reads them back off `/api/v1/status/flags`.

## Textfile Producers

| Producer | Cadence | Metrics | Logic source |
|---|---:|---|---|
| `mem_exhaustion.py` | 15s | `node_mem_exhaustion_eta_seconds`, `node_mem_runway_percent`, `node_mem_psi_full_avg10_percent` | `netdata/plugins/mem_trajectory.plugin` |
| `cpu_nonbuild.py` | 15s | `node_cpu_nonbuild_busy_percent`, `node_cpu_build_busy_percent` | `netdata/plugins/cpu_runaway.plugin` and agent-guard build allowlist |
| `tmpfs_junk.py` | 30s | `node_tmp_stale_bytes`, `node_tmp_used_percent`, `node_tmp_inodes_percent` | `netdata/plugins/tmpfs_guard.plugin` |
| `backup_age.py` | 1h | `node_backup_age_seconds` when `SM_BACKUP_PATHS` or `/etc/system-monitor/backup-paths` exists | guarded backup path scan |
| `smartmon.sh` | 5m | `node_smartmon_device_smart_healthy`, `node_smartmon_reallocated_sector_ct_raw_value`, `node_smartmon_power_on_hours_raw_value`, `node_smartmon_temperature_celsius` | node_exporter smartmon textfile pattern |

Python producers support `--stdout` for dry runs and write `.prom` files atomically when run by systemd.

## Dashboard Panels

| Row | Panel | Metrics |
|---|---|---|
| OVERVIEW | Mem available %, Swap used %, CPU busy %, Load1, Root FS free %, /tmp used %, Max temp C, Failed units, Uptime | `node_memory_*`, `node_cpu_seconds_total`, `node_load1`, `node_filesystem_*`, `node_tmp_used_percent`, `node_hwmon_temp_celsius`, `node_thermal_zone_temp`, `node_systemd_unit_state`, `node_boot_time_seconds` |
| MEMORY & PRESSURE | Memory and swap, PSI stalled %, Exhaustion ETA | `node_memory_*`, `node_pressure_{memory,cpu,io}_stalled_seconds_total`, `node_mem_exhaustion_eta_seconds` |
| CPU | Per-core busy %, Total vs build classification, Load average | `node_cpu_seconds_total`, `node_cpu_nonbuild_busy_percent`, `node_cpu_build_busy_percent`, `node_load1`, `node_load5`, `node_load15` |
| DISK & FILESYSTEM | Filesystem used %, Inodes used %, Fill prediction, Disk IO | `node_filesystem_*`, `node_disk_io_time_seconds_total`, `node_disk_read_bytes_total`, `node_disk_written_bytes_total` |
| tmpfs /tmp | /tmp pressure, Stale junk bytes | `node_tmp_used_percent`, `node_tmp_inodes_percent`, `node_tmp_stale_bytes` |
| PROCESSES & FD | Processes and threads, File descriptors | `node_processes_state`, `node_processes_threads`, `node_filefd_allocated`, `node_filefd_maximum` |
| NETWORK | Throughput, Errors and drops | `node_network_receive_bytes_total`, `node_network_transmit_bytes_total`, `node_network_receive_errs_total`, `node_network_transmit_errs_total`, `node_network_receive_drop_total`, `node_network_transmit_drop_total` |
| HEALTH | Temperatures, SMART, Failed systemd units, OOM kills | `node_hwmon_temp_celsius`, `node_thermal_zone_temp`, `node_smartmon_*`, `node_systemd_unit_state`, `node_vmstat_oom_kill` |

Dashboard UID is `system-monitor-overview`; provisioning updates it in place.

## Alert Rules

| Rule | Severity | Metric/expression | Tune in |
|---|---|---|---|
| `mem_avail_low` | warn/crit | `node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 15/<8` | `grafana/provisioning/alerting/system-monitor.yaml` |
| `mem_exhaustion_eta` | warn/crit | `node_mem_exhaustion_eta_seconds < 360/<90` | alert yaml |
| `mem_psi_high` | crit | `rate(node_pressure_memory_stalled_seconds_total[1m]) > 0.20` | alert yaml |
| `io_pressure_thrash` | warn | `rate(node_pressure_io_stalled_seconds_total[1m]) > 0.30` | alert yaml |
| `swap_near_full` | warn | `node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes * 100 < 10` | alert yaml |
| `disk_fill_predict` | warn/crit | `predict_linear(node_filesystem_avail_bytes[1h], 14400/3600) < 0` | alert yaml |
| `disk_used_high` | warn | filesystem used percent `> 90` | alert yaml |
| `inodes_low` | warn | filesystem free inodes percent `< 10` | alert yaml |
| `tmpfs_junk` | warn/crit | `node_tmp_used_percent` or `node_tmp_inodes_percent > 85/>95` | alert yaml |
| `cpu_runaway_nonbuild` | warn | `node_cpu_nonbuild_busy_percent > 85` | alert yaml |
| `fd_exhaustion` | warn | `node_filefd_allocated / node_filefd_maximum * 100 > 80` | alert yaml |
| `proc_explosion` | warn | `sum(node_processes_threads) > 12000` | alert yaml |
| `unit_failed` | warn | `node_systemd_unit_state{state="failed"} == 1` | alert yaml |
| `oom_kill` | crit | `increase(node_vmstat_oom_kill[5m]) > 0` | alert yaml |
| `temp_high` | warn/crit | `node_hwmon_temp_celsius` or `node_thermal_zone_temp > 90/>95` | alert yaml |
| `smart_unhealthy` | crit | `node_smartmon_device_smart_healthy == 0` | alert yaml |
| `smart_realloc` | warn | `node_smartmon_reallocated_sector_ct_raw_value > 0` | alert yaml |
| `backup_stale` | warn | `node_backup_age_seconds > 604800` | alert yaml |

## Tuning Notes

Timer cadence is installed through systemd drop-ins in `grafana/install-dashboards.sh`. For build CPU classification, set `SM_BUILD_ALLOWLIST` as a comma-separated list in the service environment if the agent-guard defaults need local additions. For backup age, put one path per line in `/etc/system-monitor/backup-paths` or set `SM_BACKUP_PATHS=/path/a:/path/b`; with no configured path, no backup metric is emitted and the alert stays absent.