# SPEC — Phase 2: make the guard the useful layer

audience: AI coding agents first. Optimize for model activation, not prose.

## Goal (BLUF)

Turn the passive metric stack into an **active desktop guard**. Silent while healthy; on a real
firing alert → **Cinnamon desktop notification** with **[Info] [Kill|Clean]** buttons. Grafana +
Prometheus demoted to investigation-only (open when curious). This fixes "overwhelming + useless":
day-to-day the user sees NOTHING until a real popup.

Wiring: **Grafana evaluates the 23 rules → webhook → guard → desktop popup.** Single source of
truth for thresholds stays in Grafana. Guard stays thin: receive + notify + remediate + native
event-watch.

## READ FIRST (do NOT assume — the guard skeleton already exists)

Read and REUSE these; extend, never rebuild:
- `agent-guard/src/agent_guard/daemon.py` — ingress (`socket_server`), main loop (cooldown-gated
  `queue`), `handle_event`, `handle_action`. Event threads: journald/fswatch/ports.
- `agent-guard/src/agent_guard/notifier.py` — `gdbus_notify`, `notify_event`, `action_monitor`
  (gdbus `ActionInvoked` → callback), `clean_tmp_safe`.
- `agent-guard/src/agent_guard/events/__init__.py` — `Event` dataclass.
- `agent-guard/src/agent_guard/config.py`, `culprit.py`, `state.py`.
- `grafana/provisioning/alerting/{system-monitor.yaml,contactpoints.yaml}` — the 23 rules + contact points.
- `grafana/install-dashboards.sh` — the idempotent installer to extend.

Environment (verified): Debian forky, **Cinnamon** (X-Cinnamon) notification daemon — closes a
notification when ANY action is clicked (does NOT honor `resident`). `notify-send`+`gdbus` present.
Grafana OSS 13.1.0 localhost:3000, Prometheus localhost:9090.

## Deltas (each pins a seam; implement the body)

### D1 — new ingress: Grafana webhook receiver
- NEW `agent-guard/src/agent_guard/events/webhook.py`: `http.server`-based thread, bind
  **127.0.0.1:9099** ONLY (localhost, never 0.0.0.0). Accept `POST /` with Grafana's webhook JSON.
- Grafana webhook payload = `{"alerts": [ {status, labels{}, annotations{}, dashboardURL,
  panelURL, generatorURL, silenceURL, valueString}, ... ], ...}`. For each alert with
  `status == "firing"`, map → `Event` and put on the SAME `queue` the daemon already drains.
- Mapping (exact):
  - `tier` = `labels.severity` → `"critical"` if `critical` else `"warning"`.
  - `source` = `"grafana"`.
  - `reason` = `annotations.summary` (fallback `labels.alertname` + `valueString`).
  - `info_url` = first non-empty of `panelURL`, `dashboardURL`, `generatorURL` (see D3).
  - `remediation` = `labels.remediation` ∈ {`kill`,`clean`,`none`} (default `none`) (see D2).
- Ignore `status == "resolved"` posts (no popup on recovery; keep quiet). Malformed body →
  `journal_send(... SM_ACTION="bad_webhook")`, HTTP 400, never crash the thread.
- Wire the thread into `daemon.main()` threads list.

### D2 — Event model + two-button notification with Info re-raise
- Extend `Event` dataclass: add `info_url: str | None = None` and `remediation: str = "none"`.
  Keep existing fields/order-compatibility (existing socket + event threads must still construct it).
- Buttons per event (in `handle_event`):
  - `remediation == "kill"` → actions `["Info", "Kill <culprit>"]` (reuse `rescan_culprit` to name it).
  - `remediation == "clean"` → actions `["Info", "Clean /tmp"]`.
  - else → actions `["Info", "Dismiss"]`.
- `gdbus_notify`: add `replaces_id: int = 0` param (currently hardcoded `"0"`); pass it through so a
  re-raise can update in place.
- **Info re-raise (daemon-agnostic, REQUIRED shape):** in `handle_action`, when `action`
  startswith `info` → `xdg-open` (or `gio open`) `event.info_url`, THEN re-call `notify_event(event)`
  so the popup reappears with BOTH buttons intact. Rationale: Cinnamon closes the notification on any
  action click; re-raising is the only reliable way to keep Kill/Clean available after Info. Do NOT
  rely on the `resident` hint.
- `Kill <culprit>` / `Clean /tmp` actions: existing `handle_action` kill/clean branches already
  cover this — ensure the culprit rescan trigger (currently `event.source == "netdata"`) also fires
  for a `kill`-remediation grafana event. Change the gate to remediation-driven, not source-string.

### D3 — Grafana provisioning: route rules → webhook + deep-links + button metadata
- `contactpoints.yaml`: ADD a webhook contact point `name: agent-guard` →
  `url: http://127.0.0.1:9099/`, `httpMethod: POST`. Keep it localhost.
- ADD a notification policy (provisioned) routing ALL `system-monitor` rules to contact point
  `agent-guard` (match on a shared label, e.g. `stack=system-monitor`). Do NOT route to the default
  contact point (avoid the unconfigured-email noise).
- On EACH of the 23 rules in `system-monitor.yaml`:
  - add label `stack: system-monitor` (routing key).
  - add label `remediation:` = `kill` (mem-exhaustion, cpu-runaway, load), `clean` (tmpfs junk,
    disk-fill on `/tmp` or a stray-junk fill), else `none`.
  - add label `severity:` = `critical` | `warning` (map from the existing warn/crit split).
  - add annotation `__dashboardUid__: system-monitor-overview` + `__panelId__: <the panel showing
    this metric>` so Grafana emits a real `panelURL`/`dashboardURL` for the Info button. Where no
    single panel fits, point at the overview dashboard (panel omitted).
- Extend `install-dashboards.sh` to install the guard (systemd **user** service `agent-guard`,
  already at `agent-guard/systemd/agent-guard.service`) and restart it. Guard runs as the LOGIN USER
  (needs the user session D-Bus for notifications) — install to `~/.config/systemd/user/`, enable via
  `systemctl --user`. NO sudo for the guard itself; only Grafana provisioning files need root.

### D4 — tests (extend `agent-guard/tests/`)
- webhook JSON (firing) → correct `Event` (tier/reason/info_url/remediation) on queue.
- `status: resolved` → no event.
- malformed body → 400 + no crash.
- `handle_action("info", ...)` → info_url opened AND `notify_event` re-called (mock both).
- kill-remediation event → culprit rescan fires; clean-remediation → `clean_tmp_safe` called.
- localhost bind assertion (receiver never binds non-loopback).

## Constraints (HARD)
- Webhook + all binds **127.0.0.1 only**. Never 0.0.0.0.
- Guard = login-user systemd `--user` service (D-Bus session access). Grafana files = root via the
  installer the USER runs (`sudo -A`). Codex builds files only — NEVER runs sudo/systemctl/apt or
  writes to `/etc`,`/usr`,`/var`.
- Reuse cooldown/state/protect-list/clean_tmp_safe AS-IS. No new alert logic in the guard — Grafana
  owns thresholds.
- Keep the native event watchers (journald/fswatch/ports) — they cover what Grafana can't
  (kernel errors, fs-remount-ro, new ports). They already emit Events; leave them.
- No stubs. Full functionality. Fail-closed. Idempotent installer.