# Overdeck observability — activity store, Logs page, activity charts

audience: AI coding agents first.

Slug: `overdeck-observability`. Date: 2026-08-08.

## 1. Problem

Owner statement: *"i have NO IDEA what happened, or if anything happened at all"* and *"currently system is blind"*.

Measured today: 47 notification sources suppressed with **no per-attempt record anywhere**; harness/factory runs, cdx/cursor/claude runs, lands, gates, deploys, buildbox jobs and guards each write to a different place in a different format; five of the highest-value sources write **no trail at all**. There is no range-query read path over anything.

Notifications are explicitly rejected as the fix — owner: *"notification spam is just harrassing me, but having a log file could be useful to analyze when i'm up for it."* This surface is **pull, never push**.

## 2. Goals / non-goals

**Goals**
- One durable, append-only, range-queryable activity store spanning every Overdeck-hosted activity.
- A Logs page: categorized, sortable, searchable, filterable, correlated into episodes.
- Activity charts over `1h / 1d / 3d / 7d / 30d / all`, per-project selectable lines, cross-filtered with the table.
- Honest data: a series with no coverage renders "no data before `<date>`", never `0`.

**Non-goals**
- No new desktop notification of any kind. Default-deny gate stays; this page replaces the need.
- No ingestion of transcript *content* (7.1 GB codex + 1.55 GB cursor + 556 MB claude). Metadata only.
- Not a replacement for Prometheus host metrics. Split defined in §4.4.

## 3. Findings that constrain the design

These are measured (2026-08-07/08), not assumed. Each one kills a naive design.

1. **`~/.overdeck/items.jsonl` cannot be the activity source.** 162,125 lines / 38.2 MB over 37.2 days, never rotated, and schema-compatible — but `append()` dedupes unchanged serialization, so it is a **state-change** log, not an activity log. Distribution is brutally skewed (systray-ai 69,064 · botmaster 31,918 · prometheus 18,013 · ghci 17,706 · offload 12,204 · gates 6,255 · harness 2,262 · factory 11 · permissions 5). Charting "activity" from it measures adapter churn.
2. **Its read path, not its storage, is the missing piece.** `Journal.load()` collapses the file to `Map<id,Item>` at boot; `GET /items` serves only that snapshot. 162k appended lines are on disk and unreachable.
3. **Ragged coverage floor.** Per-source earliest real data: codex 2025-09-19 · claude history 2025-09-30 · claude transcripts 2025-12-31 · cursor 2026-04-22 · items.jsonl 2026-07-01 · runstate + reaper 2026-07-17 · deploy reflog 2026-07-19 · controller 2026-07-21 · journald 2026-08-03 · factory sssf 2026-08-05 · agent-sessions ledger + landq 2026-08-07. An all-time chart therefore shows agent activity to Sep 2025 and infra activity only to Jul 2026.
4. **Prometheus cannot serve this page.** 15d default retention (no `--storage.tsdb.retention.time`) can't answer 30d or all-time; per-project/per-session selectable lines are the high-cardinality antipattern; its textfile feed has been dead since 2026-07-12 with 0 custom metric families.
5. **Suppressed-notification history has a hard ceiling.** `~/.local/state/notif-gate/pending.json` is a whole-file-rewritten per-source aggregate `{program, summary_template, count, first, last}`. 47 sources / 251 attempts, window 2026-08-07 12:09→23:51 only. Per-attempt records do not exist. The page must present "N sources tried, M times, last at T" and MUST NOT imply a per-attempt timeline.
6. **Two working patterns already exist in-repo** and the store is modeled on them, not invented: `collector/src/journal.ts` (append-only JSONL + typed records) and `~/.config/overdeck/controller/events.jsonl` (3,225 lines, `{ts, job, repo, host, snapshot, attempt, stage, reason, rc, durationSeconds}` — the best dispatch history on the box).
7. **Sources with no trail at all** need an *emitter* before they can ever appear: `deploy-local.sh`, cpu-guard, buildslot admission, stall-guard, local-gate (14,233 lock/epoch files, zero events), claudex/dyn-wf, gate runs, botmaster-proxy (in-memory only).

## 4. Architecture

### 4.1 Shape

```
 emitters (new, §4.5) ─┐
 shippers  (§4.6) ─────┼──▶  activity store  ──▶  rollups  ──▶  read API  ──▶  Logs page
 existing JSONL ───────┘      (append-only)       (§4.3)         (§4.7)        + charts
```

One store, many small writers, rollups for range queries, every view a read over the same place. Adding a category later is one shipper and zero UI work.

**Build order is fixed by that property: the page ships before the sources do.** Wave 1 = store + read API + `/logs` rendering from the sources that are ALREADY append-only with timestamps — controller `events.jsonl`, runstate run logs, landq, reaper — plus the account gauge (§4.6a), whose producer already exists. Every later emitter and shipper is additive and lights up its own rows on arrival. **Do NOT sequence all writers ahead of the first render**; the owner has spent this week waiting on work that was complete but not visible, and a ten-shipper prerequisite chain reproduces exactly that.

### 4.2 Event record — the central seam

```
ActivityEvent
  id          string        ULID (sortable by time, collision-free across writers)
  ts          string        ISO-8601 with offset
  category    Category      closed enum, §4.8
  source      string        stable emitter id, e.g. "controller", "landq", "reaper"
  actor       Actor         "agent" | "human" | "timer" | "system"
  runtime     string?       normalized CLI: claude | codex | cursor-agent | runplan | opencode | …
  project     string?       repo/project key; null when genuinely host-scoped
  host        string?       laptop | debian1 | debian2 | debian3
  session     string?       session/run/job id — the correlation key within a runtime
  episode     string?       correlation id, §4.9
  severity    Severity      "debug" | "info" | "notice" | "warn" | "error"
  title       string        one line, human-readable, no interpolated volatile ids
  detail      object?       arbitrary JSON, source-specific
  dedupe_key  string?       identical repeats collapse on this, §4.10
  duration_ms number?       for completed spans
  rc          number?       exit code where meaningful
```

`runtime` is deliberately lifted from `~/.local/state/agent-sessions/sessions/*.json` — the only normalized cross-CLI runtime field that exists on this box (codex 434 · claude 211 · cursor-agent 111 · runplan 5 · grok 5 · opencode 2 · kiro-cli 1 · agy 1).

### 4.3 Storage and rollups

Store: SQLite in the collector's existing state dir (`OVERDECK_STATE_DIR`, `~/.local/state/overdeck`). **`OVERDECK_STATE_DIR` is canonical for this store** — `~/.overdeck/` holds the older journal and factory db and is NOT extended here. Every emitter and shipper resolves the path from that env var with that default; none hardcodes a second location. Table `activity_events` on the shape above.

Indexes: `(ts)`, `(category, ts)`, `(project, ts)`, `(episode)`, `(dedupe_key, ts)`.

Three record kinds, deliberately separate — a count table cannot answer a concurrency or a percentage question:

| table | kind | answers |
|---|---|---|
| `activity_events` | discrete event, point in time | "what happened", counts, rates |
| `activity_sessions` | **interval** — `(id, runtime, project, host, account, started_at, ended_at?, turns?, tool_calls?, tokens_in?, tokens_out?)` | "how many agents were active at time T" |
| `activity_gauge` | **sample** — `(ts, metric, subject, value, unit, meta?)` | "what was the 7d usage percentage over time" |

`activity_sessions` rows are opened on session start and closed on end; a row with `ended_at IS NULL` older than the runtime's stall horizon is rendered `unknown end`, never silently closed at now(). **Active-agent concurrency is an interval-overlap read** (`started_at < bucket_end AND (ended_at IS NULL OR ended_at > bucket_start)`), NOT a count of start events — counting starts is the wrong answer and MUST NOT be implemented.

**"Log lines generated"** means rows written to `activity_events`, per source — an honest measure of this system's own intake. It does NOT mean transcript lines; transcripts are indexed as metadata only (§4.6) and their line counts are not collected.

Rollups — this is what makes `all` answerable instead of aspirational:

| tier | grain | horizon | serves |
|---|---|---|---|
| raw | per event | 7d | `1h`, `1d`, `3d`, table detail |
| `activity_rollup_min` | 1 minute | 30d | `7d`, `30d` |
| `activity_rollup_hour` | 1 hour | unbounded | `30d`, `all` |

Rollup rows key on `(bucket_ts, category, project, runtime, host, severity)` with a `count`, plus `sum_duration_ms`/`error_count`. Charts read rollups exclusively; the table reads raw within 7d and rollups beyond.

Sessions and gauges roll up on their own semantics, never as counts: `activity_sessions` → per-bucket `max_concurrent` / `distinct_sessions` from the overlap read; `activity_gauge` → per-bucket `last`, `max`, `min` per `(metric, subject)`, because a percentage is a level and averaging it across a bucket hides the peak that mattered.

Rollup is derived state — droppable and rebuildable from raw plus the source files. Retention of raw beyond 7d is a prune job, not a delete-on-write.

### 4.4 Prometheus split

Prometheus keeps host/system metrics (CPU, memory, disk, load). This store keeps discrete activity with identity and text. The Logs page reads Prometheus **only** for the load overlay in §5.4, via its HTTP API — it never writes to it.

Blocked on #86 (textfile feed frozen since 2026-07-12, retention unset). The overlay must degrade to "no data" if Prometheus is unavailable, never to a flat line.

### 4.5 Emitters (new — sources in §3.7 that log nothing)

Each is a one-line append at the existing decision point, not a rework:

- `deploy-local.sh` — emit its existing stdout JSON (`stage`, `status`, `sha`) as an event.
- `local-gate` / `buildslot` — admission decision: admitted / queued / refused-97, with wait time.
- `cpu-guard` — classification decision (heavy/local-only/dispatched) and target host.
- `gate runs` — start/end, mode (`strict`), local vs remote, host, duration, pass/fail.
- `stall-guard` — transitions only, never per-probe.
- `claudex/dyn-wf` — inferred from the underlying CLI store; `~/.claudex/proxy.log` is a 14-line static banner and unusable.
- `botmaster-proxy` — flush in-memory history to the store; it currently writes nothing to disk.

Emitters MUST be fire-and-forget and MUST NOT block or fail their caller. A store write failure is dropped with a counter, never propagated — an observability layer must not become an outage.

### 4.6 Shippers (existing append-only sources)

One shipper per source in §A of the inventory, each keeping a `(inode, offset)` cursor and resuming — the same incremental-read pattern already proven in `usage_agg.py`, including full reparse on inode change and never consuming a partial trailing line.

`state.json`-style sources (§B) get a **sampler** at their natural interval; they cannot produce history retroactively and each MUST record its first-sample date so §4.11 renders honestly.

Samplers are **edge-triggered — they emit an `activity_events` row ONLY when the observed state differs from the last sample.** Four sources polled at ~10min would otherwise write ~576 no-change rows/day each and drown every activity chart in their own polling. Liveness is carried by a separate per-sampler `last_sampled_at` field (a heartbeat, read by §4.11 to distinguish "nothing happened" from "sampler dead") — the heartbeat is NOT an activity event and MUST NOT appear in counts or the table.

Continuously-varying levels (usage percentages, §4.6a) go to `activity_gauge` on every sample instead — that IS the signal, and it is never counted as activity.

**Monotonic counter sources emit the DELTA between consecutive samples, never the running total.** `notif-gate/pending.json` is whole-file-rewritten and carries a cumulative attempt count (251 at time of survey); shipping the total would make the §5.2 digest re-report every past attempt on every poll. A counter that decreases (reset/rotation) is treated as a restart: emit the new value as the delta, never a negative.

#### 4.6a Account usage gauge — per-account limit percentage over time

Owner request, 2026-08-08: *"charts of account usage, with the 7d limit percentage over time, as one of the charts. with each account has a line on the chart"*.

The data is **already polled and already thrown away**. `modules/systray/` health clients (`claude_health_client.py`, `codex_appserver.py`, `grok_health_client.py`) return, per account, a snapshot carrying `primary_used_pct` (5h window), `secondary_used_pct` (7d window), and each window's `resets_at`, keyed by `account.tray_key`. `quota_notification_store.py` already keys its records `(account_key, window)`. Nothing persists a time series — the systray renders the current number and drops it.

The gauge shipper subscribes to the same snapshot the systray already fetches and writes to `activity_gauge`:

```
metric   "limit_used_pct"
subject  "<provider>:<account_key>:<window>"   window ∈ {"5h","7d"}
value    0–100 integer
unit     "percent"
meta     { resets_at, provider, account_key, window }
```

Contract rules — all load-bearing:
- **Sample-on-poll, not on change** — a level is the signal; §4.6's edge-trigger rule applies to events, never to gauges.
- **A window reset is a discontinuity, never a downward slope.** `resets_at` moving forward starts a new segment; the chart MUST break the line at the reset boundary rather than draw a fall from 96% to 3%, which would read as usage being refunded.
- **A missing poll is a gap, not a hold.** No sample ⇒ no point. Never carry the last value forward across a gap; §4.11 applies.
- **No account is invented and none is aggregated away** — one series per `(provider, account_key)`, even for accounts at 0%.
- Absolute token/cost totals stay in `activity_sessions` (§4.3); this gauge is percentage-of-limit only, because that is the number that actually stops work.

Transcripts (claude / codex / cursor): index **metadata only** — runtime, project, session, start/end, turn count, tool-call count, token counts. Never content. ~40 MB/day combined across the three would otherwise dominate the store.

### 4.7 Read API

```
GET /api/activity            filters: from, to, category[], project[], runtime[], host[],
                             severity>=, actor[], q (FTS over title), episode
                             → { events[], total, coverage }
GET /api/activity/series     params: range, bucket, groupBy=project|category|runtime|host
                             → { series[{key, points[{t, v}], coverageFrom}], bucket }
GET /api/activity/concurrency params: range, bucket, groupBy=runtime|project|host
                             → { series[…], bucket }   // interval-overlap, §4.3 — NOT event counts
GET /api/activity/gauge      params: range, bucket, metric, subject[]
                             → { series[{key, points[{t, v, break?, projected?}],
                                 coverageFrom, meta}] }
GET /api/activity/episodes   → correlated episodes, §4.9
GET /api/activity/digest     params: since  → §5.2
```

`coverage` / `coverageFrom` are mandatory in every response — they carry §3.3 to the UI so a series can say "no data before X".

### 4.8 Categories

Closed enum, one per event: `land` · `gate` · `deploy` · `run` (harness/factory/run-plan) · `agent` (CLI sessions) · `buildbox` · `guard` · `notification` · `git` · `service` · `ci`.

Owner asked for "several log categories" — these are the tabs/facets.

### 4.9 Episodes — the idea that stops this being a data dump

A land is six sources today: branch merged → gate run → remote dispatch to a box → push → deploy → convergence re-probe. It is **one story**.

Correlation rules, first match wins:
1. explicit `episode` set by the emitter (preferred — emitters in §4.5 propagate it),
2. shared `session`/`runId` within a time window,
3. shared `project` + branch/sha within a window.

Rule 3 is heuristic and MUST be marked as inferred in the UI, never presented as certain.

Episodes collapse to one row, expand to the trace. `SwimlaneTrace` and `TraceSegment` already exist in `deck-ui` for the expanded view.

### 4.10 Dedupe

**Dedupe is READ-TIME, exactly once, and never at write time.** Every occurrence is stored as its own raw row carrying `dedupe_key`; the table and the digest group on `dedupe_key` and render one line with `count`, `first`, `last`.

Write-time collapsing is explicitly rejected: it would attribute all 190 fswatch occurrences to the first bucket, understating activity in every chart for every later bucket and making the raw store lie about when things happened. Raw stays honest; rollup counts stay truthful; only the presentation collapses.

Load-bearing case: the fswatch alert fired **190 times**. The table shows one line saying 190, never 190 lines — while the chart still shows 190 events spread across the hours they actually occurred. Same mechanism serves the notification aggregate in §3.5, whose source data is already in this shape.

### 4.11 Honest data law

Repo rule, restated because it is the easiest thing to get wrong here: never fabricate a value. A series with no coverage in a range renders `no data before <date>`; a dead source renders `no data`; a source that is current-state-only renders its first-sample date. Zero is a measurement and MUST NOT be used as a stand-in for absence.

## 5. UI

Route: `/logs`. Charts and table on one page, cross-filtered.

MANDATORY: obey `.claude/skills/od-ui-dev/SKILL.md`. Compose `@overdeck/deck-ui` exports — `DeckTable`, `FilterInput`, `StatusChip`, `SectionCard`, `KpiTile`, `DetailDrawer`, `SwimlaneTrace`, `TraceSegment`, `ProjectGroupRow`, `LiveDuration`, `project-colors`. No hand-rolled tables/chips/menus/tooltips.

### 5.1 New primitives — owner-approved

visx approved by the owner on 2026-08-08 as the charting foundation (chosen over ECharts and Recharts specifically because it is unstyled, so design tokens and both themes apply directly).

```
TimeSeriesChart({ series: Series[], range: Range, bucket: Bucket,
                  stacked?: boolean, yDomain?: [number, number],
                  yUnit?: "count" | "percent" | "ms",
                  threshold?: { value: number, label: string },
                  onBrush?: (from, to) => void }): JSX
  - renders one line/area per series; per-series color from project-colors
  - hover tooltip snaps to nearest point across all series
  - brush emits (from, to); parent cross-filters the table
  - a series with coverageFrom > range start renders its pre-coverage span as
    an explicit "no data" band, never as zero
  - a Point may carry `break: true` (segment boundary — e.g. a quota reset) or
    be absent entirely (gap): the line is CUT at both, never interpolated
  - a Point may carry `projected: true`; rendered dashed and excluded from
    the "measured" tooltip band
  - tokens only (var(--*)), light + dark

Series = { id, label, color?, points: Point[], coverageFrom?: string }
Point  = { ts: string, value: number, break?: boolean, projected?: boolean }

ChartLegend({ series, hidden, onToggle }): JSX      // click toggles a project line
TimeRangePicker({ value, onChange }): JSX           // 1h | 1d | 3d | 7d | 30d | all
```

All three registered in `/design-system` with every state; barrel⊆gallery test and slopgate stay green.

### 5.2 Default view — "what happened while you were away"

The page opens on a digest since last visit, not a firehose. Direct answer to the owner's rejection of notifications: pull, not push.

Contents: episodes completed, failures, first-seen new problems, suppressed-notification count, budget state (§5.5). Last-visit timestamp is stored client-side.

### 5.3 Table

`DeckTable`, sortable and searchable, `FilterInput` plus facets on category / project / runtime / host / severity / actor. Rows collapse by episode and by `dedupe_key`. Row expands into `DetailDrawer` with the raw record. Newest first.

### 5.4 Charts

Owner's requested series: active agents, log lines generated, commits, bots activity, and per-account limit usage (§5.5) — each per-project selectable, over `1h/1d/3d/7d/30d/all`.

Two of these are NOT event counts and MUST NOT be implemented as such: **active agents** is the interval-overlap read over `activity_sessions`, and **limit usage** is the `activity_gauge` level (§4.3). **Log lines generated** = rows written to `activity_events` per source (§4.3), not transcript lines.

Plus, because these answer the questions actually asked this week:

| chart | question it answers |
|---|---|
| load overlaid with agent + gate events | "why is my laptop on fire" — the causal view, permanently |
| time-to-land p50/p95 | proves the land path got better or didn't (owner waited 12h on 2026-08-07) |
| gate runs local vs remote | is offload actually working, or only theoretically enabled |
| buildbox utilization + queue wait | same, from the box side |
| failures by source | what breaks most, ranked |

### 5.5 Budget

**Primary chart: 7d limit used %, one line per account.** X = time over the selected range, Y = 0–100%, one series per `(provider, account_key)` from `activity_gauge` (§4.6a), rendered by `TimeSeriesChart` with `yDomain: [0,100]` and a threshold rule at 100%. A 5h-window toggle switches `subject` suffix `7d` → `5h`; the chart is otherwise identical. Reset boundaries break the line (§4.6a) and each series' tooltip shows `resets_at`. `ChartLegend` toggles individual accounts, so a single account's burn can be isolated.

This is the chart that would have been read on 2026-08-07: the owner lost ~31% of a week's budget in a day and discovered it only by hitting the wall, with five agents dying mid-work at the cap. Projection-to-reset is drawn as a dashed extension from the current slope, labelled projection, and MUST be visually distinct from measured samples.

Secondary: absolute token/cost burn per runtime and per project, from `activity_sessions` metadata (§4.6) — answers *which work* spent the budget the percentage chart shows draining.

### 5.6 CI page

Memory-pressure kill events surface as a notice on the buildbox in the CI page — owner: *"put it as a notice on the buildbox in the CI page in overdeck ui. not a push notification to systray"*. Follow the `MachineCard` parity-row pattern landed 2026-08-08.

## 6. Error handling

- Store write failure: dropped, counted, surfaced as a `service` event. Never propagated to the emitting caller.
- Shipper failure: that source degrades to its last cursor and renders stale-with-timestamp. One dead shipper never blanks the page.
- Missing/unreachable Prometheus: overlay renders "no data".
- Corrupt trailing line in a source JSONL: skipped, counted, never fatal.
- Read API timeout: partial results with an explicit `partial: true`; never a fabricated total.

## 7. Testing

- Rollup correctness: raw→minute→hour aggregation equals direct raw aggregation, on a fixture spanning a DST-free and a DST-crossing window.
- Shipper resume: inode change, truncation, partial trailing line, and no-double-count across restarts.
- Coverage honesty: a series whose source starts mid-range renders a no-data band, asserted — not zero.
- Dedupe: 190 identical inputs produce one row with `count: 190`.
- Episode correlation: the six-source land sequence collapses to one episode; an unrelated concurrent land does not merge into it.
- Emitter non-blocking: a store made unwritable does not fail or delay the emitting caller.
- UI: barrel⊆gallery, slopgate, both themes, `pnpm --filter web build|typecheck`, `pnpm --filter @overdeck/deck-ui test|typecheck`, collector `bun test`.
- Browser probes go through `~/.claude/bin/e2e-remote` only. SSE ⇒ `domcontentloaded` + `waitForSelector`, never `networkidle`.

## 8. Architecture decisions

- **Store is new; not reusing `items.jsonl`.** Deletion test passes: it is a state-change log with adapter-skewed distribution (§3.1) and no range read path (§3.2). Reusing it would silently measure adapter churn as activity.
- **Rollups are a separate module, not a view.** They survive the deletion test: without them, `all` requires scanning raw forever. Deep — callers cannot tell how bucketing works.
- **Shippers and emitters are one module with two entry points**, not two subsystems. Both write the same record through the same non-blocking path; splitting them would be a decorative seam.
- **Episodes are a read-time correlation, not a write-time join.** Reversible: correlation rules can change without a migration, and rule 3 stays honestly marked as inferred.
- **Prometheus not extended.** High-cardinality per-project/per-session series are its documented antipattern (§3.4), and its retention cannot answer the requested ranges.
- **No new desktop notification.** Explicit owner constraint; the digest in §5.2 is the replacement.

## 9. Dependencies

- #86 — retention floor (textfile frozen, Prometheus retention unset, journald 4.4d cap) gates the load overlay (§5.4) and any claim about history depth.
- #85 — agent-guard poll mismatch keeps that panel empty; its events are a `guard` source here.
- #82 — dead OOM early-warning feed; `mem-pressure-notify` has no binary, so that emitter cannot exist until it is repaired.