# Agent and Cluster Observability

audience: AI coding agents first.

status: DONE
task IDs: #1
source request: Make Overdeck observability useful: unified activity logs, agent/build/session/node visibility, time-series charts, drill-down, and terminal attach. Never substitute notification-delivery metadata for actual event evidence.

## Outcome

Overdeck shows what work ran, where, when, and with what result across Factory, harness, Claude, CDX, bots, builds, Git, incidents, and k3s/SSH nodes. Owner can filter, search, chart, drill into one run, inspect logs, and attach to its terminal.

## Acceptance criteria

- Inventory every authoritative event/log source before UI work.
- Normalize event identity, project, category, host/node, session/run, timestamps, lifecycle, result, and evidence link without fabricating missing fields.
- Use `@platform-modules/ui-primitives` `DataTable` for searchable/sortable activity rows and existing platform primitives elsewhere.
- Row click opens run/session detail with individual logs and lifecycle evidence.
- Charts support 1h, 1d, 3d, 7d, 30d, all-time; project/category filtering; active-agent, log-volume, commit, bot, build, and incident activity.
- Cluster view maps sessions/builds/agents to nodes and links to agent detail and tmux attach.
- Notification suppression is not an incident. Record underlying event; do not harass owner with “notification failed” noise.
- Both themes work; report and product surfaces do not regress to light-only.
- Production is browser-proven with honest live data.

## Preserved context

- Existing report: `/home/user/.local/share/overdeck/reports/2026-08-08-session-report.html`.
- Earlier activity/log/session UI tasks may contain reusable delivery; recover before rebuilding.
- Existing Prometheus/node_exporter/Grafana stack supplies host metrics but does not replace agentic event semantics.
- `live-report-sync` owns current status truth. This plan owns product observability and drill-down UX.

## Constraints

- UI MUST obey `.claude/skills/od-ui-dev/SKILL.md` and use platform primitives first.
- Do not dump raw logs as the primary UX.
- Do not claim observability from registration, suppression, or stale task labels.
- Keep SSH and k3s execution sources visible during incremental migration.
- This plan must not block emergency offload or Factory restoration.

## Execution steps

1. Recover prior observability brainstorm, shipped components, schemas, and task receipts.
2. Inventory authoritative event sources and retention/query limits.
3. Write normalized event/detail/time-series contract.
4. Implement ingestion and deterministic correlation before presentation.
5. Build DataTable activity view, detail drill-down, charts, node/session map, and terminal attach links from reusable components.
6. Verify source counts, lifecycle transitions, filters, theme behavior, and browser paths.
7. Land and deploy; record owner-visible URL and proof.

## Latest acceptance delta

2026-08-13 wire-contract acceptance delta: remove duplicated collector public wire definitions. Collector MUST import/re-export shared `@overdeck/activity-contract` response/event/coverage/series types directly; raw source-reader types may remain collector-internal. Re-open `collector/src/activity/wire-contract.ts`, `collector/src/activity/types.ts`, and `packages/activity-contract/src/index.ts`; run exact collector typecheck and record command/output. Standalone JSX/module-resolution diagnostics are non-authoritative only if web project typecheck remains clean. Do not spawn/delegate, commit, deploy, or run browser.

2026-08-13 final-review acceptance delta: fix all nine current defects. (1) Enforce half-open interval ordering when one session ends exactly as another starts; simultaneous boundary MUST NOT peak at 2, with exact regression test. (2) Derive series coverage from authoritative retained coverage, NEVER first observed event; distinguish quiet measured time from unknown gaps and test both. (3) Generic agent-session exit MUST end with unknown result unless producer supplies typed outcome. (4) Kubernetes lifecycle MUST use authoritative transition/phase fields only, NEVER resource/title text; successful job named `failed-migration` MUST remain success. (5) `TimeSeriesChart` MUST split SVG paths at declared gaps and test no connecting segment. (6) Chart interactive targets MUST share SVG coordinate space/layout under narrow scrolling and remain accessible, with test. (7) `/sessions?session=` MUST robustly consume identity on load/reload/history and select exact session, or route to an existing exact-record destination; test reload/history. (8) Browser proof MUST fail closed when required charts, facets, evidence, session, or terminal capability is absent; honest live absence MUST use explicit capability-aware unavailable assertions, NEVER silent conditional skip. (9) Shared `ActivitySourceCoverage` MUST own `storage` and `queryBounds`; collector and UI MUST use/render required source inventory and caveats through strict shared contract ownership. Run all full gates cleanly. Do not spawn/delegate, commit, deploy, or run browser.

2026-08-13 coordinator acceptance delta: complete approved reusable deck-ui `TimeSeriesChart` with actual points, truthful coverage/gaps, accessible states, token-only light/dark themes, colocated tests, barrel export, and `/design-system` gallery registration; use it in `LogsContent`; replace standalone query-cache last-good test with rendered `ClusterContent` failed-refresh coverage through production query path; complete real-proxy browser-proof script interactions for facets, evidence, cluster, sessions/tmux, themes, refresh cadence, and console cleanliness without interception, but MUST NOT run browser. Run full collector suite/typecheck, deck-ui tests/typecheck, focused web tests, web build/typecheck, and diff checks. Do not spawn/delegate, commit, deploy, or run browser.

2026-08-13 frozen-review corrections: canonical active intervals MUST be built independently of display filters and requested endpoint clipping; reusable source snapshots/indexes MUST serve changing sliding bounds and filters without rescanning; session finish reason MUST distinguish cancellation, failure, success, and unknown; action records with no authoritative outcome MUST remain unknown; `projectCounts`, `series`, and `errorCount` MUST remain optional for rolling deployment; URL hydration/history tests MUST cover category, severity, grouping, search, project, range, and selected record; aggregate and source drawers MUST reuse normalized detail rows; direct collector-state last-good coverage and promised rendered failed-refresh coverage MUST be truthful; real-proxy browser harness MUST cover all acceptance interactions, themes, request success, and console cleanliness without interception, but MUST NOT run yet. Synchronize plan/index next action after full gates.

2026-08-12 stable review: use one shared workspace activity wire contract; one source normalizer for aggregate and entries; select exact source records before evidence navigation; derive active-agent concurrency from canonical ledger intervals; expose commits, bots, builds, and incidents only from authoritative descriptor capabilities or mark unavailable; build proof from the real collector or a shared fixture builder; remove category/project facet duplication with a local generic `FacetButtonGroup`; preserve rolling-deploy compatibility; make URL state complete and evidence navigation same-origin and allowlisted; keep sliding windows fresh; remove the duplicate error request; add deterministic direct last-good fleet state coverage; remove sensitive and duplicate proof artifacts; run all requested gates without commit, deploy, or browser execution. Measured load acceptance: replace two `/activity` polls per 15 seconds with one response carrying the error total, and cache or index source reads so repeated consumers do not rescan the full corpus; deterministic tests MUST prove one activity poll and reuse of source reads. Baseline lower bound is about 168 MB parsed and 61,000 file opens per minute.

## Current receipt

2026-08-13 final-review repair complete in `/home/user/Projects/overdeck/.worktrees/agent-observability-final`; no browser, commit, or deploy. Active-agent sweep now orders ends before starts at equal timestamps and exact half-open regression peaks at 1. Count series require explicit authoritative `coverageFrom`; healthy retained quiet ranges emit measured zero while unknown retention is unavailable. Generic session exits complete with unknown result unless producer emits exact typed success. Kubernetes classification reads only authoritative `detail.phase/status/reason`; successful `failed-migration` title remains success. `TimeSeriesChart` splits paths across declared gaps and places accessible hit targets inside the same SVG coordinate system as visible marks. Persistent session terminal consumes `?session=` after client hydration, follows `popstate`, writes identity on open, and clears it on close; reload/history test passes. Browser proof has no silent capability skips: required charts/facets/evidence/session/terminal paths either execute or assert explicit response-backed unavailable/zero states. Shared `ActivitySourceCoverage` requires `storage` and `queryBounds`; raw source coverage remains internal, normalized aggregate/source-entry responses strictly own required metadata, and Logs renders both inventory caveats.

Landed observability candidate reached `origin/main`; automatic deployment stopped safely because runtime Pi authentication at `modules/workstation/pi/agent/auth.json` is preserved inside the deploy clone but was not excluded from source cleanliness checks. Follow-up worker uses clean landed worktree `/home/user/Projects/overdeck/.worktrees/disk-admission-land`; packaging fix adds exact private exclude without reading, moving, or deleting credentials. Deployment regression suite passes 30/30. Deployment, installed proof, report, DONE receipt, and final land remain pending.

Verification: exact collector regressions 20/20 pass; full collector suite and TypeScript pass. Chart regressions 4/4 pass; full deck-ui suite and TypeScript pass. Focused sessions/logs 40/40 pass; full web suite passes; production web build passes; web typecheck 221 files with 0 errors, 0 warnings, 0 hints. Browser-proof source parses without execution. `git diff --check` passes. Build initially caught server-side `window` access in session URL hydration; moved hydration into client effect and reran affected test/build/typecheck cleanly.

Test and land exact runtime-auth exclusion, deploy landed main, verify installed observability routes/endpoints, write installed report, then set and land `DONE`.

2026-08-13 wire-contract correction complete in `/home/user/Projects/overdeck/.worktrees/agent-observability-final`; no browser, commit, or deploy. `collector/src/activity/types.ts` now imports and re-exports public events, coverage, response, series, skipped-entry, and suppression types directly from `@overdeck/activity-contract`; only raw source-reader coverage, reader options, and collector query options remain local. `collector/src/activity/wire-contract.ts` directly re-exports shared response types instead of checking structural assignability. Exact collector command `cd /home/user/Projects/overdeck/.worktrees/agent-observability-final/collector && bun run typecheck` passed with output `$ tsc --noEmit`. Authoritative web command `cd /home/user/Projects/overdeck/.worktrees/agent-observability-final && pnpm --filter web typecheck` passed 221 files with 0 errors, 0 warnings, and 0 hints; runner additionally reported one package-install warning without warning text, outside Astro diagnostics. `git diff --check` passed with no output.

2026-08-13 follow-up acceptance delta: direct-re-export `ACTIVITY_SEVERITY_ORDER` and `ALL_ACTIVITY_CATEGORIES` from `@overdeck/activity-contract`; MUST NOT import these values into collector local scope. Preserve collector public API. Rerun exact collector typecheck and `git diff --check`. Do not spawn/delegate, commit, deploy, or run browser.

2026-08-13 follow-up complete: `collector/src/activity/types.ts` now direct-re-exports `ACTIVITY_SEVERITY_ORDER` and `ALL_ACTIVITY_CATEGORIES` from `@overdeck/activity-contract` without importing either into local scope, preserving the collector public API and eliminating both no-unused diagnostics. Exact command `cd /home/user/Projects/overdeck/.worktrees/agent-observability-final/collector && bun run typecheck` passed with output `$ tsc --noEmit`; `git diff --check` passed with no output. No spawn/delegation, commit, deploy, or browser run.

2026-08-13 delivery acceptance delta: audit candidate scope; create one local commit with short imperative message and no co-author; run prepared real-proxy browser proof through `~/.claude/bin/e2e-remote` with server and browser both remote, server bound to `127.0.0.1`, no interception; inspect every proof artifact/result; fix defects or warnings and use a new commit for follow-up changes. MUST NOT land or deploy.

2026-08-13 delivery complete through local commit and live proof; MUST NOT land or deploy. Base observability commit `8be2f6c9e`; browser-proof follow-up commits `2590dfbbd`, `f7e181299`, `28829269f`, `8378ba93a`, `68c760029`, `a7974b0eb`, `b4cd492b4`, `ed7127a70`, `371cdf3e6`, `7db05011c`, `732a8a841`, `0dcd27376`, `c5bf814d4`, `e4cec6cd8`, `7d799a9af`, `450e38b46`, and `9840712ab`. Final exact proof command used `~/.claude/bin/e2e-remote` with remote pnpm bootstrap, collector plus Astro server bound to `127.0.0.1:31337`, real server-side collector proxy, no interception, and remote Chromium; server ready after 3s on `debian1`, client exit 0. Inspected all seven artifacts under `apps/web/browser-proof/agent-observability-live/results/`: `logs.png`, `cluster-arrival.png`, `cluster-arrival.txt`, `cluster-response.json`, `cluster.png`, `sessions.png`, `receipt.json`. Receipt: 1h events 0; nine source inventories; active-agent series available with 48 points; remaining five series explicitly unavailable; exact request list includes filter/range polling and no action requests; cluster returned honest 503 `cluster-snapshot-unavailable` and rendered visible error; sessions rendered zero reconnectable with host coverage; only console error exactly matched cluster 503. Full follow-up gates passed: collector `bun run typecheck && bun test --timeout 20000`; web `pnpm --filter web typecheck && pnpm --filter web build` with 0 errors/warnings/hints; deck-ui `pnpm --filter @overdeck/deck-ui test && pnpm --filter @overdeck/deck-ui typecheck`; `git diff --check` clean.

2026-08-13 final-delivery acceptance delta: reproduce `SourceEntriesContent` implicit-any diagnostics only with authoritative project check; MUST NOT alter code for standalone JSX noise. Fetch and integrate `origin/main` safely. Rerun all affected clean gates and `git diff --check`. Land ONLY through `.claude/scripts/ship.sh`; then run `packaging/deploy-local.sh`. Verify installed `/logs`, `/cluster`, and collector endpoints. Use `e2e-remote` for browser+server pair. If live cluster remains unavailable, verify visible unavailable state and MUST NOT claim links proven. Write installed report `/home/user/.local/share/overdeck/reports/2026-08-10-agent-observability.md` with exact receipts. Set plan/index `DONE` only after installed proof; commit, land, and deploy final receipt updates. Mark task #1 complete only then.

2026-08-13 final-delivery receipt: authoritative web typecheck passed 221 files with 0 errors, 0 warnings, and 0 hints; no standalone JSX/module noise changed source. Observability and runtime-auth preservation landed through canonical `.claude/scripts/ship.sh`; served implementation release identifies commit `b8941c371b6520d795d72b1c2b569871e2e1db47`. Canonical deployment completed far enough to promote that release and restart installed services. `overdeck-web.service`, `overdeck-collector.service`, and `overdeck-web-watchdog.service` are active with successful results. Installed `http://127.0.0.1:31337/logs` and `/cluster` return HTTP 200. Installed `/api/collector/activity` returns HTTP 200 with 500 retained events and nine coverage sources. Installed `/api/collector/cluster` returns HTTP 200 with authoritative cluster, node, source, workload, build, and session fields. Earlier remote real-proxy browser proof remains valid and inspected all seven artifacts with no interception or action requests; its cluster request honestly returned 503 and visible unavailable state at proof time, so that run did not claim cluster-link navigation. Installed cluster data became available during final verification; endpoint availability is proven, but cluster-link browser navigation was not rerun or claimed. Installed report: `/home/user/.local/share/overdeck/reports/2026-08-10-agent-observability.md`.

Next executable action: none — task #1 is complete.

## Recovered lost-session receipt — 2026-08-12

Ground truth transcript: `/home/user/.claude/projects/-home-user-Projects-overdeck/38549e85-6aeb-4b67-a9ce-8fe2e649b559.jsonl`. Session closed cleanly after substantial continuation not reflected in earlier handoff.

- Implementation commits `14cb92ee6` and `fcd1baa3d` are ancestors of `origin/main`; observability backend and UI were landed rather than lost.
- Current integration worktree: `/home/user/Projects/overdeck/.worktrees/agent-observability-final`.
- Final independent review replaced earlier low-priority normalization suggestion with two required corrections:
  1. `collector/src/activity/read.ts:127`: generated “Open evidence” URL uses dead deep-link state. Build functional evidence navigation and test it.
  2. Event normalization derives lifecycle/result only from exit code. Preserve explicit source-owned lifecycle/result for denied actions, Kubernetes transitions, and agent sessions without exit codes; add focused behavior tests.
- UI review additionally requires shared test setup and focused behavior tests. Never accept visual-only proof for evidence links or filters.
- Preserve already landed source. Fix only verified residual findings; do not rebuild completed observability work.


Browser verification requires a supported `e2e-remote` input transport for uncommitted worktrees, or a committed candidate ref. User explicitly prohibited commits. Current wrapper cannot verify this candidate as framed.

## Historical blocked receipt

Remote build infrastructure was fenced (`epoch-mismatch` / `runtime-missing` on all three buildboxes), so that earlier gate stopped after collector and deck-ui suites rather than silently falling back to the laptop. This blocker was later cleared; current receipt above supersedes it.

## 2026-08-13 correction receipt

- Fixed source-owned outcomes: denied actions now remain failed/failure even with exit code 0; agent start and finish records emit explicit lifecycle/result instead of becoming unknown. Focused source tests cover both paths.
- Fixed evidence navigation end to end: evidence URLs target `/logs/<source>?event=<record>`, source pages read the URL selection, query the exact record across all retained history, reset paging for exact selection, and open the existing detail drawer. Collector, client serialization, and UI initial-state tests cover the contract.
- Activated existing shared Vitest setup (`apps/web/src/test-setup.ts`) across web tests by removing redundant per-file matcher imports. Focused web result: 49 passed.
- Focused collector result: 26 passed. Full collector and deck-ui runs advanced cleanly until the fleet fence stopped the combined gate; no failure was ignored or treated as green.
- Full completion remains pending because the mandatory remote gate and remote browser proof cannot run while every registered buildbox fails seat audit with `runtime-missing`.
