# Overdeck unified observability — cluster and agent execution

audience: AI coding agents first. Contract-level: seams, shapes, routes, and ownership;
no implementation bodies.

Slug: `overdeck-observability`. Date: 2026-08-08. Task: #84.

## Problem

Overdeck currently has two incomplete answers to one operator question: **what is the
machine doing, and which agent caused it?** Prometheus describes host pressure without
agent identity. The collector describes sessions, harness runs, builds, and activity
without the Kubernetes placement and resource context in which they ran. Grafana and
Headlamp can inspect either side, but neither owns Overdeck's session ledger, build
identity, or safe tmux attach path.

No FOSS product provides this join. Overdeck therefore renders it natively. It does not
embed or fork Grafana or Headlamp.

## Goals

- Show cluster health and workload state from the local k3s cluster beside agent and
  build activity from the collector.
- Answer, per node, which recorded agent sessions and remote builds ran there, with the
  evidence used for every association.
- Drill from node → workload/build → agent observability and the existing safe session
  attach experience.
- Add Kubernetes lifecycle events to the existing Logs page and activity charts without
  creating a second event vocabulary or history store.
- Keep all service endpoints on loopback. Browser traffic continues through the existing
  web collector proxy and never receives Kubernetes or Prometheus credentials.
- Represent missing coverage, expired Jobs, ambiguous joins, and stale sources honestly.

## Non-goals

- No embedded Grafana, Headlamp, iframe, or reverse-proxied third-party UI. Both are
  optional link-outs only.
- No replacement for Prometheus, kube-state-metrics, cAdvisor, the Kubernetes API, or the
  agent-session ledger.
- No pod log warehouse, transcript-content ingestion, distributed tracing backend, or
  general-purpose Kubernetes console.
- No high-cardinality Prometheus labels for session ids, build keys, branches, prompts,
  or transcript paths.
- No mutation of cluster resources from the Cluster page in this design.
- No new deck-ui primitive is authorized here. If the implementation cannot compose the
  registry in `packages/deck-ui/src/index.ts`, it stops and requests owner approval with
  the proposed primitive contract.

## Architecture decisions

### One product surface, two stores by data semantics

“Unified” means one correlated read model and navigation graph, not forcing unlike data
into one database.

- Existing Prometheus remains the time-series store for numeric infrastructure metrics.
  kube-state-metrics and cAdvisor feed it through loopback-only scrape targets.
- The existing Overdeck activity store remains the durable event store for agent,
  build, and Kubernetes lifecycle events. Kubernetes events use the existing
  `ActivityEvent` contract and its retention, dedupe, coverage, Logs, and chart paths.
- The Kubernetes API remains the current-state authority for objects and placement. The
  collector polls it; the browser never does.
- The session ledger remains the authority for agent identity, liveness, host, parentage,
  and tmux/resume facts. Prometheus labels and pod-name parsing never override it.

A single replacement database is rejected: Prometheus is correct for sampled numeric
series and wrong for session identity; the event store is correct for durable discrete
facts and wrong for cAdvisor-rate queries. Grafana-as-shell and Headlamp-as-shell are
rejected because they cannot express the trusted join or reuse Overdeck's attach posture.

### Correlation is an explicit projection

The collector owns a rebuildable `ClusterSnapshot` projection assembled from Kubernetes,
Prometheus, the session ledger, and existing run/build panels. Source records are not
rewritten during a join. Every association carries `confidence` and `evidence`; an
unmatched workload and an unplaced session remain visible.

The projection may be cached for the adapter poll interval but is not a new source of
truth. Historical placement comes from Kubernetes lifecycle events in the activity store,
not from replaying the latest snapshot.

## Data flow and trust boundary

```text
kube-state-metrics ─┐
cAdvisor ───────────┼─ loopback scrape ─▶ existing Prometheus ─┐
                    │                                           │
k3s API ─ loopback/read-only ─▶ collector k8s adapter ──────────┼─▶ ClusterSnapshot
session ledger ───────────────▶ existing sessions adapter ──────┤
run/build panels + events ────▶ existing collector seams ───────┘
                                  │
                                  ├─ Kubernetes lifecycle ─▶ existing activity store
                                  └─ read APIs ─▶ web proxy ─▶ /cluster, /logs, session pages
```

Collector-to-k3s authorization is read-only and limited to nodes, namespaces, Jobs, Pods,
containers, conditions, owner references, and Events needed by this design. Prometheus,
k3s, Grafana, and Headlamp bind to loopback or an existing authenticated local tunnel.
External URLs are server-configured allowlisted bases; object-derived text never becomes
an arbitrary link target.

## Ingestion contracts

### Metrics producers

kube-state-metrics supplies Kubernetes object state: node conditions, allocatable
resources, Job status, Pod phase, scheduling, owner references, restart counts, and
requested/limited resources. cAdvisor supplies container CPU, memory working set,
filesystem, and network series. Existing node-exporter metrics remain the authority for
host-level CPU, memory, disk, pressure, and thermal signals.

Scrapes use the existing Prometheus instance and its retention. The collector's metric
reader uses the Prometheus HTTP API on loopback and performs bounded, allowlisted query
templates. It does not accept raw PromQL from browser parameters.

Metric identity is limited to stable cluster dimensions such as cluster, namespace,
node, workload kind/name, pod, and container. `overdeck.dev/build-key` is resolved through
the Kubernetes adapter and activity store, never copied into Prometheus labels.

### Kubernetes adapter

Add `collector/src/adapters/kubernetes.ts` with adapter id `kubernetes`. The existing
`cluster` adapter reports workstation agent-guard/buildslot state and is not renamed or
silently repurposed.

The adapter returns a `cluster:k3s` panel containing a `ClusterSnapshot`; discrete
transitions also enter the existing activity ingestion seam. A failed poll throws so the
collector retains the last known panel under normal adapter semantics, while the panel's
source timestamp exposes that it is stale. A successful scoped poll may remove objects
only for scopes declared complete; partial namespace or metric failures never erase the
last known state for unrelated scopes.

`ClusterSnapshot` has these contract-level fields:

| field | shape | meaning |
|---|---|---|
| `observedAt` | ISO timestamp | collector projection time |
| `cluster` | `{ id, version?, state }` | configured identity and observed reachability |
| `sources` | per-source `{ status, observedAt?, coverageFrom?, error? }` | Kubernetes, Prometheus, ledger, and builds independently |
| `summary` | nullable measured totals | ready/total nodes, running/pending/failed workloads, requested and used CPU/memory |
| `nodes` | `ClusterNode[]` | node drill-down rows |
| `unplacedSessions` | `SessionPlacement[]` | ledger sessions with no proven node |
| `unmatchedWorkloads` | `WorkloadPlacement[]` | relevant workloads with no session/build join |
| `links` | optional allowlisted Grafana/Headlamp links | link-out only |

`ClusterNode` carries node identity, Kubernetes conditions and taints, capacity and
allocatable values, current Prometheus utilization, resource requests/limits, workload
counts, pressure/restart facts, `sessions[]`, `builds[]`, source timestamps, and coverage
gaps. Numeric values are nullable; absence is never normalized to zero.

`WorkloadPlacement` carries cluster/namespace/kind/name/uid, pod uid/name, node name,
phase, reason, created/started/finished times, container states and restart counts,
requests/limits, observed utilization, owner reference, raw annotation join keys, and
links to correlated records. Kubernetes uid is the stable object identity; names are
display fields.

### Activity-event adapter

Kubernetes lifecycle observations normalize to the existing `ActivityEvent` shape. They
use category `buildbox` for Overdeck remote-build placement/completion and `service` for
node or non-build workload health. `source` is `kubernetes`; `host` is the Kubernetes node;
`session` is populated only when the join is proven. `detail` may carry namespace, kind,
uid, pod uid, build key, phase, reason, and resource deltas.

Emit transitions, not poll samples: node Ready/pressure changes, Pod scheduled/started/
terminal/restarted, and Job accepted/terminal. Stable event identity derives from object
uid plus transition identity/resource version so repeated polls are idempotent. Numeric
utilization samples stay in Prometheus and do not inflate the activity store.

The activity store is the **unified event store decision**: Kubernetes does not get a
second JSONL/SQLite history, and existing agent/build events are not copied into
Prometheus. The current `/activity` contract remains backward compatible; adding any new
category requires an explicit schema/UI migration rather than an undocumented string.

## Per-node session and build mapping

### Join keys

The remote-build launcher already writes `overdeck.dev/build-key` to both Job metadata
and the Pod template. The Pod copy is load-bearing because Kubernetes does not propagate
Job annotations to Pods. The exact annotation value is the canonical build key; the
hashed label and sanitized Job-name stem are display/search aids only and MUST NOT be
used as identity.

The collector joins in this order:

1. Pod → node from `pod.spec.nodeName`; Pod → Job from controller owner uid.
2. Job/Pod → build from exact `overdeck.dev/build-key` equality with the existing build
   record. Namespace/name/uid are retained as supporting evidence.
3. Build → session from an explicit build/run correlation field in the session ledger,
   sessions panel, or activity event.
4. Direct workload → session from an explicit ledger id/session id annotation, if a
   future launcher writes one.
5. Session → node from the ledger's recorded `host`, normalized through the buildbox/
   Kubernetes node registry.

There is no fuzzy join on title, cwd, branch, pod name, timestamps, pid, or sanitized
build-key stem. Temporal proximity may be shown as a candidate only in a diagnostics
gap; it never creates a clickable association.

`SessionPlacement` contains `sessionId`, `buildKey?`, `nodeName?`, `workloadUid?`,
`confidence: exact | host-only | unplaced`, `evidence[]`, and the current session view.
`exact` requires an explicit key chain. `host-only` means only the ledger host maps to a
node and must be labelled accordingly. Contradictory exact evidence yields `unplaced`
with a conflict gap rather than choosing a winner.

Expired Jobs remain historically discoverable through activity events after Kubernetes
TTL deletion. The live Cluster page labels them as historical and does not pretend their
current resource usage exists.

### Ledger extension boundary

If the present ledger cannot express Build → session, extend its schema once with optional
correlation fields such as `buildKey`, `runId`, and `workloadUid`; absorb version differences
inside `collector/src/sessions/ledger.ts`. The sessions panel and Cluster projection consume
the normalized `SessionRecord`. Components never parse raw ledger files or annotations.

## Read routes

Collector routes are authenticated like existing collector reads and proxied through the
existing default-deny web allowlist:

| route | response responsibility |
|---|---|
| `GET /cluster` | latest `ClusterSnapshot`, including independent source coverage |
| `GET /cluster/nodes/:node` | one node projection; 404 only for unknown node, not for a stale source |
| `GET /cluster/workloads/:uid` | workload detail and exact correlation evidence |
| `GET /cluster/metrics` | bounded range/bucket/query-id response for approved node/workload charts |
| `GET /activity` | existing event list, extended with Kubernetes events and node/build/session filters |
| `GET /activity/series` | bucketed activity series; counts events, never Prometheus samples |
| `GET /sessions/:id/screen` | existing safe tmux capture route; unchanged |

`/cluster/metrics` accepts a closed metric id (`cpu`, `memory`, `cpu-request`,
`memory-request`, `restarts`) plus node/workload identity, range, and bucket. It returns
series with unit, points, `coverageFrom`, gaps, source timestamp, and query status. No
raw PromQL, Kubernetes selector, filesystem path, or tmux target crosses the browser API.

The canonical agent observability href is resolved by the server/view-model from recorded
runtime/run/task identity. If no agent page exists for that record, the UI links to the
existing `/sessions` detail experience and states that agent observability was not
recorded. It never constructs a plausible-looking route from a build key.

## Native Cluster page

Route: `/cluster`. The page is a thin Astro shell; application components live under
`apps/web/src/components/cluster/**` and compose only exports from `@overdeck/deck-ui`.
The implementation reads the barrel and `/design-system` gallery before coding.

The default view answers three questions in order:

1. Is the cluster observable and healthy?
2. Where is pressure or failed work?
3. Which agents/builds are responsible, and can I inspect or rejoin them?

The summary uses existing `KpiTile`, `SectionCard`, `StatusChip`, and
`DataCoveragePanel`. The node list uses `DeckTable` with sortable node, readiness,
CPU, memory, pressure, workloads, builds, and sessions columns. Missing measurements
render as an em dash plus the named coverage gap. Relative timestamps use
`formatRelativeTime` with the absolute ISO value through `useDeckTooltip`; comparable
numbers use tabular numerals.

Selecting a node opens an existing `DetailDrawer` (or an existing approved modal where
the registry requires it) containing:

- Kubernetes conditions, capacity/allocatable, requests/limits, and current utilization;
- measured activity charts for the selected range;
- relevant workload/build rows via `DeckTable`;
- session rows using the same normalized status and reconnect vocabulary as `/sessions`;
- correlation evidence and source coverage;
- external “Open in Grafana” and “Open in Headlamp” `LinkButton`s when configured.

Selecting a session offers two separate actions:

- **Open agent observability** navigates to the canonical recorded agent/run/task page;
- **Open session / tmux** opens the existing session attach drawer/path, which resolves
  `agent-sessions attach <ledger-id>` and uses `/sessions/:id/screen` only when the
  collector's loopback, host, liveness, and tmux checks permit it.

The Cluster page does not synthesize tmux commands, expose sockets, or attach directly.
Remote sessions retain the existing SSH-wrapped attach descriptor. A disabled or unknown
path renders its precise reason.

Grafana and Headlamp links open a new top-level browsing context with safe rel attributes.
Their absence removes the action; it does not create an error card. They are never framed,
and Overdeck remains usable without either service.

## Logs page and activity charts

`/logs` remains the single event investigation surface. Kubernetes events appear through
the existing source/category/host/session facets. Add node and workload/build correlation
to event detail; a node/build/session selection on `/cluster` deep-links to `/logs` with
the corresponding filters and time range. Logs rows deep-link back to the node, workload,
build, session, or agent page only when that exact identity exists.

Activity charts deliberately distinguish two kinds of series:

- event charts from `/activity/series`: workload starts/completions/failures, restarts,
  session/build activity, and node condition transitions;
- metric charts from `/cluster/metrics`: CPU, memory, requests, and restart counters.

They may share a time-range control and brush selection but never share a y-axis or imply
that an event count is utilization. A metric outage creates a broken line and a coverage
message, not a zero. An activity source that began later shows its coverage boundary.
Prometheus retention bounds the metric range even when event history is longer.

Current deck-ui has no generic time-series primitive in the barrel. Phase 3 must first
attempt composition from approved exports. If that cannot meet the chart contract, work
pauses for explicit owner approval of a named primitive, props, tests, token additions,
both-theme states, barrel export, and `/design-system` registration. This document does
not pre-approve one.

## Component responsibilities

- `ClusterApp`: owns queries, range/filter URL state, refresh policy, and error boundaries;
  it does not normalize raw payloads in render functions.
- `ClusterContent`: composes summary, coverage, node table, and external links from a
  normalized view-model.
- `NodeDetail`: composes node metrics, workloads, builds, sessions, and join evidence;
  it does not fetch Kubernetes or Prometheus directly.
- `ClusterSessionActions`: receives canonical href and existing attach descriptor; it
  does not derive either.
- collector Kubernetes adapter: reads and normalizes Kubernetes current state.
- collector metric reader: executes only approved Prometheus query templates.
- cluster projection service: performs joins and emits coverage/evidence.
- activity ingestion: records idempotent Kubernetes transitions in the existing event
  store and supplies existing Logs/series reads.
- sessions layer: remains the sole owner of ledger normalization and attach capability.

All empty, loading, partial, stale, permission-denied, and source-error states are part of
the component contract. Fixtures may fabricate data only in tests and the design-system
gallery, where they are clearly fixture-scoped.

## Failure and freshness behavior

- Kubernetes unavailable: retain the last snapshot, mark it stale, show its timestamp,
  and disable object-specific links that cannot be validated.
- Prometheus unavailable or outside retention: keep Kubernetes topology and sessions;
  omit numeric values and show the metric coverage gap.
- Ledger unavailable: keep nodes/workloads/builds; show no session mapping and name the
  missing ledger source.
- One namespace forbidden: report partial scope and do not erase its previous objects.
- Clock skew or invalid timestamps: surface a data error; never compute negative duration.
- Duplicate/late Kubernetes transitions: idempotent event identity prevents duplicate
  history; late events retain source time and ingestion time where available.
- Conflicting joins: display both evidence records as a conflict, attach neither.
- External UI unavailable: link-out may fail independently; no native panel is replaced.

## Phased delivery

### Phase 0 — contracts and local scrape proof

- Pin Kubernetes RBAC, loopback bindings, source identities, Prometheus query ids, and
  coverage semantics.
- Configure kube-state-metrics and cAdvisor to feed the existing Prometheus; prove targets
  and representative series without adding session/build labels.
- Add fixtures for exact, host-only, missing, conflicting, expired-Job, and partial-source
  cases.

Exit: Prometheus answers node/container queries locally; the collector can read the k3s
API read-only; no browser or LAN endpoint can reach either directly.

### Phase 1 — Kubernetes snapshot and exact joins

- Add the distinct `kubernetes` adapter, normalized snapshot, coverage, and collector
  routes.
- Join Pod → Job → build key → build/session ledger; expose evidence and unplaced lists.
- Contract-test annotation handling against the real manifest shape in
  `k3s-remote-build.mjs`, including the Pod-template annotation.

Exit: an API consumer can name every node and show exact, host-only, or unplaced status
for every relevant session/build without fuzzy inference.

### Phase 2 — native Cluster page and drill-through

- Ship `/cluster` using existing deck-ui exports, node drill-down, source coverage,
  canonical agent links, existing session/tmux attach path, and optional link-outs.
- Add web proxy allowlist entries only for the exact new read routes.

Exit: from a node, the owner can reach the correct recorded agent view and safely open an
attachable tmux session; unavailable actions state why. Both themes, keyboard operation,
viewport containment, deck-ui tests/typecheck, web build/typecheck, gallery test, and
slopgate are green.

### Phase 3 — unified event history and charts

- Emit Kubernetes lifecycle transitions into the existing activity store and integrate
  node/workload/build filters and bidirectional deep links with `/logs`.
- Add bounded activity and Prometheus series reads.
- Compose charts from approved deck-ui exports or stop for primitive approval as required
  by the UI skill.

Exit: the same time selection explains both resource pressure and the exact agent/build
events on the affected node, with visible and correct coverage boundaries.

### Phase 4 — historical hardening

- Validate behavior across Job TTL deletion, node replacement, restarts, Prometheus
  retention, and ledger schema migration.
- Add rebuild/reconciliation checks for activity transitions and alert only through
  existing Overdeck policy; this feature introduces no new notification channel.

Exit: a deleted Job remains traceable from Logs to its recorded build/session, while live
Cluster state stays explicitly historical or absent rather than fabricated.

## Verification contract

- Unit: Kubernetes normalization, stable identities, scoped reconciliation, Prometheus
  response/gap parsing, and every join confidence branch.
- Contract: collector payload schemas; raw PromQL/selectors rejected; proxy allowlist
  default-denies adjacent paths; activity schema remains backward compatible.
- Integration: fixture cluster with two nodes, annotated Job/Pod, matched and unmatched
  sessions, partial metric failure, and Job deletion.
- UI: node sorting, partial/stale/empty states, exact deep links, attach refusal reasons,
  external links, both themes, keyboard/focus behavior, and no invented zeros.
- Live: submit one annotated remote build, observe its scheduled node, open the correlated
  session/agent page, exercise the installed `agent-sessions attach` path, find its
  lifecycle in Logs, and compare the node metric against the local Prometheus query.

## Dependencies and sequencing

- The activity-store and Logs contracts in this repository are reused, not restarted.
- The session ledger, sessions panel, and tmux attach path must remain the single reconnect
  authority.
- The k3s scheduler/remote-build work must preserve exact
  `overdeck.dev/build-key` annotations on both Job and Pod template.
- UI implementation is governed by `.claude/skills/od-ui-dev/SKILL.md`; no task may add a
  primitive, raw visual value, hand-rolled table/status control, or unregistered barrel
  export without the required owner decision and gallery work.