# Unified observability reports

**Date:** 2026-08-16
**Status:** implementation-ready
**Plan slug:** `observability-reports`

**Owner-visible evidence:** each slice ends with a newly usable capability on the installed `/reports` page, exercised through normal navigation against authoritative collector data. A before/after capture and an owner-flow assertion name the capability added by that deploy.

## Purpose

`/reports` is the place to understand what changed, what succeeded, what failed, where time and capacity went, whether the evidence is trustworthy, and what deserves attention. It unifies operational reporting without replacing the focused operational pages that own live intervention.

The page serves one owner first. It must answer, without knowing internal IDs or subsystem vocabulary:

1. What happened in the selected period?
2. What outcome did the work produce?
3. What failed or is getting worse?
4. Which project, request, run, account, host, source, or gate explains it?
5. Is the answer complete, partial, stale, or unavailable?
6. What recorded evidence supports the answer?

Commit count and raw activity volume are context, never headline measures of progress. Delivery outcomes, verified criteria, unresolved owner decisions, failure classes, recovery, and data coverage are the primary measures.

## Scope

### Reporting domains

The complete page covers:

- **Outcomes and delivery:** requests asked, started, blocked, shipped, deployed, and proven; verified acceptance-criteria movement when recorded.
- **Execution:** factory and agent runs, phases, attempts, gates, tool calls, diffs, models, accounts, sessions, hosts, durations, retries, failures, and cancellations.
- **Data health:** source freshness, authority, coverage windows, retention, records read, records skipped, parse/schema failures, gaps, and volume trends.
- **Reliability:** incidents, services, buildboxes, CI, deploys, guards, reaper events, recovery time, recurrence, and unresolved failures.
- **Capacity and spend:** model/token/cost usage, account limits, seat identity, host workload, queue pressure, and available capacity.
- **Correlation and evidence:** request → run → gate → change → land → deploy → proof, with every missing link called out as a coverage gap rather than guessed.

Focused pages such as `/requests`, `/factory`, `/logs`, `/incidents`, `/cluster`, `/ci`, and `/limits` remain the intervention and raw-detail surfaces. `/reports` links into them with the current report context.

### Included report interactions

- Global time range, comparison period, project, host, account, source, status, and text filters.
- URL-backed query state so a report can be bookmarked and revisited.
- A compact report header with generated time and a single coverage verdict.
- Outcome summary, changes from the comparison period, and attention items.
- Trend and breakdown views whose numbers are selectable and drill into supporting records.
- A right-side evidence drawer for the selected metric, anomaly, or row.
- Absolute timestamps alongside relative labels; owner timezone is used for buckets and displayed explicitly.
- CSV export for the currently visible tabular result and JSON export of the report envelope, with coverage metadata included.
- Loading, empty, partial, stale, and error states at section level; one failed source does not erase healthy sections.
- Both themes, token-only styling, keyboard operation, viewport containment, and honest unknown states.

### Explicit exclusions

- No fabricated forecasts, anomaly scores, or causal claims.
- No arbitrary SQL/report builder in this project.
- No second alerting system and no direct infrastructure notifications to the owner.
- No report mutation controls in the first six slices; intervention stays on the owning focused page.
- No warehouse or snapshot store before a visible historical-comparison slice needs it.

## Product structure

### Report header

The header identifies the selected period in owner language, shows when the report was generated, and labels coverage as `Complete`, `Partial`, `Stale`, or `Unavailable`.

The header owns the global filters. Filter changes update the URL and refetch one report envelope. Applying a filter never silently drops a domain; each section reports whether the filter is supported and what data remains uncovered.

Default view: the last 24 hours ending now, compared with the immediately preceding 24 hours, in the configured local timezone. The comparison control is hidden until Slice 5 provides durable history.

### Overview

The overview answers “what changed?” using recorded outcomes:

- shipped and deployed work;
- verified criteria gained or lost;
- work still blocked on the owner;
- failed runs, gates, deploys, or services;
- incidents opened and resolved;
- source coverage degradation;
- capacity or spend pressure.

A metric is absent until its authoritative projection exists. The page never reserves fake KPI tiles for future slices.

### Domain sections

Each domain follows one interaction grammar:

1. **Summary:** a small set of comparable measures with explicit units and coverage.
2. **Trend:** only when at least two trustworthy buckets exist.
3. **Breakdown:** ranked projects, failure classes, sources, accounts, hosts, or models.
4. **Records:** an Astryx table of the rows supporting the current selection.
5. **Evidence:** selecting a measure or row opens the shared `DetailDrawer` with provenance, query window, source records, links, and named gaps.

Numbers always expose their denominator where one exists. Percentages are never emitted when the denominator is zero or partial without a coverage label.

## Incremental delivery contract

Future sections stay dark—not empty placeholders—until their slice has authoritative data, owner-flow tests, and installed proof. Slice N lands, deploys, and is proven before Slice N+1 starts.

### Slice 1 — Activity and source trust

**Visible difference:** `/reports` appears in navigation and gives a trustworthy report of the last 24 hours from the existing activity registry.

It includes:

- report header, range, project/source/severity/text filters, and URL state;
- total recorded events, failures, active sources, and source problems;
- activity volume by category and time bucket;
- “Needs attention” rows for error events, source read errors, and skipped records;
- a source-health table showing authority, freshness, coverage window, storage, retention, records, skipped count, and error;
- drill-down to source entries and raw evidence using existing `/logs/[source]` semantics;
- report-level coverage verdict derived from source coverage, never from event count alone.

This slice reuses `/activity` and `/activity/sources/:id/entries`; no persistence is added.

### Slice 2 — Work execution and delivery

**Visible difference:** the page explains how owner requests moved through work and delivery, not merely how many events occurred.

It adds:

- request outcomes and current backlog composition;
- request → factory run correlation where an explicit identifier exists;
- run completion/failure/cancellation rates;
- phase and gate duration distributions;
- retries, failed attempts, tool failures, file diffs, models, accounts, sessions, and hosts;
- land/deploy/proof outcomes;
- a run/request evidence story linked to `/requests` and `/factory/:id`;
- named correlation gaps when an identifier was not recorded.

Collector projection joins only explicit IDs and recorded evidence links. Title, timestamp proximity, branch-name similarity, and receipt prose are not correlation keys.

### Slice 3 — Reliability and recovery

**Visible difference:** the report shows what broke, how long it stayed broken, whether it recovered, and whether it recurs.

It adds:

- incidents opened/resolved and unresolved age;
- deploy, CI, service, guard, reaper, and buildbox failures grouped by recorded failure class;
- recovery duration only for explicitly paired lifecycle records;
- recurrence by stable failure key;
- host/service/project breakdowns;
- evidence links to `/incidents`, `/ci`, `/cluster`, and `/logs`.

No inferred recovery time is calculated from the absence of later errors.

### Slice 4 — Capacity, accounts, and spend

**Visible difference:** the report attributes resource pressure and model spend to the recorded account, model, host, project, and run.

It adds:

- tokens, recorded spend, billed/unbilled state, and cost per completed outcome;
- account usage windows and limit pressure;
- seat/account identity coverage;
- host workload, queue pressure, and active capacity;
- project/model/account/host breakdowns;
- links to `/limits`, `/sessions`, `/agents`, and `/cluster`.

Costs remain unavailable where the provider or run did not record them. The UI never multiplies tokens by a guessed price.

### Slice 5 — Durable history and comparison

**Visible difference:** 7-, 30-, and 90-day comparisons remain trustworthy across log rotation and service restart.

It adds an append-only, versioned reporting snapshot store containing normalized aggregate buckets and coverage metadata—not copied raw logs. It supports:

- previous-period comparison;
- trustworthy trends and coverage gaps;
- daily, weekly, and monthly ranges;
- explicit retention and earliest-available labels;
- backfill only from authoritative records still available, with backfilled buckets marked.

Snapshot generation is idempotent by bucket, metric version, and dimension key. A schema-version change writes a new metric version; it does not rewrite historical meaning silently.

### Slice 6 — Cross-domain lineage, saved views, and export

**Visible difference:** one report can follow a request to proof, be bookmarked, and export exactly what the owner sees.

It adds:

- cross-domain lineage with missing-link coverage;
- saved named report views stored locally for the owner;
- CSV export per visible table;
- JSON export of the complete report envelope, including query and coverage;
- stable links into focused pages preserving supported filters;
- a concise printable view.

Export is generated from the same validated envelope rendered on screen, not a second query path.

## Architecture

### 1. Report projection

`buildObservabilityReport(query, dependencies): Promise<ObservabilityReport>`

A collector-side projection is the single authority for calculations, coverage, and correlations. It reads existing activity sources and collector panels through narrow reader functions. It returns domain sections independently so one source failure produces a partial report rather than a global error.

The projection owns:

- query validation and timezone-aware bucket boundaries;
- metric definitions, units, denominators, and comparison semantics;
- explicit-ID correlation;
- source freshness and coverage aggregation;
- section-local errors;
- stable evidence references.

The web app does not recalculate domain metrics from raw rows.

### 2. Shared report contract

A package-level contract is shared by collector and web. The stable envelope is:

```ts
interface ObservabilityReportQuery {
  from: string
  to: string
  timezone: string
  compareFrom?: string
  compareTo?: string
  projects?: string[]
  hosts?: string[]
  accounts?: string[]
  sources?: string[]
  statuses?: string[]
  severity?: ActivitySeverity
  q?: string
}

interface ReportCoverage {
  status: 'complete' | 'partial' | 'stale' | 'unavailable'
  from?: string
  to?: string
  generatedAt: string
  sources: Array<{
    id: string
    authority: 'authoritative' | 'derived'
    status: 'ok' | 'absent' | 'error' | 'stale'
    earliest?: string
    latest?: string
    records: number
    skipped: number
    reason?: string
  }>
  gaps: Array<{
    domain: string
    metric?: string
    from?: string
    to?: string
    reason: string
  }>
}

interface ReportMetric {
  id: string
  label: string
  value: number | null
  unit: 'count' | 'percent' | 'milliseconds' | 'tokens' | 'currency'
  numerator?: number
  denominator?: number
  comparison?: { value: number | null; delta: number | null; direction: 'up' | 'down' | 'flat' | 'unknown' }
  coverage: 'complete' | 'partial' | 'stale' | 'unavailable'
  evidenceQuery?: ReportEvidenceQuery
}

interface ObservabilityReport {
  schemaVersion: 1
  query: ObservabilityReportQuery
  generatedAt: string
  coverage: ReportCoverage
  sections: ReportSection[]
}
```

`ReportSection` is a discriminated union by domain (`activity`, `execution`, `reliability`, `capacity`, `lineage`). Each section carries its own status, metrics, series, breakdowns, records, evidence queries, and errors. New sections can be added without changing existing section semantics.

### 3. HTTP routes

- `GET /reports/observability` accepts the validated query and returns `ObservabilityReport`.
- `GET /reports/observability/evidence` accepts one bounded `ReportEvidenceQuery` emitted by the report projection and returns supporting records plus coverage.
- Slice 5 adds bounded history reads behind the same report route.
- Slice 6 adds owner-local saved-view routes and export responses derived from an already validated query.

All list filters are repeatable query parameters. Invalid timestamps, inverted ranges, unsupported timezone names, excessive ranges, and unbounded evidence requests return `400` with one owner-readable reason. Source read failures return `200` with section-level partial coverage unless no section can be produced, in which case the route returns `503` with the report coverage envelope.

### 4. Web report feature

`ReportsContent(): JSX`

The page owns only URL/filter state, query loading, section selection, and drawer selection. It renders report sections from the contract and never reconstructs metrics.

Generic controls, cards, tabs, tables, and disclosures come from Astryx first. Existing overdeck-specific `DetailDrawer`, `StatusChip`, time formatters, tooltips, chart/report compositions, and safe URL helpers are reused from `@overdeck/deck-ui`. No new primitive is created without explicit owner approval and gallery registration.

Route shell: `apps/web/src/pages/reports.astro`. Navigation label: `Reports`, placed after `Overview` in the primary section.

### 5. Evidence drawer

`ReportEvidenceDrawer({ query, selection, onClose }): JSX`

The drawer shows:

- the selected measure in owner language;
- exact filter and time window;
- calculation definition and denominator;
- coverage verdict and every relevant gap;
- supporting records with who, what, when, where, how, account, host, project, run/session identity, gate/tool/diff links when recorded;
- safe links to owning focused pages.

Routine raw detail is collapsed by default. Missing values render as `Not recorded`; internal IDs appear only in evidence detail, never on summary cards.

### 6. Historical snapshot store (Slice 5 only)

`writeReportBucket(bucket): Promise<'inserted' | 'unchanged'>`
`readReportBuckets(query): Promise<ReportBucket[]>`

The store persists versioned aggregate buckets and their coverage, source watermark, and generation metadata. It does not duplicate raw evidence. Writes are atomic and idempotent. Corrupt rows are quarantined and surfaced as coverage gaps; healthy buckets remain readable.

The store boundary is delayed until Slice 5 because it has a second implementation possibility and hides retention/versioning complexity that live projections should not own.

## Data flow

```mermaid
flowchart LR
  Sources[Activity sources and collector panels] --> Projection[Report projection]
  Projection --> Contract[Validated report envelope]
  Contract --> Route[Collector report route]
  Route --> Page[Reports page]
  Page --> Drawer[Evidence drawer]
  Drawer --> Focused[Requests / Factory / Logs / Incidents / CI / Limits]
  Projection -. Slice 5 aggregate buckets .-> History[Versioned report history]
  History -. comparisons .-> Projection
```

## Coverage and freshness rules

- `Complete`: every required authoritative source for the emitted metric covered the full query window with zero known gaps.
- `Partial`: a metric is calculable but at least one required source, range, identity, or join is missing.
- `Stale`: required source data exists but exceeds that source’s declared freshness threshold.
- `Unavailable`: no defensible value can be calculated.

Derived sources can enrich a metric but cannot upgrade an incomplete authoritative source to complete. Report coverage is the worst required-domain status among visible sections. A filtered report states which sources do not support the selected filter.

Every trend carries bucket-level gaps. Lines do not bridge missing buckets. Comparison deltas are unavailable unless both periods use the same metric version and compatible coverage.

## Security and privacy

- Existing collector authentication applies to all report, evidence, saved-view, and export routes.
- Evidence queries are server-issued bounded descriptors; clients cannot pass arbitrary file paths, SQL, commands, or URLs.
- Raw source paths are available only where existing logs authorization already exposes them.
- Export escapes spreadsheet formulas, sets an explicit content type, and contains only fields visible in the current report authorization context.
- Safe-link validation rejects non-HTTP(S) external URLs and malformed internal routes.
- Saved views store filter definitions, not credentials or source contents.

## Error handling

- One failed domain renders its own error and coverage gaps while healthy domains remain usable.
- A query error preserves the last successful report and labels it out of date; it never presents old data as current.
- Empty means no matching recorded facts in a complete window. It is distinct from unavailable or filtered-out.
- Snapshot write failure in Slice 5 does not block the live report; comparison becomes unavailable with the failed bucket named.
- Export failure leaves the page intact and presents one actionable error.

## Performance bounds

- Live ranges are bounded to seven days before Slice 5; longer ranges require report history.
- Evidence reads are paginated and capped; the drawer loads detail on demand.
- The projection fans into bounded source readers concurrently and caches identical report queries for a short collector-defined freshness interval.
- Aggregation is server-side; the browser receives bounded series, breakdowns, and records.
- Every section reports truncation as partial coverage; silent top-N truncation is forbidden.

## Testing

### Contract and projection

- Query parsing: timezone, repeated filters, bounds, invalid ranges, and default 24-hour window under a fake clock.
- Coverage lattice: complete, partial, stale, unavailable, mixed authority, skipped records, unsupported filters, and bucket gaps.
- Metric definitions: numerator/denominator, zero denominator, units, comparison compatibility, and no value when evidence is insufficient.
- Correlation: explicit identifiers join; title/time/prose similarity never joins.
- Partial failure: one source error preserves healthy sections and names the gap.
- Security: bounded evidence descriptors, no path injection, export formula escaping, content types, and safe links.
- Snapshot store in Slice 5: idempotency, schema versioning, atomic write, corruption quarantine, retention, and coverage-preserving backfill.

### Web feature

- URL state round-trip and back/forward behavior.
- Every delivered filter changes the authoritative query and visible report.
- Section states: loading, complete, empty, partial, stale, unavailable, refetch error with old data.
- Summary values, units, denominators, and coverage match the response without client recomputation.
- Keyboard-selectable measures and rows open the correct evidence drawer.
- Drill-down links preserve supported context and reject unsafe URLs.
- CSV and JSON exports match the rendered query and include coverage.
- Both themes, narrow viewport containment, focus order, accessible names, and reduced motion.

### Installed owner-flow proof per slice

Each slice creates authoritative fixtures through supported collector seams, enters `/reports` through sidebar navigation, exercises every new filter and drill-down, asserts the new visible output and forbidden inferred output, then repeats against the installed runtime from `origin/main`.

Proof matrix:

1. **Slice 1:** inject healthy, stale, skipped, and failed activity-source records; verify source trust and evidence drill-down.
2. **Slice 2:** record a request and explicitly linked successful and failed runs; verify funnel, gate/tool/diff detail, delivery proof, and an unlinked row labeled partial.
3. **Slice 3:** record opened/resolved and unresolved incidents plus recurring failure keys; verify recovery only for paired lifecycle records.
4. **Slice 4:** record attributed and unattributed usage; verify cost/account/host breakdown and unavailable guessed cost.
5. **Slice 5:** create versioned buckets with one gap; verify 7/30/90-day comparison, unbridged gap, restart persistence, and incompatible-version suppression.
6. **Slice 6:** trace one complete and one incomplete request lineage, save/reopen a view, and verify CSV/JSON content against the visible query.

Follow `/od-testing`: focused behavior tests, static checks, mandatory broad suites once, land/deploy, then installed owner-flow proof. A route smoke test or component-existence assertion never satisfies acceptance.

## Architecture decisions

- **Adopted incremental hybrid:** live projections ship first; historical persistence begins only with the comparison slice.
- **Kept one report projection boundary:** calculations and honesty rules must not diverge between UI, export, and history.
- **Kept domain sections inside one versioned envelope:** each domain fails independently while global filters and coverage remain coherent.
- **Kept evidence lookup separate from the report envelope:** deleting it would force unbounded raw rows into every report; its bounded descriptor hides authorization and source-reading complexity.
- **Delayed the history store:** before Slice 5 it has one speculative consumer and fails the single-adapter/YAGNI test.
- **Rejected browser-side joins:** they scatter metric definitions, expose source-shape churn, and make partial coverage ambiguous.
- **Rejected a copied raw-event warehouse:** it duplicates sensitive evidence and creates retention/backfill obligations before a visible need.
- **Rejected inferred correlations and recovery:** proximity and prose similarity cannot support owner-facing claims.
- **Rejected placeholder sections:** future slices remain absent until their data and owner flow are real.
- **Retained focused pages:** reports explain and correlate; focused pages intervene and inspect raw subsystem detail.

## Incremental completion definition

The feature is complete only after all six slices are on `origin/main`, deployed, and proven through their installed owner flows. Until then, the page names itself by delivered capability and renders no controls, cards, comparisons, or domains whose contract is not yet live. Every slice report states what became usable and what remains intentionally absent.