# Feature Correctness + UI Oracle Findings Report

Audience: AI coding agents first.

## Verdict

Do NOT create green baseline from current behavior. Current behavior is evidence, not truth. Build reviewed oracle registry from invariant, contract, implementation, history, and runtime evidence; unresolved claims stay `UNKNOWN`, never green.

Automate discovery, clustering, execution, and contradiction detection. Require human decision only for irreducibly subjective intent or conflicting business rules. Reduce 150 pages × 4 roles to representative page-archetype/state clusters for review; retain full automated sweeps.

## Scope Finding

Repository in scope does not match described application:

- Tracked tree contains no Playwright configuration and no Playwright dependency. `mega-plan-harness/web/package.json:5-20` defines Astro build/check plus four source-level tests.
- UI has six Astro page files, not 150+ product pages: `mega-plan-harness/web/src/pages/`.
- No four-role authentication model is visible in UI sources searched.
- Repository contains two UI roots, `web/` and `mega-plan-harness/web/`; route ownership is not singular.
- Existing UI tests mainly regex-match source text. Example: `mega-plan-harness/web/test/overview.test.ts:21-67`. These prove implementation tokens exist, not browser behavior or feature correctness.

Therefore: recommendations below are project-specific to observed harness architecture where evidence exists, and target-app architecture where user supplied facts define scope. Actual 150-page/4-role inventory cannot be certified from this repository.

## Critical Findings

### F1 — Summary's core oracle remains circular

Deriving contracts from code, then testing code against those contracts, proves internal consistency only. “Test passes → contract is real” is false. Passing test proves observed implementation matches asserted oracle under tested conditions.

Required separation:

1. `EVIDENCE`: code, stale specs, tests, history, production behavior, support incidents.
2. `CLAIM`: one precise expected outcome.
3. `AUTHORITY`: invariant, external contract, reviewed product decision, or unresolved inference.
4. `VERIFICATION`: executable test linked to claim.

No authoritative source → claim remains `UNKNOWN`; implementation may not self-promote it to `CONFIRMED`.

### F2 — Majority UI is anomaly detector, not oracle

Dominant navbar fingerprint can reveal outliers. It cannot decide correctness: shared systemic bug may dominate all pages. Never auto-accept majority cluster.

Use majority only to create review packet:

- dominant structure;
- exact outlier diffs;
- implementation ownership/import path;
- route archetype;
- role/locale/state applicability;
- historical introduction commit.

Reviewed page-archetype contract becomes oracle. Majority never does.

### F3 — “Green baseline” currently means hiding known defects

Snapshotting buggy output converts defects into approved behavior. Correct baseline requires discovery run, finding disposition, systemic fixes, then explicit baseline approval. Baseline update must be review-only and provenance-bearing.

### F4 — Route × role × locale × viewport misses UI state dimension

Most subtle defects exist between states, not pages. Required cell:

`route archetype × authorization class × locale direction × viewport class × data state × transition`

Minimum data states:

- SSR/initial;
- loading/skeleton;
- empty;
- populated-minimal;
- populated-dense;
- partial/degraded;
- error;
- stale/reconnect;
- optimistic mutation;
- settled after reload.

Minimum transitions:

- cold navigation;
- client navigation;
- hydration;
- data arrival;
- retry/reconnect;
- mutation success/failure;
- back/forward restoration;
- viewport/font completion.

### F5 — CLS `< 0.1` is insufficient

Route-level CLS misses local instability, input-adjacent shifts, scroll jumps, container collapse, focus movement, and visually disruptive swaps whose total score stays below threshold.

Record both:

- browser CLS entries with sources and `hadRecentInput` classification;
- semantic element geometry timeline: shell, nav, heading, primary action, content frame, list/table rows, footer, focused element, scroll position.

Fail on contract violations, not only aggregate score. Example: navbar bounding box MUST remain stable after first committed shell paint; reserved content frame MUST NOT shrink/grow when data arrives beyond declared tolerance.

### F6 — Screenshot diff alone cannot establish intent

Pixel snapshots are strong regression detectors after approval, weak discovery oracles before approval. Dynamic content, fonts, clocks, animation, GPU rasterization, and data ordering create noise.

Use three independent signals:

1. semantic structure fingerprint;
2. geometry/timeline contract;
3. visual screenshot diff.

Two signals agreeing raises confidence; disagreement produces triage packet.

### F7 — Zero failed requests/console errors needs taxonomy

Expected aborts, canceled navigation, deliberate 401/403 branches, telemetry failure, and browser noise must not equal product failure. Unclassified failures must fail. Explicit expected-failure rules need method, URL pattern, state, status, and reason; global ignore patterns forbidden.

### F8 — Current harness structurally invites page drift

Observed pages independently emit `<html>`, `<head>`, body styling, and page framing:

- `mega-plan-harness/web/src/pages/index.astro:5-14`
- `mega-plan-harness/web/src/pages/adapters.astro:5-74`
- `mega-plan-harness/web/src/pages/okf.astro:267-288`
- `mega-plan-harness/web/src/pages/runs/[runId].astro:8-18`
- `mega-plan-harness/web/src/pages/runs/[runId]/agents/[agentId].astro:8-27`

No shared shell owns metadata, navigation, typography, content width, or background. Cross-page drift is architectural, not primarily test debt.

### F9 — `client:load` makes transient correctness first-class

Observed primary surfaces hydrate client-side (`index.astro:13`, `adapters.astro:71`, `runs/[runId].astro:16-17`, `okf.astro:275-285`). This does not prove layout shift, but creates mandatory initial→hydrated transition coverage. Source inspection cannot certify it; browser geometry probe is required.

## Recommended Correctness Architecture

### R1 — Create oracle registry, not another test catalog

One record per material claim:

```yaml
claim_id: FC-ORDER-017
subject: checkout.finalization
preconditions: []
trigger: exact actor action
must: []
must_not: []
authority: invariant|external-contract|reviewed-decision|unknown
evidence: [{path, lines, kind, supports_or_conflicts}]
confidence: confirmed|disputed|unknown
tests: [{id, layer, last_result}]
owner: product-area
decision: {status, reviewer, date, rationale}
```

Tests reference `claim_id`. Specs reference same ID. Deleted test, route, or baseline cannot silently delete claim.

### R2 — Build feature map from effects, not pages

Discover features by authoritative effects:

- state-machine transitions;
- database writes and ownership fields;
- money/inventory/entitlement changes;
- external calls;
- queue/cron/webhook consumers;
- notifications;
- authorization decisions.

Join UI triggers and API entrypoints onto effect graph. Hundreds of pages collapse into materially fewer domain capabilities and transitions.

### R3 — Triangulate stale truth

Authority order:

1. legal, accounting, security, and mathematical invariants;
2. external protocol/contracts;
3. explicit reviewed product decisions;
4. current production behavior and incident evidence;
5. specs, history, tests, and implementation as non-authoritative evidence.

Contradiction packet contains exact competing claims plus blast radius. Agent recommends one decision; user adjudicates only unresolved product intent. Update registry first, then implementation/spec/tests atomically.

### R4 — Test oracle strength

For each confirmed claim, require strongest applicable layers:

- property/invariant tests across generated inputs;
- model-based state-transition tests;
- metamorphic tests where exact output is unknown but relations are known;
- contract tests at external boundaries;
- outcome E2E with fresh-read durable state;
- concurrency, retry, replay, and idempotency tests;
- fault injection for downstream/async failure;
- mutation testing to prove assertion kills plausible defects.

Mutation score belongs per claim/risk class. Test count and route coverage remain non-correctness metrics.

## Recommended UI Architecture

### R5 — Define page archetypes and shared shell first

Create reviewed contracts for 5–12 archetypes, not 150 individual pages: public content, authenticated app, list/table, detail, form/editor, dashboard, checkout, reduced-chrome flow, error/offline.

Each contract owns:

- shell/navigation/footer;
- content grid and max width;
- landmarks and heading hierarchy;
- responsive breakpoints;
- role-dependent slots;
- loading-space reservation;
- permitted exceptions.

Enforce one shared shell implementation plus static import/architecture check. Screenshot tests then verify rendering; they do not compensate for duplicated navbar/auth code.

For current harness, first architectural target is shared Astro layout wrapping all non-demo pages. `viz-demo.astro` may remain explicit exception.

### R6 — Use state laboratories at component seams

Every data-bearing shared component gets deterministic fixtures for all states listed in F4. Render state laboratory/gallery in browser and measure it once. Page sweep asserts correct component + state wiring; it does not repeat primitive mechanics 600 times.

Required stability contracts:

- skeleton and settled component share reserved outer geometry;
- loading exit removes spinner/skeleton;
- empty/error states preserve page frame;
- image/media dimensions known before load;
- async list growth occurs below stable anchor or via explicit user action;
- focus and scroll position remain valid through transitions.

### R7 — Build two visual lanes

`PR lane`:

- changed components;
- changed route archetypes;
- representative role/locale/viewport covering array;
- deterministic fixtures;
- semantic + geometry + screenshot gates.

`nightly/full lane`:

- full route manifest;
- all authorization classes;
- cold/warm cache;
- throttled network;
- dense/empty/error datasets;
- cross-browser subset;
- anomaly clustering and contact sheets.

Do not run Cartesian product on every PR. Use equivalence classes and pairwise covering arrays; full sweep catches manifest omissions nightly.

### R8 — Establish real baseline through adjudication

1. Validate harness: auth fixtures, dynamic-route expansion, deterministic data, font/image readiness, route ownership.
2. Run detector-only census with no screenshot acceptance.
3. Cluster by shared component, shell, root cause, and visual similarity.
4. Fix systemic defects before reviewing pixels.
5. Generate one representative contact sheet per cluster plus outliers.
6. Review residual subjective decisions; record archetype/component contract.
7. Generate snapshots from fixed reviewed state.
8. Freeze baseline; CI forbids unreviewed updates, missing cells, and silent deletion.

This reduces manual review from every page/role cell to representative clusters and exceptions. Zero human review is impossible where intended appearance was never decided.

## Detector Set Missing From Summary

- unexpected scroll-position change;
- focus loss or focus moved behind overlay;
- content obscured by sticky/fixed chrome;
- breakpoint discontinuity and responsive overlap;
- text truncation/wrapping orphan;
- font-swap geometry shift;
- image intrinsic-size/aspect-ratio shift;
- overlay stacking, clipping, and scroll-lock leak;
- skeleton/settled geometry mismatch;
- empty/error state frame collapse;
- SPA navigation shell remount/flicker;
- back-forward cache restoration defect;
- stale data flash or wrong-user data flash;
- optimistic rollback geometry/state defect;
- duplicate landmark/ID and broken heading hierarchy;
- tap-target collision and pointer obstruction;
- viewport safe-area/virtual-keyboard obstruction;
- theme/locale direction token drift;
- animation causing persistent movement or reduced-motion violation;
- table column/header misalignment and late width redistribution.

## False-Green Guards

- NEVER auto-update screenshots in CI.
- NEVER accept current dominant cluster as intended design.
- NEVER mark `UNKNOWN`/blocked cell passed.
- Fail when route/role/state cell disappears without reviewed manifest change.
- Fail when baseline file disappears or changes without approval metadata.
- Keep finding IDs stable across reruns; closed finding requires fix evidence or accepted-exception record.
- Seed known UI and domain mutants; detector/test suite must catch them before trust.
- Track harness failures separately; they cannot downgrade product findings to pass.
- Report coverage by confirmed claims and transition/state coverage, never scalar test count.

## How Codex Resolves This

Given correct target repository, Codex can own:

1. Inventory route, role, effect, state-machine, async, and shared-component graphs.
2. Mine specs/history/tests/runtime evidence into claim registry with conflicts and confidence.
3. Produce ranked contradiction packets and one recommended resolution each.
4. Implement deterministic fixtures and state laboratories.
5. Implement semantic, geometry, CLS, network, console, accessibility, and screenshot collectors.
6. Run census; eliminate harness noise; cluster findings by root cause.
7. Fix systemic implementation defects before baseline capture.
8. Present only irreducible product/design decisions for human adjudication.
9. Encode decisions into oracle registry, archetype contracts, tests, and immutable baselines.
10. Verify with mutants/fault injection; ratchet claims and visual cells thereafter.

Codex cannot truthfully decide subjective product intent absent evidence. Codex can reduce that burden to a small, evidence-rich decision queue and prevent every resolved decision from becoming ambiguous again.

## Recommended Priority

1. Confirm target repository and canonical UI root.
2. Create effect-based claim/oracle registry for highest-risk domain paths.
3. Create shared shell + page-archetype contracts.
4. Add state laboratories and transition geometry instrumentation.
5. Run unbaselined census; fix systemic clusters.
6. Approve residual visual clusters; freeze baselines.
7. Add PR covering-array lane, nightly full lane, mutation calibration, and deletion ratchets.

## Acceptance Criteria For Future Implementation

- Every correctness test names confirmed `claim_id` and authoritative oracle.
- Every mutable journey asserts visible, durable, fresh-read, forbidden, retry, and ownership outcomes where applicable.
- Every page maps to one archetype or reviewed exception.
- Every data-bearing component covers initial/loading/empty/populated/error/stale states.
- Shared shell does not remount or change geometry across route transitions unless contract allows it.
- Detector suite catches seeded navbar drift, spinner swap, image-size shift, overflow, focus loss, stale-data flash, and missing authorization.
- Screenshot baseline contains approval provenance and cannot update automatically.
- Full manifest reports zero unclassified cells; `UNKNOWN` remains red/blocked.