# E2E reconciliation model observations — 2026-07-15

audience: AI coding agents first.

## Scope

Use these observations only for legacy E2E classification/reconciliation: read test source, product source, journey contracts, sibling coverage, then select explicit disposition without inflating journey ownership.

Do NOT treat this as controlled cross-model benchmark yet. Luna/low received 75 specs; Terra/low received five. Batch size and prompt differed. Record measured behavior; assign no comparative pass rate.

## Observations

| Config | Batch | Result | Routing decision |
|---|---:|---|---|
| `gpt-5.6-luna/low` | 75 | REJECT. Replaced source review with filename-substring classification; risk gate passed while evidence remained unauditable. Follow-up repair packet moved 27 unsafe specs into generic fixture debt without source repair and erased precise blockers. | MUST NOT own autonomous reconciliation/classification. May extract inventories or propose candidates; reviewer MUST reject counter-only closure and category laundering. |
| `gpt-5.6-terra/low` | 5 | PARTIAL. Read individual tests and product surfaces; produced defensible non-journey classifications and explicit rationales. Invented/inflated replacement equivalence for several specs; added pilot-specific plumbing; validator checked non-empty evidence, not truth. | MAY produce 5–10-spec first pass. Reviewer MUST verify replacement equivalence, consolidation choice, referenced coverage, and canonical-manifest integration. |
| `gpt-5.6-terra/medium` | same 5 | PASS for analysis/recommendation. Distinguished repair, migration, consolidation, and deletion instead of one status; proved one exact duplicate by named case; returned `null` for absent replacements; identified stale route; produced concrete matrix row and missing runner states; separated spatial UI coverage from temporal/durable journey invariants. | PREFERRED reviewer for 5–10-spec reconciliation batches. Still require root verification before edits/deletes. |

## Terra low→medium capability comparison

Use observed semantic failures for seat routing. Do not use these runs for token, speed, or statistical pass-rate comparison because execution mode and repository state differed.

## Qualitative capability evidence

Medium output contained stronger semantic judgment:

- Corrected false replacement claims from low.
- Found exact checkout-background duplicate: `checkout-modal-bg-live-run.spec.ts` case `checkout modal has solid background on live`.
- Selected `delete` for obsolete/hollow deal-detail probe instead of preserving it as generic UI coverage.
- Selected matrix migration only for purchases UI; specified `viewport`, `selected`, and `current` runner capabilities.
- Kept admin-table integration and burger portal interaction outside element-state rows; routed reusable mechanics to component contracts.
- Detected current `UJ-012-single-deal-purchase.spec.ts` trigger/contract gaps instead of treating it as complete replacement authority.

Observed run telemetry:

| Config | Mode | Tokens | Wall | Comparability |
|---|---|---:|---:|---|
| Terra/low | audit + edits + validation | 177,695 | 285s | semantic output comparable; cost not comparable |
| Terra/medium | read-only audit | 137,542 | 216s | semantic output comparable; cost not comparable |

Do NOT infer medium is cheaper/faster. Low performed writes and gates; medium did not. Add no graphical Pareto point from these runs.

## Task discriminator

Require all four checks:

1. Source-specific evidence names test behavior and product contract.
2. Journey ownership requires exact documented branch plus visible, durable, persistence, and forbidden-effect oracles.
3. Replacement claim names exact owning test/matrix row and proves coverage equivalence.
4. Classification lives in canonical manifest; no filename heuristic, default bucket, inline-tag authority, or batch-specific generator branch.

Reject:

```text
if path includes "refund" => journey-supplement
```

Accept only evidence-shaped record:

```json
{
  "path": "apps/web/tests/example.spec.ts",
  "classification": "out-of-journey-ui",
  "sourceEvidence": "exact behavior asserted",
  "productEvidence": "exact route/component contract",
  "coverageDisposition": "retain|repair|consolidate|delete",
  "replacement": { "path": "...", "case": "...", "equivalentAssertions": ["..."] }
}
```

Omit `replacement` when no equivalent owner exists. Never use aspirational replacement paths.

## Controlled comparison protocol for cost/statistics only

Run low and medium from byte-identical repository snapshots on same unseen five-spec batch. Use identical prompt, write permissions, tools, and gates. Change only `model_reasoning_effort`. Blind-score source accuracy, disposition, replacement equivalence, matrix fit, unsupported claims, source diff, and validation. Add chart records only after scoring rubric yields comparable pass counts.

## User routing decision — capability verdict VALID

Attempted to reuse existing Terra/low result and run only Terra/medium from reconstructed pre-low worktree. Reconstruction failed equivalence: mutable journey docs now produce 678 documented branches, while Terra/low's original run saw 685 and passed canonical risk gate. Terra/medium stopped fail-closed on `documented branch count changed: 678 != 685` before completing same write-and-validate task.

Capability routing does not depend on 678-versus-685 denominator. Terra/low already failed required semantic-quality bar on same five-spec class: false replacement equivalence, generic status preservation, and batch-specific plumbing. Additional documented branches cannot repair those observed errors. Terra/medium produced materially stronger source-specific disposition, consolidation, deletion, matrix-fit, and journey-boundary judgment.

Route this seat to `gpt-5.6-terra/medium`; do not spend another Terra/low run.

Keep cost/speed comparison unmeasured. Starting repository state and completed gate work differed, so tokens and wall time remain non-comparable.

Optional statistical experiment only:

1. Freeze one repository snapshot.
2. Clone snapshot twice.
3. Run Terra/low and Terra/medium once each with byte-identical prompt and permissions.
4. Compare diffs and gates from those frozen clones.

No further experiment required for seat routing. Run frozen-clone A/B only when exact pass-rate or cost frontier data is needed.