# CI Failures — findings ledger

Live working document. Updated as work proceeds. Written to survive context compaction.

Repo: `multideal` · worktree `/home/user/Projects/multideal/.worktrees/w1-ci-coverage` · branch `wt/w1-ci-coverage` → pushes to `main`
Started: 2026-07-26 · Last updated: 2026-07-27 07:40

> **REVISION 2.** Revision 1's headline verdict ("the pipeline is structurally incapable of
> emitting a pass") was **wrong and is retracted** — see §0.1. It was falsified by data in this
> same document. Corrected below.

---

## 0. Verdict (corrected)

**The fast gate stopped running on 2026-07-22.** `.github/workflows/test.yml` — four fully-parallel
jobs, no `needs:` chain, ~15 min — triggers only on `pull_request`. The trunk flip to direct-push-
to-main on **2026-07-20** removed PRs and orphaned it. From that date the *only* gate on every
commit was the 82-minute aggregate, which `cancel-in-progress` killed on the next push.

That is the whole answer to "why did CI never allow a clear pass," and it is a dated, verifiable,
six-line defect — not "months of redundant patches."

Secondary and true: **most** reds were CI artifacts. But **not all** — CI also caught at least one
genuine production-cutover blocker (§3). The honest framing is: *green was invisible and rarely
reachable, not impossible; and we spent two weeks reading a broken signal instead of fixing the
signal.*

### 0.1 What Revision 1 got wrong (retracted)

Revision 1 said the pipeline had "no reachable green state." **Run `30220029756` (`e9a17742b`,
2026-07-26 20:56) was a full clear pass on a `push` — all 14 work jobs green**, including factory
journeys (55 passed, 28.7m) and both browser lanes. Revision 1's own §1 table recorded that success
and the prose ignored it. That is rationalization, and it was caught by review, not by me.

Trajectory across 2026-07-26 was 4 reds → 3 reds → 0 reds → 1 cause. **Converging, not broken.**

---

## 1. Evidence: run statistics

Last 40 `main-validation.yml` runs:

| Outcome | Count |
|---|---|
| success | 1 (run `30220029756`, sha `e9a17742` — **a genuine full pass**) |
| failure | 8 |
| **cancelled** | **29** |
| in flight | 2 |

Wall clock of the success: **82 minutes**. Commit interval during active work: **5–20 minutes**.

Job durations on that run: browser lanes 36m each · factory journeys 34m · oracle 13m ·
typecheck+unit 8m · UI matrix 7m · locale parity 6m · integration-db 5m · sweep 4m · smoke 4m ·
tooling 3m · realpg 2m · integration-http 1m. **Green baseline aggregate: skipped.**

---

## 2. Structural defects — corrected severity

| # | Claim | Status |
|---|---|---|
| 2.1 | `green-baseline` gated to `workflow_dispatch`, so a push never emits a green verdict | **Overstated.** Real harm is narrower: the one fully-green run produced **no receipt**. Green was *invisible*, not unreachable |
| 2.2 | `if: always()` on 14 jobs ⇒ cascades masquerade as bugs | **Not supported.** The reds examined were independent causes, not a cascade. `if: always()` is still wrong (no fail-fast, wasted runner time) but it did not manufacture the failures |
| 2.3 | 82-min runtime + `cancel-in-progress` on push ⇒ 29/40 cancelled | **Correct, well-evidenced. The primary defect.** |
| 2.4 | Bookkeeping/ratchet manifests fail on drift, not defects | **Asserted, not evidenced.** Zero runs in the dataset failed on manifest drift. Do NOT delete on this basis |
| 2.5 | 6 runners online but jobs hand-serialized into two chains as if 2 | **Correct.** `multideal-debian1{,-2,-3}`, `multideal-debian2{,-2,-3}` |

---

## 3. The four-lane signature — root-caused, and it was NOT one shared cause

Revision 1 assumed one shared infra cause behind `unit + realpg-replay + locale-parity +
factory-journeys`. **Wrong — four independent causes.** `realpg` was red in `30199485432` and
*green* in `30204189072`; the signatures were never identical.

| Run | Lane | Actual cause | Kind |
|---|---|---|---|
| 30199485432 | unit | `tests/unit/tooling/test-lanes.test.ts` — expected 4 lanes, got 3 | bookkeeping |
| 30199485432 | realpg | `PostgresEarlyExitError` — Unix socket path 112 bytes > 107 limit under `RUNNER_TEMP/job-tmp/...` | **runner-environment defect** |
| 30204189072 | unit | `i18n/languages.cache.test.ts` failed to load: `PUBLIC_LAUNCH_STATE must be "prelaunch" or "live"`. 2271 passed, **0 assertions failed** | **product defect** |
| 30204189072 | locale-parity | `type "account_state_user" already exists` | **product defect** |
| 30204189072 | factory | 3 real journey failures + hundreds of `RangeError: Responses may only be constructed with status codes 200–599` from the app server | **product defect** |

**Two were defects in deliverable code:**

- **`PUBLIC_LAUNCH_STATE`** — a fail-closed validator added to `apps/web/src/server/env.ts:265-268`
  without updating `apps/web/vitest-mocks/cloudflare-workers.ts` or the `wrangler.toml` var blocks.
- **locale-parity IS the production migration rehearsal.** `apps/web/scripts/ci-locale-pair-seed.ts:335`
  calls `applyMigrationChain(pool)`. Its red was the production provisioning path failing against an
  already-provisioned database — exactly production's shape (cf. memory
  `lessons_migration_chain_empty_only_idempotency`). **CI caught a real cutover blocker.**

⇒ This directly refutes a blanket "it's all bad CI." Keep locale-parity blocking before any tag.

---

## 4. The smoking gun (unchanged — this part held up)

`apps/web/tests/a11y/pages.a11y.spec.ts` scanned `/checkout`. `apps/web/src/pages/checkout/`
contains only `[purchaseId]/complete.astro` — **there is no `/checkout` page.** The test passed only
because a middleware defect rendered `pages/[locale]/index.astro` for any unmatched single-segment
path. The test therefore (1) gave zero checkout coverage and (2) **encoded a production bug as the
expected result**. Fixing the bug turned it red: the CI red was evidence the fix worked.

---

## 5. Real product bug found (by reading code, not by CI)

`apps/web/src/server/middleware/locale-prefix.ts:30` — `const LOCALE_PATH_RE = /^\/([a-z]{2})\//;`
matches only 2-letter segments, so any 3+ char first segment bypassed locale validation, fell
through to `pages/[locale]/**`, defaulted to `he`, and returned **200**. `/banana`, `/zzz/deals`,
`/kiwi/deals`, `/papaya` all served real pages. Unbounded duplicate content; violates hard rule 10.
Live SEO harm zero — prod sends `x-robots-tag: noindex, nofollow`.

Fix: `locals.localeValidated`, set only when the middleware accepts the segment; six
`pages/[locale]/**` routes 404 without it. Rewrites re-run the middleware and the second pass sets
the flag, so both rewrite paths survive.

---

## 6. Landed

| SHA | What |
|---|---|
| `58427d2e4` | Locale validation fix + `navigation.smoke.spec.ts` regression test (7×404 / 3×200) |
| `fec1bb0c6` | Delete the dead `/checkout` a11y scan |
| `ffcd6fc90` | `complete.astro`: Stripe return without `payment_intent` → `/purchases/<id>` (was a 404 on the money path) |
| `3a3403667` | **Fast gate runs on push to main; aggregate moved to nightly** |
| `066c0ef0e` | **Release gate: tag deploys now blocked on typecheck+unit+provenance, money-path DB lanes, migration replay; releases serialize** |
| **tag `v0.1.2`** | **Production release — deploy run `30227760825` SUCCESS** |

**Verified on production after v0.1.2:** `/kiwi/deals` 404 · `/papaya` 404 · `/` 200 ·
`/deals` 200 · `/en/deals` 200 · `/he/deals` 200. The duplicate-content defect is gone from prod.

**Fast gate proven live:** run `30227857516` triggered on push `3a3403667` — the first push-
triggered fast-gate run since 2026-07-22.

All: typecheck 0 errors / 0 warnings / 0 hints; eslint `--max-warnings=0`; gitleaks clean.
Unit suite on current main: **exit 0**.

---

## 7. Live environment state

| Route | preview `dev.multi.deal` | prod (before v0.1.2) |
|---|---|---|
| `/kiwi/deals` | 404 ✅ | 200 ❌ |
| `/papaya` | — | 200 ❌ |
| `/deals`, `/en/deals`, `/he/deals` | 200 ✅ | 200 ✅ |

`deploy-prod.yml` triggers on `push: tags: ['v*']`. **Production has never been gated by CI.**
The red pipeline was not what blocked the release.

---

## 8. Rebuild — corrected shape: gate the RELEASE, not the push

Revision 1 proposed demoting factory journeys / browser / oracle / locale-parity to nightly
non-blocking. **On a live marketplace handling real money that is reckless** — a checkout, refund,
or payout regression would land, deploy, and surface up to 24h later.

1. **Blocks every push, <15 min** — revived `test.yml` (**DONE**, `3a3403667`).
2. **Blocks the tag** — **DONE**, `066c0ef0e`. Three parallel gate jobs in `deploy-prod.yml`:
   typecheck+unit+tag-is-ancestor-of-main provenance; cart-realpg + integration-db money lanes;
   migration replay on disposable Postgres (the production cutover rehearsal). Plus
   `concurrency: deploy-production, cancel-in-progress: false` so releases serialize.
   Applies from `v0.1.3` onward — `v0.1.2` shipped through `deploy-prod.sh`'s own gates.
3. **Nightly + dispatch, non-blocking** — the full aggregate (**DONE**, `3a3403667`).

**Tier 3: quarantine, do not delete.** `branch-coverage-baseline` and `oracle-certification` on a
payments codebase are regression ratchets. False-positive rate is unmeasured. Make them
non-blocking for two weeks, then decide with data. Deletion is irreversible.

Promotion rule: a lane moves back to blocking only after repeated passes on unchanged code.

---

## 9. Open / unresolved

- ~~`PUBLIC_LAUNCH_STATE` mock/wrangler drift~~ — **CLOSED, verified on current main.** Validator
  `env.ts:265-268`; mock `vitest-mocks/cloudflare-workers.ts:42`; `wrangler.toml` lines 262 and 455.
- ~~`RangeError: Responses may only be constructed with status codes 200–599`~~ — **CLOSED.** Cause
  was launch middleware constructing an invalid `101`; fixed by `b81114aa5`. Verified on current
  main: WS upgrades are passed through (`if (response.status === 101) return response`) at five
  middleware sites instead of being reconstructed.
- **Socket-path-length runner defect** (`.s.PGSQL.*` path 112 bytes > 107 limit under
  `RUNNER_TEMP/job-tmp/...`) — still UNVERIFIED. It is deterministic, environment-dependent, and
  never reproduces locally. It now sits inside the **release gate** (migration replay), so it can
  block a tag. Watch the first `v0.1.3` gate run.
- **`.gitignore` vs tracked test files — deliberately NOT changed.** `apps/web/tests/a11y/`,
  `tests/e2e/smoke/` etc. are ignored while their files are tracked, which prints a benign
  `lint-staged [FAILED] paths are ignored` on every commit (verified benign: committed blob is
  byte-identical to worktree). Those same directories are documented in `.gitignore` as holding
  LOCAL fixtures with **real credentials** and `.auth-state.json`. A negation pattern risks staging
  secret-bearing files. Cosmetic noise is the correct trade here; do not "fix" it with a broad
  un-ignore.
- **Quarantine the ratchets** (`branch-coverage-baseline`, `oracle-certification`,
  `expectedTests` counts, the 632-entry `spec-inventory.json`) as non-blocking for a two-week
  measurement window, then decide with data. Load-bearing and to be KEPT: `risk-catalog.yaml`,
  `risk-map.yaml`, the suspicious-pattern scanner, money-path spec selection,
  `execution-result.schema.json`.
- Neon `branches_limit_exceeded` seen in CI ⇒ missing quota/reaping control.
- `smoke` hits mutable `dev.multi.deal`, not the checked-out SHA — it validates an environment,
  not the commit.
- 6 runners are 6 processes across 2 hosts; unbounded parallelism can oversubscribe them.
- `.gitignore` ignores `apps/web/tests/a11y/`, `apps/web/tests/e2e/smoke/` while those files are
  **tracked** ⇒ recurring `lint-staged [FAILED] paths are ignored` on every commit. Verified benign
  (committed blob byte-identical to worktree) but real drift against project rule 12.

---

## 10. Discipline notes (carry across compaction)

- Land via `bash scripts/land.sh`. Never `--force`, never `--no-verify`.
- Heavy commands via `~/.claude/bin/local-gate --key <key> --mode full -- <cmd>`.
- Never point destructive ops at dev/prod DB. Delete every Neon branch created. Never print secrets.
- RTK truncates `git` output — use `/usr/bin/git`.
- `pnpm typecheck | tail -N` hides the errors line; grep the whole summary.
- Bash tool cwd persists; `dead-advisor/resolve.sh` keys off cwd, so run it from the repo root.
- **Validate workflow YAML before landing** — an invalid file silently never runs (0s/no-log run).
- **Check the data before writing the verdict.** Revision 1 asserted "never green" with a green run
  sitting in its own table.

---

## 11. Public-domain split — LANDED 2026-07-27 (commit 99e3063eb, tag v0.1.3)

Owner directive: until real registration and real payments exist, `www.multideal.co.il` must show
the marketing pre-registration site; the app moves to `app.multideal.co.il`.

**Live now (probe-verified):**

| Host | Serves | Mechanism |
|---|---|---|
| `www.multideal.co.il` | marketing | Worker `multideal-landing` (static assets), custom domain |
| `multideal.co.il` (apex) | marketing | Pages project `multideal-www` |
| `www.multi.deal` | marketing | Pages project `multideal-www` |
| `app.multideal.co.il` | app (`noindex, nofollow`, `PUBLIC_LAUNCH_STATE=prelaunch`) | Worker `multideal`, custom domain |

Canonical on every marketing host is `https://www.multideal.co.il` (was `multideal-www.pages.dev`
on all of them — a real SEO defect, now fixed in `apps/landing/src/layouts/LandingLayout.astro`).

**Repo changes:** `apps/web/wrangler.toml` production route + `PUBLIC_SITE_URL`;
`deploy-prod.sh`, `deploy-lib.sh` (route assertion + 4 `DEPLOY_URL` defaults),
`.github/workflows/deploy-prod.yml` `DEPLOY_URL`, `prod-test-surface-smoke.yml` `PROD_URL`,
`scripts/audit-prod-test-surface.mjs`, `tests/unit/prelaunch-noindex.test.tsx`.
`apps/landing/wrangler.jsonc` gained the assets directory + `www.multideal.co.il` custom domain.

**Permission ceiling hit:** the wrangler OAuth token has `zone (read)` only — `POST /zones/*/dns_records`
returns 10000 Authentication error, and even DNS reads are denied. Consequences:
- Deleting the `www` worker custom domain deleted its DNS record and left `www` dark (~10 min).
  Recovery was to bind `www` to a Worker (workers scope creates the record) rather than Pages
  (which requires a pre-existing CNAME and only reports `CNAME record not set`).
- The apex could NOT move to the Worker: it already carries proxied A records that only a DNS-write
  credential can remove (`workers/scripts/.../domains/records` 500s). Apex therefore stays on Pages.
- `~/.cloudflare/api-token` is a stale `KEY=VALUE` file whose token returns 9109 Invalid access token.

**Open, needs a DNS-capable credential:** consolidate the apex onto the landing Worker so one target
serves the whole brand zone. Not blocking — apex serves the right content today.

**BLOCKED on a credential — Stripe production webhook endpoint.** The registered endpoint still points
at `www.multideal.co.il/api/payments/stripe/webhook`, which now returns **404** (verified);
`app.multideal.co.il/api/payments/stripe/webhook` returns **400** on an unsigned body, i.e. the route is
alive and rejecting correctly. `STRIPE_SECRET_KEY` and `STRIPE_WEBHOOK_SECRET` are both set on the
production worker, so an endpoint was registered. No live Stripe credential is reachable from this
machine: no `stripe` CLI, no `~/.config/stripe`, no Stripe keys in `.dev.vars`, `/home/user/Projects/multideal/.env`,
or `Docs/production-secrets-providers.local`; the Stripe MCP requires a browser OAuth grant.

Remediation (needs Stripe dashboard or an authorized MCP session), in order:
1. Delete the `www.multideal.co.il` webhook endpoint.
2. Create `https://app.multideal.co.il/api/payments/stripe/webhook` with the same event set.
3. **Rotate `STRIPE_WEBHOOK_SECRET`** in the production CF binding to the NEW endpoint's signing secret.
   Skipping this is the silent failure mode — the push succeeds and every signature verification fails.
Do NOT re-run `put-production-secrets.mjs` expecting it to move the endpoint: `resolveStripeWebhookSecret`
short-circuits on an existing `STRIPE_WEBHOOK_SECRET`, and with the URL constant now changed it would
CREATE a second endpoint rather than move the first.

**Verified after the v0.1.3 deploy (2026-07-27):** `deploy-production` success; `app.multideal.co.il`
serves the app with `canonical=https://app.multideal.co.il/` and `noindex, nofollow` (the new build,
not v0.1.2); `www.multideal.co.il`, `www.multideal.co.il/en/`, `www.multi.deal`, and the apex all 200
with canonical `https://www.multideal.co.il`; both Pages custom domains report `active` (no lingering
`pending`). Fast gate green on `99e3063eb` (all four lanes) and on `7db4dc95`.

**CI contract tests updated with the new design** (all four fast-gate failures resolved):
`land-contract` (main-validation is scheduled, fast gate is the per-push gate),
`main-validation-neon-capacity-contract` (never-cancel concurrency),
`quality-workflow-contract` (`HEAD~1` push base),
`setup-pnpm-install-contract` + `workflow-contract.mjs` (`deploy-prod.yml` now has 4 setup callers).
Local: `pnpm quality:changed:test` 44/44; `tests/unit/tooling` + `prelaunch-noindex` 153/153;
`pnpm --filter web typecheck` 0 errors.

## 12. Blocker classification ledger — 2026-07-27

The owner's question is "IS THERE A BUG IT'S BLOCKING?", not "IS CI RED?". Every red gets reproduced
locally and classified before it is reported. Running tally:

| Red | Where | Reproduced | Classification | Resolution |
|---|---|---|---|---|
| `Playwright smoke` probing `https://dev.multi.deal` while the same commit was mid-deploy | fast gate (per-push) | yes — by inspection of the job vs `deploy.yml` trigger | **CI defect.** It validated whatever build was live, usually the previous commit. A red was unattributable to the pushed change. | Moved to `deploy.yml` as a post-deploy job pinned to `needs.deploy-preview.outputs.preview_url`, fail-closed if that output is empty. Landed `f25ea41cb`; first run passed against the deployed build. |
| `Tooling contracts` | Main Validation (dispatch on `f25ea41c`) | yes — `pnpm quality:changed:test` 43/44 locally in ~1 min | **CI defect, self-inflicted.** The smoke-location rule exists in TWO copies; the vitest copy (`quality-workflow-contract.test.ts`) was updated, the node-runner copy (`scripts/workflow-contract.test.mjs`) was not. | Second copy updated to the post-deploy shape. 44/44. Landed `f66d46f7a`. |

| `Green baseline aggregate` — `data` phase `executed=1 passed=0 failed=1` in 75s | Main Validation (dispatch on `f66d46f7a`) | yes — env diff, see below | CI defect #3 — harness never provisioned the DB the phase needs | `main-validation.yml` now nests `ci-with-disposable-pg.sh` outside `ci-with-disposable-factory-pg.sh`, each forking from its own Neon parent |
| `Upload green baseline receipt` — `No files were found with the provided path: .green-baseline` | Main Validation (both dispatch runs) | yes — dot-directory, `upload-artifact` default | CI defect #4 — receipts were never uploaded, so no run's phase logs were ever recoverable | `include-hidden-files: true` on both `.green-baseline` uploads (`main-validation.yml`, `ui-sweep-full.yml`) |

| `Changed browser journeys` + `UI sweep ratchet` — `Push payload did not provide a usable base SHA`, both dead in ~60s | Main Validation (nightly `schedule` run `30237359249`) | yes — by reading the step and the `on:` block | CI defect #5 — the step reads `github.event.before`, which only exists on a `push` event. This workflow has NO `push` trigger: only `schedule` and `workflow_dispatch`. So on every nightly run both jobs failed before doing any work, and had done so for as long as the step existed. | Dead branch deleted. Both jobs now always resolve `base="$(git rev-parse "${VALIDATION_COMMIT}^")"`, the same path the dispatch receipt already used. `PUSH_BASE_SHA` env removed; contract test `main-validation-neon-capacity-contract.test.ts` updated to pin the new shape (4/4 green). |

| `Green baseline aggregate` — `bash: apps/web/scripts/ci-with-disposable-factory-pg.sh: No such file or directory`, exit 127 in 3s | Main Validation (dispatch `30248147127` on `cbca4d392`) | yes — from the step log | CI defect #6, introduced by the defect-#3 fix. `ci-with-disposable-pg.sh` does `cd apps/web`, so the nested inner wrapper could no longer be found at a repo-root-relative path. | Inner wrapper invoked as `"$GITHUB_WORKSPACE/apps/web/scripts/ci-with-disposable-factory-pg.sh"`. Landed `5983b6eae`. |

**App bugs found so far: 0. CI defects found so far: 6.**

Coverage caveat on that tally: `Changed browser journeys` and `UI sweep ratchet` are *changed-surface*
lanes — they test what the validated commit touched. Every receipt so far has validated a CI-only
commit (`git diff --name-only f66d46f7a cbca4d392` = two workflow YAMLs), so their selection input
was empty. Measured, not assumed: with that empty surface the lane still ran its always-on baseline
set — `Running 15 tests using 1 worker` / `15 passed (1.2m)` in run `30248147127`. So it is not a
zero-coverage green, but it has never exercised app-change-driven selection on these receipts. The
whole-surface coverage in this workflow comes from `Factory journeys`, `Factory UI matrix` and the
green-baseline aggregate.

### CI defect #3 — evidence

`green:baseline:data` = `typecheck:e2e && test:integration-db`. The `integration-db` lane declares
`requiredEnv: ["DATABASE_URL"]` in `test-lanes.json`, and `vitest.integration-db.config.ts` calls
`assertLaneEnv('integration-db')` at config load, which throws before collection when it is missing.

`ci-with-disposable-factory-pg.sh` exports only `E2E_FACTORY_DATABASE_URL` and `MIGRATE_DATABASE_URL`.
It never exports `DATABASE_URL`, `PII_KEY`, `PASSWORD_PEPPER_V1`, or `ALLOW_LIVE_DB` — those come from
`ci-with-disposable-pg.sh`, which the green-baseline job did not run. So the phase spent ~70s in
`typecheck:e2e`, then vitest aborted in seconds: hence 75s for a suite that takes far longer.

The same suite is green under the correct harness: `bash apps/web/scripts/ci-with-disposable-pg.sh
pnpm --filter web test:integration-db` → 44 files (1 skipped), 308 tests passed, 10 skipped. The
standalone `Integration database lanes` job, which uses that wrapper, passed in the same CI run.
Same tests, two harnesses, one of them unwired. No application defect involved.

The job previously set `NEON_MAIN_BRANCH_ID` to the factory branch, so both wrappers would have forked
from the factory parent. The fix passes each wrapper its own parent: the job keeps
`NEON_CI_MAIN_BRANCH_ID`, and the factory wrapper is invoked through
`env NEON_MAIN_BRANCH_ID="$NEON_FACTORY_BRANCH_ID"`.

Note the duplication class: workflow expectations are asserted in `scripts/workflow-contract.mjs`
(+ its inline copy in `setup-pnpm-install-contract.test.ts`) AND in `scripts/workflow-contract.test.mjs`.
A workflow edit must update all copies or it reds a lane that has nothing to do with the app. This is
the second time in one session this duplication caused a red.

Validation receipts (dispatch-pinned, never cancelled):
- run `30236617215` on `f25ea41c` — carries the known `Tooling contracts` red, fixed by `f66d46f7a`.
- run `30236958996` on `f66d46f7a` — the clean-head verdict.

---

## 13. Speed + redundancy audit — 2026-07-27 (codex gpt-5.6-sol/medium + own verification)

### Measured baseline (run `30248147127`)

Wall clock **53 min**; sum of job time **91 machine-minutes** across two 8-core boxes.
Critical path: locale 5m → factory-journeys 33m → pr-browser 6m → sweep 4m → aggregate 1m.
Fixed host assignment splits the load **debian1 ≈54m / debian2 ≈34m** — the real bottleneck is the
imbalance, not CPU.

The "85 minutes" figure was never the true cost. `green-baseline` re-runs deterministic + data +
journeys + **3× journeys** + ui-matrix + **3× ui-matrix** + slop, serially, in one job, with declared
phase budgets totalling **365 min** behind a 480-min timeout. A genuinely passing aggregate would
make a validation run **3–4 hours**. It has only ever "passed" by being skipped or failing fast.

### Verified redundancy (checked in the files, not taken on trust)

| Finding | Evidence |
|---|---|
| `browser-risk` and `pr-browser` run the **identical command** (`pnpm --filter web test:pr-browser`) | both jobs' run lines are byte-identical; 6 min wasted per run |
| `typecheck` + `test:unit` + `test:realpg` + `test:cart-realpg` run in **both** `test.yml` and `main-validation.yml` | grep of both workflows |
| Nine `needs:` edges are **not data dependencies** — hand-rolled serialisation | e.g. `test-oracle ← test-integration-http ← test-integration-db ← test-realpg ← typecheck-unit`, five independent suites in a 27-min chain |
| …because `cpu-limit.sh` already holds a **machine-wide `flock`** (line 35-42) | so two jobs on one host cannot run heavy work concurrently anyway — deleting the edges alone does NOT reach 27 min |
| `playwright.config.ts:37` hardcodes `workers: 1` | the 33-min journey suite is 39 files / 55 tests, single-threaded |
| checkout + pnpm install repeats **14×** per run | one per job |

### CI defect #7 — 210 test files exist and no workflow has ever run them

`apps/web/tests/test-lanes.json` declares **612** files in the `default` lane. Every workflow runs
`pnpm --filter web test:unit`, which is `vitest run tests/unit` — **402** files. The other **210**
(185 `tests/components`, 21 `tests/api-migration`, 3 `tests/contract`, 1 `tests/integration`) are
run by nothing — not the fast gate, not main validation, **not `deploy-prod.yml`**.

Measured by running them: **210 files, 1331 tests, 2 min 36 s. 209 files pass.** The single failure
is `tests/contract/ci-artifact-gate.test.ts` (5 cases) asserting Main Validation triggers on `push`
— an expectation made obsolete by the deliberate 2026-07-26 change to nightly+dispatch. A stale
meta-test, not an app defect. Cost to close the hole: **~2.6 min**, less than the 6 min wasted by
the duplicate `browser-risk` job alone.

### CI defect #8 — non-deterministic fixture in a money-path test (FIXED, landed `c08508423`)

`Typecheck and unit lane` went red in run `30251813190` on `settlements-consumer.test.ts >
recovers releasing rows with the same key and no balance check`, while the same lane passed locally
(401/402 files, 2289 tests) and the file passed 5/5 in isolation.

Root cause, not a guess: the fixture built `payoutIdempotencyKeyCreatedAt: new Date()` **once per
row inside `.map()`**. `validatePayoutKeyState` (`settlements-consumer.ts:120-131`) collects the
distinct key timestamps and holds the batch when `keyCreatedAtEpochs.size !== 1`. On a loaded
runner the two `new Date()` calls land on different milliseconds → invariant error → the payout is
never created → `expect(create).toHaveBeenCalledWith(...)` sees 0 calls. A latent time-race in the
test, guaranteed to keep flaking exactly when CI is busy. Fixed by hoisting one timestamp per batch
in both fixtures.

**App bugs found: 0. CI/test defects found: 8.**

### CI defect #9 — safe-merge required checks named jobs that no longer exist (FIXED)

`scripts/safe-merge.mjs` required check names `test.yml` stopped emitting after the gate was
consolidated. Every PR merge would have blocked on checks that can never report. Reduced to the
three names the workflow actually emits: `Quality / changed`, `Typecheck / unit`,
`Real-Postgres lanes`.

### CI defect #10 — cpu-limit's caps were silently inert on every CI job (FIXED, landed `e6c9e853b`)

The GitHub runner is a **system** service with no login session, so a job inherits no
`DBUS_SESSION_BUS_ADDRESS`. Every `systemctl --user` / `systemd-run --user` call in
`apps/web/scripts/cpu-limit.sh` therefore failed — **silently** — and the cgroup CPU/memory caps it
exists to apply were never applied. Fixed by recovering the bus address from
`/run/user/$(id -u)/bus`. Two latent bugs found alongside: a nested wrapper tried to start a
transient scope inside another one, and the CI fail-closed guards had no test coverage. Eight
behavioural tests added (`apps/web/tests/unit/tooling/cpu-limit.test.ts`).

### CI defect #11 — a contract test pinned the cache action's major version (FIXED, landed `4fbe48d2`)

`scripts/workflow-contract.mjs` asserted the literal string `actions/cache@v4`. The intent is
"dependency caching is present and lockfile-keyed"; the version pin made a routine, required
upgrade (v4 runs on deprecated Node 20 and GitHub force-migrates it to Node 24, annotating every
job) fail three contract tests. Now matches `actions/cache@v\d+`.

## Fast gate: first green, and where the wall clock actually goes

`e6c9e853b` was the **first fully green Fast gate** of the campaign — all three jobs, whole 612-file
default lane, no skips.

Measured job durations (run 30264496807), all three jobs starting the same second:

| Job | Duration |
|---|---|
| Typecheck / unit | 11.9 min |
| Quality / changed | 4.4 min |
| Real-Postgres lanes | 2.1 min |

They run **in parallel**, so wall clock = the longest job. This kills the change-detection idea:
skipping the two short jobs saves **zero** wall clock, and the long one — typecheck + the unit lane
— can never be skipped. Docs-only pushes are 2 of the last 300 commits and `paths-ignore` already
handles them.

The lever was inside the long job. Step breakdown: setup 1.2 min, typecheck 1.6 min, ledger/risk/
stock < 0.2 min, **default test lane 8.3 min** — running with `BUILD_VITEST_MAX_FORKS: '2'`,
hardcoded for the worst case of three co-resident jobs on an 8-core host.

That worst case is not what happens. The two short jobs finish at 4.4 and 2.1 min; the unit lane
runs its remaining ~7 minutes **alone** on the box, throttled to 2 of the 7 available cores.
Raising the cap to `6` and letting `build.slice`'s 700% aggregate quota absorb the brief three-job
overlap (run 30266238703, green):

| | before | after |
|---|---|---|
| Default test lane | 8.3 min | **4.4 min** |
| Fast gate wall clock | 11.9 min | **7.5 min** |

A load-aware divisor in `cpu-limit.sh` was considered and rejected on evidence: vitest reads
`VITEST_MAX_FORKS` once at startup, and at that instant all three jobs are alive — the divisor would
compute exactly the `2` that was already hardcoded. The load question cannot be answered at the only
moment the tool is listening.

**App bugs found: 0. CI/test defects found: 11.**

### CI defect #12 — a 180-minute validation job that duplicated another one (FIXED, landed `c121f4494`)

`main-validation.yml` had two jobs running the identical command `pnpm --filter web test:pr-browser`
against the identical disposable factory Neon branch on the identical host: `browser-risk` (budget
180 min) and `pr-browser` (60 min). `browser-risk` additionally derived its diff base from
`github.event.before`, which is empty on a schedule or dispatch — so it fell back to `origin/main`
and, running *on* main, compared main against itself. It was a duplicate of `pr-browser` that also
had nothing to diff. Deleted, along with its `green-baseline` prerequisite assertion and its slot in
`notify-failure`. One fewer concurrent Neon branch on debian1.

**App bugs found: 0. CI/test defects found: 12.**

### CI defect #13 — the whole validation ran twice on every dispatch (FIXED)

`main-validation.yml` had two verdict-producers for the same work. Thirteen lane jobs ran, and then
`green-baseline` asserted each one succeeded — and re-executed the same commands a second time,
sequentially, under an 8-hour budget:

| green-baseline phase | its command | lane that already ran it |
|---|---|---|
| `data` | `typecheck:e2e && test:integration-db` | `test-integration-db` |
| `journeys` | `test:factory-journeys-certified` | `factory-journeys` (`test:factory-journeys` — the same script body) |
| `ui-matrix` | `run-factory-e2e.mjs tests/e2e/ui-matrix …` | `factory-ui-matrix` (byte-identical argv) |

`test:factory-journeys` and `test:factory-journeys-certified` are the same script. The lanes were
throwaway pass/fail; only the second run produced the receipt, so the first run's hours bought
nothing but a `needs:` edge.

The fix uses the seam `green-baseline.mjs` already had: `--phase <name>` writes
`.green-baseline/<phase>.json` plus its log and Playwright evidence, and `--phase aggregate` reads
whatever phase files are present and writes the receipt. The three lanes now run their phase through
that runner and upload `.green-baseline` as an artifact. `green-baseline` downloads them, runs only
the four phases no lane covers (`deterministic`, `journey-flake`, `ui-matrix-flake`, `slop`), and
aggregates. Each phase now executes exactly once, and the lanes stopped being decorative — their
output is the receipt.

`scripts/workflow-contract.mjs` gained the guard that keeps it from growing back: for each of the
three lane/phase pairs it requires the lane to run `--phase <p>` and upload
`green-baseline-phase-<p>-<sha>`, and it *fails* if `green-baseline` runs that phase itself.
Verified by mutation — stripping the download step, re-adding a phase to the aggregate job, or
reverting a lane to its old command each throws.

**App bugs found: 0. CI/test defects found: 13.**

## 14. green-baseline meta-harness retired (2026-07-27, landed 37b3563)
Not a defect fix — a layer removal. The 955-line receipt/phase/aggregate machine
(scripts/green-baseline.mjs + schema + manifest + 5 satellite test files +
validate-green-baseline.mjs) is deleted. Lanes now run their real suites directly
(test:integration-db, test:factory-journeys-certified, test:factory-ui-matrix,
sweep:full); the aggregate job (which never executed end-to-end) is gone; the
verdict is the workflow run conclusion. Contract consolidated to ONE source:
workflow-contract.mjs (assertMainValidationWorkflow/assertFullSweepWorkflow,
mutation-tested in workflow-contract.test.mjs, incl. an anti-resurrection assert);
the duplicated expectations literal in setup-pnpm-install-contract.test.ts removed
(it had bitten twice; it bit again during this change — third strike). The open
"preflight failed: disposableDatabase" regression from #13 dissolved with the layer.

## 15. Orphaned-test claim corrected + a11y lane wired (2026-07-27)

- The "210 never-run test files" claim (#entry above) was STALE: Fast gate `test` job runs the full 613-file default vitest lane (`pnpm --filter web test`); all other lanes run in Fast gate or main-validation. Real orphans = 4 files.
- 3 node:test tooling suites (e2e-main-factory-journeys, sweep-zero-baseline, slopgate-gate) wired into `quality:changed:test` (root package.json + workflow-contract string). All pass.
- tests/a11y (13 axe WCAG specs incl. admin-languages.test.ts) had a runner script (`test:a11y`) but no workflow. Added `a11y` job to main-validation on the debian2 chain after factory-ui-matrix (disposable factory Neon). Verified green locally first (13/13, 1.1m).
- Contract updates: MAIN_VALIDATION_JOBS +a11y, debian2 chain +a11y, main-validation.yml setup calls 12→13.
- Latent Fast-gate red found+fixed: commit e4f1cd0d6 rewrote db-dump.yml to actions/setup-node (pg tooling only) but setup-pnpm-install contract still tracked it → removed db-dump.yml from SETUP_ACTION_WORKFLOW_EXPECTATIONS, retargeted the fail-closed mutation to deploy.yml.
- Task 3 (prod smoke): PROD_URL was already app.multideal.co.il. MIN_TEST_ROUTES 12→18 (matches actual route count). Audit tightened to 403/404-only; 4 routes (reset-customer/vendor/shipment, stripe-complete-vendor) returned 400 because defineApi body-validated before the E2E proof gate — added verifyE2eProof preflight (404-stealth, matches factory routes' documented model). Strict audit stays red against prod until the next prod deploy ships these routes.

## 16. 2026-07-28 — dispatch 30293182618 (3b4c232d2): 13/14 green, Factory journeys red on two GOLIVE spec bugs
- affiliate-referral-click-commission-pending.spec.ts:206 — `MIN(ae.entry_id)::text` on a uuid column; Postgres has no min(uuid). Fixed to `MIN(ae.entry_id::text)`.
- UJ-012-single-deal-purchase.spec.ts:221 — redirect regex demanded URL-encoded `%2Fbuy%2F`; the browser reports the decoded `/buy/`. Regex now accepts both.
- Classification: new-test defects from GOLIVE T12–T15 journey specs (first real CI execution of those tests), not infra. Manifest drift (55→58) fixed same day in 3b4c232d2.

## 17. 2026-07-28 — dispatch 30297329395 (dd07a6b3b): 13/14 green, Factory journeys red on second affiliate spec defect
- affiliate-referral-click-commission-pending.spec.ts:287/322 — inner-joined `ledger_entry_vesting` + `wallet_balances`; wallet_balances has NO live writer (legacy donor table, only 0073 migration + factory cleanup touch it) and vesting rows are backfill-only, so the readback always returned zero rows → `undefined.swept_at`.
- Fix: scalar subquery for swept_at (absent = un-swept, matching host-retained.ts LEFT JOIN semantics); replaced wallet_balances pending check with the API pending oracle (`/api/affiliate/stats` bounded decrease after clawback).
- Classification: same family as #16 — GOLIVE-authored spec never executed against live schema; oracle targeted dead tables.

## 18. 2026-07-28 — dispatch 30301489527 (ad60a97d3): 13/14 green, Factory journeys red on third+fourth affiliate spec defect layer
- affiliate-referral-click-commission-pending.spec.ts — clawback readback found no row. Local instrumented repro proved TWO real defects, one of them a genuine app-side test-harness bug:
  1. `services.queues.runOutbox()` (bundle.ts) invoked the queue consumer with `messages: []` — the `/api/queues/outbox` drain endpoint was a NO-OP; nothing ever consumed outbox rows on demand, and consumption relied on timing-dependent queue/cron delivery (racing test teardown → the CLAWBACK_ORIGINAL_NOT_FOUND server noise). Fixed by making runOutbox run the cron backstop `runProcessOutbox` (DB scan, FOR UPDATE SKIP LOCKED).
  2. The spec asserted `source_id LIKE '%<Idempotency-Key>'`, but clawback rows key on the PROVIDER refund id (execute response `refundId`), so the oracle could never match; also under-counted (commission + qualification reward = 2 clawback rows) and read the final stats as the admin session (context re-cookied by injectSession). Fixed: poll-with-drain on the provider refund id expecting 2 rows; affiliate session restored before the stats oracle.
- Classification: #16/#17 family (GOLIVE-authored spec never executed) PLUS one real infra defect (drain endpoint no-op) that the spec exposed.
- Verified: local disposable-Neon repro 2/2 passed (repro12); typecheck 0 errors; risk validator green.
- Addendum: 4th dispatch 30311814958 (ec8a758b2) — Factory journeys GREEN (affiliate spec fixed). One new red: Changed-browser-journeys, UJ-015 guest-confirmation page.goto 3×60s hang; same test at same commit passed in the sibling Factory-journeys lane of the same run → lane-local navigation flake. Rerun of the failed lane went green → run fully green, 14/14 lanes.