audience: AI coding agents first

# CI test runtime optimization plan

## Context

Overdeck PR gate now averages 6m21s on cluster runner `overdeck-debian1`: root `Test` 4m20s (68%), E2E 1m27s (23%), all setup/typecheck 34s. Queue time 2–3s. Current gate has no machine-readable suite timings or completeness receipt. Root test graph likely executes 15 local buildbox suites twice: once through `pnpm -r test`, again through `tests/os/run-all.sh` discovery.

Goal: reduce gate wall time and total cluster compute without removing coverage, weakening failure semantics, widening resource budgets, or creating another offload path.

## Execution reset — 2026-08-10

This plan owns CI runtime only. NEVER repair unrelated application, fixture, browser, cluster transport, or moving-`main` failures here. Rebase/merge current `origin/main`; if natural PR CI then fails outside changed CI machinery, record it as external and stop that candidate instead of expanding scope.

Measured evidence now available:

- Canonical measured gate and complete receipt machinery are implemented on PR #11.
- Natural cluster receipt `31357908983` proves all 15 buildbox suites execute through both callers.
- Suite IDs, source digests, argv, cwd, statuses, and output digests match.
- Proof differs only in caller-added `PATH` prefixes: pnpm lifecycle adds `modules/buildbox/node_modules/.bin` and repeated root bins; gate instrumentation adds `tools/ci/shims`. Therefore duplicate removal remains blocked until both callers run under one canonical test environment.
- Browser failures are unrelated to this optimization. Do not modify `apps/web/**` for this plan.

Execute next, in order:

1. Define one canonical buildbox-suite environment in the existing shared observation wrapper. Preserve required system/tool paths; remove caller-specific duplicate lifecycle prefixes. Resolve and record executable identities for commands whose lookup can affect tests. Never hide a real toolchain difference by deleting `PATH` from comparison.
2. Make both existing callers use that exact environment. Add focused RED/GREEN tests proving caller-specific `PATH` inputs converge while a changed resolved executable/toolchain identity still fails equivalence.
3. Push one instrumentation candidate. Accept equivalence only from its natural cluster receipt with `proofComplete=true`, `equivalent=true`, matching output digests, and all 15 suites passed twice.
4. Only then exclude the 15 buildbox-owned suites from OS discovery. Keep `modules/buildbox/test/run-all.sh` authoritative and add exact-set coverage tests.
5. Push one optimized candidate. Accept only a natural cluster receipt proving every expected suite exactly once, no missing raw report, unchanged test counts/statuses, and lower matching-cohort wall time.
6. Land through existing guarded PR path. Do not add sharding, workers, caches, transport, retries, timeout changes, or UI fixes.

Abort deduplication if canonical-environment proof remains false. Report the measured blocker; do not substitute another speculative optimization.

## Hard constraints

1. Run every test on cluster infrastructure. Keep current `[self-hosted, linux, x64, overdeck-gate]` path now.
2. Reuse existing `cpu-guard.sh` → `local-gate` → `tryRemoteBuild()` transport seam. NEVER add runner, scheduler, queue, controller, kubectl transport, or workflow matrix.
3. Treat k3s as future backend for same canonical CI entrypoint. `k3s_enabled` remains off until existing activation/comparison cutover lands.
4. Do not edit `remote-build.mjs`, `k3s-remote-build.mjs`, `modules/harness/factory/**`, runtime manifests, namespace config, or live HOME. Active cluster tasks own those files.
5. Keep Playwright `workers: 1`, retries, timeouts, CPU/memory limits, and current test set unchanged until receipts prove another lever.
6. No `actions/cache` on self-hosted runner. Keep node-local pnpm and Playwright stores.
7. Never rerun broad checks for unchanged tree. Use one natural PR candidate per evidence stage.
8. Root `package.json` is shared with active Factory/k3s work. Coordinate ownership, wait for that change to land, then rebase before editing. Never copy or overwrite the competing diff.

Canonical cluster design: `docs/specs/2026-08-08-k3s-scheduler-design.md` migration path and implementation status. Current k3s blockers: command activation policy/cutover incomplete; safe result pullback and immutable per-run execution must remain cluster-owner work.

## Stage 1 — canonical measured gate

Modify:

- `package.json`
- `.github/workflows/pr-gate.yml`
- `tools/ci/run-gate.mjs` and focused tests
- framework configs/scripts only where needed for raw machine-readable reporters
- `tools/check-offload-workflow.mjs` and focused tests

Implement one `pnpm run ci:gate` command. Preserve current fail-fast order:

1. workflow contract;
2. typecheck;
3. recursive workspace tests;
4. ten named workstation suites;
5. Bun hook tests;
6. OS/module shell suites;
7. Playwright.

Runner MUST create `.ci-results/<run-id>-<attempt>-<tree>/` before first gate phase. Record:

- contract/schema version, repository, full tree SHA, run ID/attempt, executor metadata;
- per-phase and per-suite start/end/duration/status;
- exact `expected`, `executed`, `notStarted` suite IDs;
- terminal status: `passed|failed|timedOut|interrupted|incomplete`;
- source/lockfile/toolchain fingerprints;
- raw reporter paths and SHA-256 digests;
- observed per-test duration/status where framework reporter provides it.

Do not synthesize unobserved failures. Dynamic frameworks may prove completeness at suite/file level; state that granularity explicitly.

Workflow MUST:

- set `BUILD_SCHED_IDLE: "0"`;
- execute `pnpm run ci:gate` exactly once;
- upload receipt, timings, manifest, and raw reports with pinned `actions/upload-artifact`, `if: always()`, unique run/attempt/tree name, `if-no-files-found: error`;
- keep traces/screenshots failure-only;
- retain fork guard, self-hosted label, cancellation, timeout, and persistent stores.

Contract tests MUST reject missing fork guard, matrix, `continue-on-error`, `actions/cache`, absent scheduler env, duplicate gate invocation, missing always-upload, missing canonical command, or green incomplete receipt.

## Stage 2 — complete inventory and equivalence proof

Generate gate inventory from filesystem with `--check` drift mode:

- workspace package test scripts;
- `tests/os/*.test.sh`;
- `modules/*/test/*.test.sh`;
- Playwright specs;
- Bun hook suite;
- workstation test inventory.

Classify current ten workstation suites as required. Classify every existing non-gated workstation suite with machine-readable reason. New/unclassified file MUST fail drift check. Do not silently expand or shrink current test scope in this task.

First instrumentation candidate MUST execute both existing buildbox paths. Normalize both callers through same shell-suite timer. For each overlapping suite capture canonical ID, source digest, argv, cwd, relevant sanitized environment including `HARNESS_ENGINE_DEV`, exit status, and duration.

Comparator MUST prove identical suite set, source digest, invocation contract, and result. Persist proof in receipt artifact.

## Stage 3 — remove proven duplicate only

Proceed only after natural PR receipt proves equivalence.

- Keep `modules/buildbox/test/run-all.sh` authoritative.
- Change `tests/os/run-all.sh` discovery to exclude only buildbox-owned local suites.
- Preserve every other dynamically discovered suite.
- Add contract: excluded set exactly equals buildbox runner’s local owned set.
- Receipt MUST show every buildbox local suite exactly once and full expected union complete.

RED/GREEN:

1. Comparator fixture with changed cwd/env/digest MUST fail; equivalent fixture passes.
2. Pre-dedup inventory MUST detect duplicate ownership.
3. Post-dedup inventory MUST show exact single ownership.
4. Missing/duplicate/not-started-as-passed/raw-report-mismatch receipts MUST fail.
5. Synthetic unclassified test file MUST fail manifest drift.
6. Workflow invariant mutations MUST fail contract tests.

## Stage 4 — ship and measure

1. Land instrumentation candidate through existing guarded PR path. Do not rerun unchanged tree.
2. Consume receipt from that natural PR run; record suite-level baseline and buildbox equivalence.
3. Land dedup candidate only if proof passes.
4. Consume optimized candidate receipt.
5. Compare matching executor, lockfile, toolchain, and transport cohorts. Report wall time and total cluster compute; never claim speedup from missing work.
6. Verify exact expected/executed union and raw-report cross-check before accepting timing.

Historical baseline remains GitHub step timestamps from successful runs `31235532167` and `31234975304`; no rerun required.

## Deferred by evidence

- Playwright parallelism: fixed collector port/shared state makes `workers > 1` unsafe. Revisit only after separate fixture isolation work proves old RED/new GREEN collision test and receipts show E2E remains critical path.
- Workflow sharding: one eligible runner; matrix adds setup/compute without current wall-time benefit.
- Receipt reuse across candidates: broader delivery-controller work, not needed to remove measured duplicate execution.
- k3s activation: after existing cluster tasks land immutable run workspace, result pullback, exact command allowlist, pre-accept SSH fallback, post-accept no-double-run, and comparison ledger. Add `ci:gate` identity there; do not alter transport in this task.

## End-to-end acceptance

- All tests execute on existing cluster runner; workstation runs zero tests.
- Instrumented gate preserves current expected suite inventory and terminal results.
- Buildbox duplication removed only after stored equivalence proof.
- Optimized receipt proves every expected suite executed exactly once.
- Workflow contract observed RED on injected violation and GREEN after restore.
- Optimized natural PR run is faster than matching instrumented baseline, with exact timing artifact and no coverage/completeness reduction.