# AWP Regression / Failure-Injection Test Matrix

This matrix is the primary handoff from Lane E to implementation lanes. Each scenario is written so another lane can turn it into a concrete automated gate without rereading the historical incidents.

| ID | Failure class | Concrete test / injected failure | Required assertion | Level | Owner | Environment |
|---|---|---|---|---|---|---|
| RT-001 | F02 uncollected WIP | During an AgentRun, create a nested Git worktree/repository under a path excluded by ordinary result transport, commit/dirty it, then let agent process exit. | Attempt/Workspace cannot become successfully collected/cleaned while recoverable Git state is excluded; explicit uncollected-WIP finding preserves resource/checkpoint. | failure injection + integration | C | Kubernetes Workspace test namespace with real Git repo |
| RT-002 | F01 cleanup race | Start cleanup for Workspace generation N, then before destructive provider call replace/rebind resource with generation N+1. | N cleanup fails stale/fenced and cannot delete N+1 resource. | concurrency + failure injection | C/A | fake provider + K8s integration |
| RT-003 | F01 collection-before-cleanup | Make output/checkpoint upload fail after execution terminal but before durable collection. | logical run records collection failure/attention; workspace remains recoverable; cleanup not marked complete. | integration | C/A | K8s + object/PVC/checkpoint fake |
| RT-004 | F01/F07 disconnect | Disconnect browser/client and terminate provider session while durable AgentRun is active. | Task/AgentRun remains durable; disconnect does not imply cancellation; retry/reattach creates correct Attempt relation without WIP loss. | E2E + integration | A/C/D | control plane + browser + fake/real AgentProvider |
| RT-005 | F03 Git isolation | Add an unrelated local ref pointing to missing object in an execution/workspace repo while trusted publication of a valid ChangeSet is requested. | publication still consumes exact candidate/isolation context; unrelated bad ref cannot break or alter published candidate. | integration + security | B | temporary Git repos + GitHub fake |
| RT-006 | F04 publication ambiguity | Provider applies branch/PR publication, then injected transport drops response before AWP receives it; workflow retries. | reconciliation finds existing exact publication; exactly one provider-side publication exists; AWP stores one logical operation with retry evidence. | failure injection | B/A | GitHub adapter fake with post-commit response loss |
| RT-007 | F04 merge ambiguity | Provider merges exact expected candidate, then response times out; durable workflow resumes after process restart. | provider read-back resolves merged identity; no second merge/update; exact resulting commit is persisted. | failure injection + crash recovery | B/C | GitHub fake or disposable repo/provider test |
| RT-008 | F05 credential dual-writer | Concurrently ask two components to become refresh authority for same Account/Connection and issue refresh. | at most one authority acquires mutable ownership; second is fenced/typed conflict; secret never appears in event/audit output. | security + concurrency | B/A | SecretStore/provider fake |
| RT-009 | F05 authority handoff | Begin authority migration A→B; pause after B prepared but before A fenced, then inject restart/retry. | B cannot refresh until A is durably deactivated/fenced; recovery converges to one authority, never dual-active. | crash recovery + security | B/C | provider fake + durable DBOS workflow |
| RT-010 | F06 ghost owner | Acquire durable execution lease, start waiter, kill owner process before release while leaving simulated stale external lock/resource. | reconciliation advances/fences lease generation and restores progress without trusting stale process lock; no indefinite waiter process growth. | chaos/recovery | A/C | DBOS + provider fake/K8s |
| RT-011 | F06 stale generation | Attempt N loses lease; Attempt N+1 becomes owner; deliver delayed callback from N that tries checkpoint/status/provider mutation. | stale generation callback is rejected/no-op and cannot overwrite N+1 state. | unit + concurrency | A/C | application/persistence fake |
| RT-012 | F08 waiter bound | Queue 10k blocked Tasks while execution concurrency is 2 and provider unavailable. | durable DB rows may grow but live processes/pods/workflows stay within configured bound; no one waiter process per queued item. | load + integration | A/C | DBOS/K8s test environment |
| RT-013 | F10 runtime drift | Mark host reachable, then mutate/remove required runtime/tool/image attestation before placement. | host becomes ineligible for that profile before dispatch; failure is infrastructure/placement evidence, not product failure. | integration | C | K8s/runner profile test cluster |
| RT-014 | F10 failure localization | Three eligible hosts; corrupt runtime profile on one. Dispatch several jobs. | bad host is excluded/degraded; healthy hosts remain selectable; one stale/offline host does not make fleet globally unavailable. | integration | C | 3-node/fake-node scheduler test |
| RT-015 | F10 command-route incompatibility | Configure a command/profile combination whose runner policy cannot execute (analogous to remote-only + opaque local wrapper). | admission/preflight returns typed no-eligible-execution-path before queue/run allocation; no growing doomed queue. | unit + integration | C/A | Workspace/CI provider fake |
| RT-016 | F05/F10 runner secret boundary | Launch AgentRun and CI runner pods and inspect env/mounts/runtime filesystem. | reusable Git publication/control-plane/provider refresh credentials are absent; only scoped attempt credentials are available. | security + integration | C/B | K8s/gVisor/ARC test cluster |
| RT-017 | F11 artifact identity | Build Artifact from source S, then move branch head to S2 before Release/Deployment. | Artifact/Release continue to reference exact S/digest; mutable branch move cannot change provenance. | integration | A/C (I6) | build/registry fake |
| RT-018 | F11/F21 stale runtime | Deployment provider reports apply accepted but observed service continues running previous artifact/version. | Deployment remains progressing/degraded/failed according to policy, never successful solely from apply response; observed identity is recorded. | failure injection | C/A (I6) | disposable K8s namespace |
| RT-019 | F11 broken package | Produce artifact missing declared required runtime entrypoint/file while source build itself exits zero. | promotion/activation gate rejects artifact before replacing known-good runtime; evidence names missing contract. | integration + supply-chain | C/A (I6) | artifact fixture/registry |
| RT-020 | F12 restart history | Persist completed/failed FactoryRun/AgentRun/Attempt/ChangeSet evidence; restart control plane and workflow provider. | same historical identities/evidence remain queryable immediately after recovery, independent of live pod/registry. | crash recovery | A/C | Postgres + DBOS restart test |
| RT-021 | F12 quarantine history | Quarantine/delete executor resource/provider registry projection for a completed/failed run. | UI/read model still shows historical run and evidence; resource quarantine changes resource state only. | integration + E2E | A/C/D | test control plane + browser |
| RT-022 | F22 event sequence | Write events, crash/reopen producer, leave a deliberately torn final provider/journal record, then emit more events. | persisted AWP event/outbox identity/sequence remains monotonic/no duplicate; corrupt external observation is handled explicitly. | persistence + crash recovery | A | Postgres/event fake |
| RT-023 | F13 realtime gap | Client receives sequence 100 then 103 after disconnect/reconnect. | client marks projection stale, re-queries authoritative read model, then resumes from reconciled state; it does not apply gap as complete truth. | integration + E2E | A/D | realtime server + browser |
| RT-024 | F13 stale provider endpoint | Adapter initially uses provider endpoint A; A becomes unreachable and discovery now returns B. | defined network/timeout/auth failure invalidates A; adapter rediscovers/reconciles B without changing AWP domain identity or duplicating operation. | integration | B/C | provider discovery fake |
| RT-025 | F14/F21 accepted-not-done | Submit publication/deployment operation; make downstream deployment refuse after enqueue. | UI/API shows accepted/queued then blocked/failed; never displays shipped/deployed; blocker/next action remains attached. | E2E | A/D (+B/C provider) | browser + deterministic provider fake |
| RT-026 | F14 rich failure projection | Persist CI/Agent failure with class, cause, attempts, exact source, infrastructure/product distinction; exercise API and UI. | every critical field survives persistence→application→transport→UI; absent/malformed values render explicit unknown/partial, not generic invented ERROR. | contract + E2E | A/D | browser + API fixtures |
| RT-027 | F15 stage deadline | Hold provider in pre-dispatch/pre-submit stage beyond configured deadline while total workflow remains durable. | stage transitions to typed timeout/retryable/attention outcome with last progress/deadline; no indefinite `running` spinner. | failure injection | A/C/D | DBOS/provider fake + UI |
| RT-028 | F15 stalled progress | Keep operation heartbeat alive but stop meaningful stage progress past stall threshold. | read model distinguishes alive-but-stalled from active progress; operator sees last progress age and next automatic action. | integration + E2E | A/C/D | workflow fake + browser |
| RT-029 | F16 ineffective recovery | Resolver recipe exits zero but postcondition remains false; repeat same signature twice. | resolution attempt is not marked successful; repeated ineffective recipe is bounded/escalated and evidence retained. | unit + integration | A (I8) | Incident/Resolver fake |
| RT-030 | F16 cleanup targeting | Resource with same name is deleted/recreated between recipe classification and action. | recipe revalidates immutable UID/generation and refuses destructive action on replacement. | security + failure injection | C/A | K8s fake/namespace |
| RT-031 | F17 incident duplicate | Deliver same provider failure ID twice with different human-readable error strings. | event/operation dedup uses stable provider/account/delivery identity; one lifecycle transition/resolution launch occurs while both deliveries may remain auditable. | unit + integration | A/B/C | event/provider fake |
| RT-032 | F17 idempotency conflict | Replay same AWP idempotency key with identical fingerprint, then with different target/fingerprint. | identical replay returns/reconciles original operation; conflicting replay fails closed and does not mutate. | unit + integration | A | transport/application tests |
| RT-033 | F18 missing producer capability | Consumer adapter fake advertises endpoint/action; real/provider-authoritative capability set omits it. | integration/contract gate fails explicitly before dependent workflow is declared supported; no hidden fallback to invented behavior. | contract | B/C | authoritative provider fake + optional live smoke |
| RT-034 | F19 semantic no-op | Task intent is already satisfied at immutable base; agent returns no diff but supplies evidence; compare with agent returning no diff while intent remains unsatisfied. | first can become verified already-satisfied/no-op; second remains failed/needs correction. Both preserve Attempt evidence. | domain + integration | A/B/C | domain/application + repo fixture |
| RT-035 | F19/F10 infra classification | CI job cannot start because runner infrastructure unavailable; separately run a real test assertion failure. | first is infrastructure-error and retry policy does not create failed-product evidence; second is semantic failed verification. | integration | A/C | CI provider/runner fake |
| RT-036 | F05/F07 retry provenance | Force provider/account quota failure then route retry/fallback to another eligible account/model. | existing Attempt provenance never changes; new Attempt records exact new provider/account/model/reason. | integration | A/B | AgentProvider/account router fake |
| RT-037 | F07/F20 resume | Start durable work from browser/CLI, disconnect/lose client session, reconnect from another client and request continuation. | one logical work item/run remains; continuation attaches to it or creates a new Attempt by policy; no duplicate Task from client retry. | E2E | A/D | two clients + control plane |
| RT-038 | F20 work discoverability | Create Work items across queued/running/blocked/stalled states without any chat transcript. | read model/search can answer owner/agent, current state, last activity, blocker, next action and Project/Plan relation for every non-terminal item. | integration + E2E | A/D | Postgres + browser |
| RT-039 | F20 human attention | Inject transient provider outage, revoked permission, and conflicting owner/business Decision separately. | transient outage follows reconcile/retry; revoked permission produces actionable authorization attention; irreconcilable Decision requires human action. No routine transient failure is mislabeled approval-required. | domain + E2E | A/D | policy/application fixtures |
| RT-040 | F03/F14 safety vs truncation | Candidate deletes thousands of files; configure UI/log summarizer to truncate output so deletion count is not visible to human. | trusted publication safety gate still blocks/flags according to structured candidate/tree policy; display truncation cannot alter authorization outcome. | security + integration | B/A | Git candidate fixture + publication fake |

## Lane coverage

- **Lane A — Core domain/persistence/application:** RT-002/003/004/006/007/008/010/011/012/015/020–023/025–029/031/032/034–040 where noted.
- **Lane B — Accounts/GitHub/Fabro/ACP providers:** RT-005–009, RT-016, RT-024, RT-031/033/034/036/040.
- **Lane C — DBOS/Kubernetes Workspace/gVisor/ARC:** RT-001–004, RT-007/009–019, RT-020/021, RT-027/028/030/033–035.
- **Lane D — I1 UI:** RT-004, RT-021, RT-023, RT-025–028, RT-037–039.

## Priority gate set before I1 convergence

At minimum, the following should be made blocking before I1 is called safe enough to dogfood: **RT-001, RT-002, RT-006, RT-008, RT-010, RT-013, RT-016, RT-020, RT-023, RT-025, RT-026, RT-034 and RT-040**.

These directly exercise silent-work-loss, stale-destructive-cleanup, duplicate-publication, credential-authority, ownership fencing, execution eligibility, secret isolation, restart durability, stale UI, lifecycle honesty, semantic completion and destructive-candidate safety.

## Matrix count

- **Concrete regression/failure-injection scenarios:** 40
- **I1 priority blocking subset:** 13