# Failure Taxonomy

Frequency is a conservative lower-bound classification over the normalized catalog, not a count of every textual mention.

| Class | Failure class | Severity | Frequency | Representative incidents |
|---|---|---:|---|---|
| F01 | Destructive cleanup without durable WIP/liveness proof | Critical | Recurring | OD-001, OD-002, OD-028, OD-030 |
| F02 | Work exists outside the collection/transport contract | Critical | Repeated | OD-005, OD-001 |
| F03 | Shared Git/ref/workspace state contaminates trusted publication | High | Recurring | OD-004, OD-009, OD-028 |
| F04 | Ambiguous external mutation retried without reconciliation | Critical | Repeated | OD-006, OD-019 |
| F05 | Multiple mutable credential/token authorities | Critical | Repeated | OD-007, OD-008 |
| F06 | Process-scoped locks/ownership outlive or diverge from logical ownership | Critical | Recurring | OD-003, OD-019, OD-002 |
| F07 | Agent/session lifetime is mistaken for durable work lifetime | High | Recurring | OD-015, OD-013, OD-016 |
| F08 | Unbounded sessions/helpers/waiters exhaust local resources | Critical | Recurring | OD-003, OD-016, OD-017 |
| F09 | Shared global queue/deploy state creates unrelated-work outage | High | Recurring | OD-003, OD-009, OD-011 |
| F10 | Runner/cluster/placement admitted before executable environment is proven | High | Recurring | OD-011, OD-012 |
| F11 | Runtime/artifact identity diverges from source/release intent | Critical | Recurring | OD-012, OD-022, OD-024 |
| F12 | Durable history/state is reconstructed from ephemeral registry/process state | Critical | Repeated | OD-010, OD-020 |
| F13 | Provider/realtime connection stays latched to stale topology | High | Observed/representative | OD-021 |
| F14 | UI/status suppresses or overstates authoritative state | High | Recurring | OD-014, OD-025, OD-028 |
| F15 | Async phase has no bounded stage/deadline/progress receipt | High | Recurring | OD-013, OD-003, OD-014 |
| F16 | Recovery action is symptom-driven and can amplify damage | High | Recurring | OD-002, OD-017, OD-030 |
| F17 | Duplicate execution/incident identity depends on unstable string equality | Medium/High | Repeated | OD-018, OD-019 |
| F18 | Consumer and producer disagree on cross-component contract | High | Observed/representative | OD-026 |
| F19 | Process/diff outcome is incorrectly treated as semantic task completion | High | Repeated | OD-027, OD-014 |
| F20 | Human/operator intent and progress live in conversation rather than durable work state | High | Recurring | OD-015, OD-029 |
| F21 | Deployment acceptance, release, activation and observed runtime are conflated | High | Recurring | OD-006, OD-014, OD-024 |
| F22 | Event/audit sequence or recovery identity is not restart-monotonic | High | Repeated | OD-019, OD-023 |

## F01 — Destructive cleanup without durable WIP/liveness proof

- **Trigger:** cleanup/reaper/resource-remediation decides a workspace/session/resource is disposable.
- **Root cause:** observed process age, registration state, name or apparent inactivity is weaker than the logical work identity and can race replacement/activity.
- **Contributing conditions:** cleanup pressure during CPU/disk incidents; shared mutable worktrees; lossy observability.
- **Safeguard that failed:** “inactive” heuristics and manual inspection did not bind exact generation/identity or prove collection.
- **Mitigation observed:** salvage refs, attachment checks, descriptor/generation identity, exact owned-resource cleanup.
- **Robustness:** the strongest historical direction is fail-closed on uncertainty; age alone is never destructive authority.
- **AWP lesson:** Workspace cleanup requires terminal durable lifecycle + successful WIP/output collection or explicit abandonment + fenced resource identity. A cleanup failure is separately observable.

## F02 — Work exists outside the collection/transport contract

- **Trigger:** agent creates valid source state in a nested/excluded path, or cleanup occurs before collection.
- **Root cause:** “workspace contents” and “result contents” are defined differently.
- **Safeguard that failed:** successful agent exit/commit did not guarantee collector could see the candidate.
- **Mitigation observed:** explicit contained-workspace guidance or broader transport; neither should rely solely on prompt discipline.
- **AWP lesson:** collection must enumerate/validate Git repositories, dirty state and candidate identity before terminal success. Excluded recoverable WIP is an explicit failure/attention state.

## F03 — Shared Git/ref/workspace state contaminates trusted publication

- **Trigger:** unrelated task leaves broken ref, dirty clone or partially created tree.
- **Root cause:** trusted publication shares mutable repository/ref namespace with untrusted/concurrent work.
- **Safeguard that failed:** repository-level Git correctness was assumed globally even though task refs were independent failure domains.
- **Mitigation observed:** isolated conductor Git state and exact candidate import.
- **AWP lesson:** publication adapter consumes immutable ChangeSet/base/candidate data and uses a trusted isolated provider workspace; unrelated workspace refs cannot affect it.

## F04 — Ambiguous external mutation retried without reconciliation

- **Trigger:** provider/transport times out or returns unreachable after side effect may already have committed.
- **Root cause:** response receipt is treated as mutation truth.
- **Safeguard that failed:** retry path lacked read-back identity/fidelity proof.
- **Mitigation observed:** reconcile provider state before repeating.
- **AWP lesson:** every retryable mutation has operation/idempotency identity and a reconciliation read. Timeout is “unknown outcome”, not automatically “failed”.

## F05 — Multiple mutable credential/token authorities

- **Trigger:** health poller, CLI, proxy or remote copy independently refreshes/rewrites one logical account grant.
- **Root cause:** credential material was copied as mutable state; no single authority/fencing boundary.
- **Safeguard that failed:** file ownership/flock was too local to bind multiple machines/process types.
- **Mitigation observed:** one refresh-token owner; readers receive read-only/short-lived projections; bounded fingerprint-only telemetry.
- **AWP lesson:** exactly one configured mutable credential authority per logical Account/Connection; migration fences old authority before new activation.

## F06 — Process-scoped locks/ownership diverge from logical ownership

- **Trigger:** creator dies, descriptor survives, PID/session is replaced, or heartbeat starts before ownership is acquired.
- **Root cause:** OS process/lock identity is used as durable run ownership.
- **Safeguard that failed:** dead PID or elapsed age did not imply lock/resource could be safely reclaimed.
- **Mitigation observed:** generation/lease CAS, identity revalidation, ownership receipt, lock-before-side-effects.
- **AWP lesson:** DBOS/application state owns logical lease/generation; process/resource state is reconciled projection. No child/heartbeat/public side effect before successful ownership claim.

## F07 — Agent/session lifetime mistaken for durable work lifetime

- **Trigger:** model context compacts, terminal closes, process crashes, session is killed, or account location changes.
- **Root cause:** progress/next action/intent exists only in conversational runtime.
- **Safeguard that failed:** transcript existence did not provide current work state or reliable automatic continuation.
- **Mitigation observed:** reconstruct from transcript; move work state to DB/plan registry.
- **AWP lesson:** Task/AgentRun/Attempt are durable independently of any chat/provider session. Context loss creates another Attempt/continuation path, not lost assignment identity.

## F08 — Unbounded sessions/helpers/waiters exhaust local resources

- **Trigger:** blocked queues and idle sessions accumulate faster than cleanup/resource ceilings.
- **Root cause:** no end-to-end concurrency/resource budget; blocked waiters consume process/memory resources.
- **Safeguard that failed:** local cleanup treated individual consumers rather than admission/backpressure.
- **Mitigation observed:** resource caps, offload, bounded cleanup and later capacity ceilings.
- **AWP lesson:** Queue/placement policy must bound admitted concurrent execution and blocked waiter footprint; durable queued work need not occupy an execution process.

## F09 — Shared global queue/deploy state creates unrelated-work outage

- **Trigger:** dirty global clone, wedged admission lock, or incompatible dispatch policy.
- **Root cause:** one mutable global holder mediates unrelated projects/tasks without isolation or sharding of failure domains.
- **Safeguard that failed:** FIFO/serialization preserved order but not availability.
- **Mitigation observed:** coalescing/persistent requests and isolation of some control paths.
- **AWP lesson:** queue semantics are durable domain state; one task's invalid source/environment must block that task/lane, not corrupt the queue service or unrelated candidates.

## F10 — Runner/cluster/placement admitted before executable environment is proven

- **Trigger:** host is reachable but runtime link, tool versions, command classification, image contents or manifest differ.
- **Root cause:** “host online” is weaker than “eligible for this Attempt profile”.
- **Safeguard that failed:** parity/capability validation happened after work queued/dispatched.
- **Mitigation observed:** self-test/rebuild/reprovision.
- **AWP lesson:** PlacementDecision requires an attested eligible execution profile. Environment mismatch is infrastructure evidence and may make one host ineligible without turning one bad host into fleet outage.

## F11 — Runtime/artifact identity diverges from source/release intent

- **Trigger:** stale image/bundle/service remains active after source/release moves; package omits required files.
- **Root cause:** source commit, packaged artifact, installed version and observed running version were conflated.
- **Safeguard that failed:** “landed/deployed” was declared without manifest/runtime smoke/read-back.
- **Mitigation observed:** immutable release manifest, pre-activation verification, observed executable/version proof, rollback to known-good.
- **AWP lesson:** Artifact, Release and Deployment identities remain separate. Promotion verifies digest/content contract; Deployment completion verifies observed runtime identity.

## F12 — Durable history/state reconstructed from ephemeral registry/process state

- **Trigger:** process restart, quarantine, registry cleanup or import checkpoint.
- **Root cause:** historical truth was not independently durable/queryable.
- **Safeguard that failed:** active registry/discovery cache became de facto source of historical UI state.
- **Mitigation observed:** persistent history archive always loaded; discovery checkpoint only optimizes scanning.
- **AWP lesson:** current execution resources may disappear; business state/history and accepted evidence remain in authoritative persistence.

## F13 — Provider/realtime connection stays latched to stale topology

- **Trigger:** provider endpoint/pointer changes or old endpoint hangs.
- **Root cause:** discovery result cached as durable identity.
- **Safeguard that failed:** no invalidation/re-discovery on network/timeout/auth failure.
- **Mitigation observed:** reread pointer and invalidate connection under defined failure classes.
- **AWP lesson:** provider endpoint is observation/config; reconnect/reconcile preserves AWP identity and re-fetches authoritative state after gaps.

## F14 — UI/status suppresses or overstates authoritative state

- **Trigger:** truncated output, dropped nested failure fields, hidden downstream refusal, stale process projection.
- **Root cause:** UI uses lossy projection or optimistic lifecycle copy instead of exact domain/provider evidence.
- **Safeguard that failed:** absence in projection was rendered as absence in source; accepted was visually conflated with done.
- **Mitigation observed:** typed projections, last-progress timestamps, explicit stale/unavailable states, detailed failure fields.
- **AWP lesson:** UI may summarize, never synthesize or erase critical lifecycle meaning. Unknown/stale/partial are explicit first-class states.

## F15 — Async phase has no bounded stage/deadline/progress receipt

- **Trigger:** pre-submit wait, ship/gate queue, remote process, or downstream deploy hangs.
- **Root cause:** long-running operation lacks a stage-specific liveness contract.
- **Safeguard that failed:** process presence/spinner substituted for evidence of forward progress.
- **Mitigation observed:** hard phase deadline, loud stage output, timeout code/status, last activity.
- **AWP lesson:** durable workflows expose stage, attempt, deadline/lease, last progress and retryability; timeout routes to reconcile/resolution rather than silent indefinite wait.

## F16 — Recovery action is symptom-driven and can amplify damage

- **Trigger:** resource pressure, stale socket, disk fullness, “dead” session.
- **Root cause:** automation acts on process name/path/age rather than verified ownership and root condition.
- **Safeguard that failed:** repeated cleanup was considered success even when producer/root cause remained.
- **Mitigation observed:** exact resource identity, producer control, postcondition verification, fail closed.
- **AWP lesson:** ResolutionRecipe needs applicability predicates, authority bounds, idempotency/reconciliation, postcondition and rollback/cleanup evidence. Repeated ineffective recipe escalates.

## F17 — Duplicate execution/incident identity depends on unstable string equality

- **Trigger:** same incident/operation expressed with different signature text or watchdog repeats unchanged condition.
- **Root cause:** exact text key is not stable domain/provider operation identity.
- **Safeguard that failed:** dedup had no provider delivery ID, operation key or state-transition edge semantics.
- **Mitigation observed:** manual duplicate linking; one-attempt-per-crossing designs.
- **AWP lesson:** command/event idempotency is deterministic; Incident correlation can be probabilistic/advisory but must not launch duplicate resolution blindly.

## F18 — Consumer and producer disagree on cross-component contract

- **Trigger:** parallel implementation assumes an endpoint/field/capability not present at producer.
- **Root cause:** contract described locally rather than shared/versioned/tested.
- **Safeguard that failed:** component tests mocked the desired seam rather than verified supported producer behavior.
- **Mitigation observed:** shared source-of-truth spec and seam-consistency review.
- **AWP lesson:** provider ports are explicit; adapters have contract/integration tests against real or authoritative fake semantics, including missing capability.

## F19 — Process/diff outcome incorrectly treated as semantic Task completion

- **Trigger:** no diff because intent already satisfied, agent exits without output, downstream refusal after accepted candidate.
- **Root cause:** mechanical execution result replaces domain outcome/evidence.
- **Safeguard that failed:** `exit=0`, nonempty diff, or enqueue success used as completion proxy.
- **Mitigation observed:** inspect source/evidence and distinguish already-done/no-op/infrastructure/product outcomes.
- **AWP lesson:** Task completion is governed by declared outcome/evidence; AgentRun/Attempt and ChangeSet are supporting primitives, not synonyms.

## F20 — Human/operator intent and progress live in conversation rather than durable work state

- **Trigger:** many parallel terminals/conversations, compaction, delay, forgotten request.
- **Root cause:** no searchable canonical row for “what I asked, who owns it, what happened last, what is next”.
- **Safeguard that failed:** plan docs landing later and chat memory were too slow/fragile.
- **Mitigation observed:** live requests board and plan-status registry.
- **AWP lesson:** Work/Queue/Plan/Decision state is durable and searchable; human attention should be requested only for irreconcilable authority/product decisions, not ordinary recoverable infrastructure state.

## F21 — Deployment acceptance, release, activation and observed runtime are conflated

- **Trigger:** enqueue accepted; Git land completes; deploy later refuses; service remains stale.
- **Root cause:** multi-step delivery compressed into one “ship” concept.
- **Safeguard that failed:** upstream success implied downstream completion.
- **Mitigation observed:** separate deploy status, activation proof and refusal notification.
- **AWP lesson:** ChangeSet merge, Release creation, Deployment attempt and observed Environment state have independent identities/statuses.

## F22 — Event/audit sequence or recovery identity is not restart-monotonic

- **Trigger:** journal reopen, process restart, stale generation, mixed legacy/current registries.
- **Root cause:** sequence/ownership identity derived from process-local counters or ambiguous legacy state.
- **Safeguard that failed:** restart path lacked durable tail/generation reconciliation.
- **Mitigation observed:** recover last durable sequence; generation-bound ownership receipts; canonical registry.
- **AWP lesson:** event/outbox sequence and Attempt/resource ownership remain monotonic across restart; stale generations cannot mutate newer state.

## High-severity recurring classes

The most important recurring classes for I0/I1 are F01, F03, F04, F05, F06, F08, F10, F11, F12, F14, F15 and F21. They recur across different Overdeck mechanisms, which is stronger evidence than repeated failures in one implementation.