# AWP Lessons from Overdeck FIRE Incidents

These are translations from empirical failure evidence into AWP guidance. They deliberately avoid importing Overdeck-specific tmux/systemd/file-layout mechanics where AWP already has stronger domain primitives.

## L01 — Durable work must outlive disposable compute

- **Evidence:** OD-001, OD-002, OD-005, OD-015.
- **Lesson:** A Workspace/pod/process/session is a temporary execution carrier. Task, AgentRun, Attempt, checkpoints, accepted ChangeSet/WIP, and continuation state are durable control-plane data.
- **Disposition:** `PRINCIPLE`, `INVARIANT`, `TEST`.
- **Current AWP alignment:** strong. `architecture/principles.md`, `architecture/workspace-execution.md`, `domains/agents.md`, `domains/changes.md`, and `workflows/factory-execution.md` already encode this direction.
- **Gap exposed by evidence:** the principle needs collection/cleanup failure-injection tests that detect valid WIP in paths the result collector would otherwise omit.

## L02 — Cleanup is a separately fenced workflow, never an epilogue assumption

- **Evidence:** OD-001, OD-002, OD-030.
- **Lesson:** Cleanup requires exact resource identity/generation, terminal logical state, and proof that recoverable outputs were durably collected or explicitly abandoned. Uncertainty preserves the resource and raises attention.
- **Disposition:** `INVARIANT`, `SECURITY REQUIREMENT`, `TEST`.
- **Current AWP alignment:** `workspace-execution.md` already makes cleanup separately observable/idempotent; the incident corpus justifies stronger destructive-race tests.

## L03 — Trusted publication accepts an exact candidate, not a repository-shaped environment

- **Evidence:** OD-004, OD-009, OD-028.
- **Lesson:** Publication should not enumerate arbitrary task refs, trust a shared checkout, or infer safety from truncated CLI output. It consumes immutable base/candidate/tree/digest provenance and re-reads provider state.
- **Disposition:** `INVARIANT`, `SECURITY REQUIREMENT`, `TEST`.
- **Current AWP alignment:** already explicit in `domains/changes.md`, `domains/repositories-vcs.md`, `workflows/task-to-merge.md`, and the threat model.

## L04 — Timeout means unknown outcome until reconciled

- **Evidence:** OD-006 and the broader mutation/retry patterns in OD-019/OD-021.
- **Lesson:** A transport timeout is not a domain failure if the provider may have committed the mutation. Read-back/reconciliation precedes repeat.
- **Disposition:** `INVARIANT`, `TEST`.
- **Current AWP alignment:** strongly specified in adapter/persistence/durable-execution contracts; must be present in provider implementation tests from I1.

## L05 — One logical account grant has one mutable credential authority

- **Evidence:** OD-007, OD-008.
- **Lesson:** readers/probes/runners may receive scoped projections but must not independently refresh reusable provider credentials. Authority migration is fenced and observable.
- **Disposition:** `SECURITY REQUIREMENT`, `INVARIANT`, `TEST`.
- **Current AWP alignment:** explicitly specified in `domains/accounts-models-providers.md`; this evidence elevates dual-writer failure injection to a mandatory Lane B gate.

## L06 — Logical ownership uses durable lease/generation state, not process existence

- **Evidence:** OD-003, OD-019.
- **Lesson:** a PID, file lock, heartbeat or socket does not prove current logical ownership. Side effects begin only after durable ownership claim; stale generations cannot mutate current state; waiters do not consume unbounded processes.
- **Disposition:** `INVARIANT`, `TEST`, `OBSERVABILITY REQUIREMENT`.
- **Current AWP alignment:** durable-execution/persistence concepts fit; exact lease/generation test semantics should be implemented in Lane A/C.

## L07 — Queued work should consume durable state, not an idle process

- **Evidence:** OD-003, OD-016.
- **Lesson:** a backlog of blocked logical work must not create a backlog of live wait processes/helpers. Execution resources are acquired only when admission/placement is viable.
- **Disposition:** `PRINCIPLE`, `TEST`, `OBSERVABILITY REQUIREMENT`.
- **Current AWP alignment:** consistent with Queue + durable workflow + Kubernetes placement design.

## L08 — Eligibility is profile-specific, preflighted, and localizes failure

- **Evidence:** OD-011, OD-012.
- **Lesson:** “node reachable” is not enough. The exact Attempt/CI profile must be runnable with required image/runtime/tool/capability/trust identity before admission. One bad host becomes ineligible rather than poisoning unrelated fleet work.
- **Disposition:** `INVARIANT`, `TEST`, `OBSERVABILITY REQUIREMENT`.
- **Affected AWP:** Lane C primarily; Lane A owns typed state/error semantics.

## L09 — Artifact/release/deployment/runtime identities are distinct

- **Evidence:** OD-022, OD-024, OD-021.
- **Lesson:** source landed, artifact built, release selected, deployment accepted, rollout completed, and runtime observed are different facts. The user needs exact identity at each boundary.
- **Disposition:** `PRINCIPLE`, `INVARIANT`, `TEST`, `UX REQUIREMENT`.
- **Current AWP alignment:** already a central target-state distinction in release/deployment specs; the corpus validates it empirically.

## L10 — Historical truth cannot depend on executor registry or current process

- **Evidence:** OD-010, OD-020, OD-023.
- **Lesson:** restart/quarantine/cleanup must not remove completed/failed history. Checkpoints can optimize discovery, never replace loading authoritative history. Event ordering must survive reopen.
- **Disposition:** `INVARIANT`, `TEST`, `OBSERVABILITY REQUIREMENT`.
- **Affected AWP:** Lane A persistence/event outbox, Lane C durable workflows, Lane D projections.

## L11 — Realtime is a projection; gaps and endpoint churn require re-query

- **Evidence:** OD-021, OD-025.
- **Lesson:** cached provider endpoint/SSE/WebSocket state can become stale. Revision/sequence gaps, reconnect, auth failure, and endpoint rediscovery lead back to authoritative queries; they never mutate lifecycle truth themselves.
- **Disposition:** `INVARIANT`, `TEST`, `UX REQUIREMENT`.
- **Current AWP alignment:** `event-audit-realtime.md` already calls for revision/sequence markers and re-query on gaps.

## L12 — UI state must be weaker than or equal to authoritative evidence, never stronger

- **Evidence:** OD-014, OD-025, OD-028.
- **Lesson:** accepted is not shipped; missing projection field is not “no evidence”; a spinner is not progress; truncated logs are not a safety gate. Unknown, stale, partial, queued, blocked, infrastructure-error and terminal success remain distinguishable.
- **Disposition:** `UX REQUIREMENT`, `OBSERVABILITY REQUIREMENT`, `TEST`.
- **Affected AWP:** Lane D + Lane A read models.

## L13 — Every external wait needs a stage, deadline and last-progress fact

- **Evidence:** OD-013, OD-003, OD-014.
- **Lesson:** long-running is acceptable; unbounded ambiguous waiting is not. Stage deadlines can differ from total workflow lifetime, but expiration produces typed retry/reconcile/attention state.
- **Disposition:** `INVARIANT`, `OBSERVABILITY REQUIREMENT`, `UX REQUIREMENT`, `TEST`.

## L14 — Recovery is governed by applicability and postcondition, not a successful shell exit

- **Evidence:** OD-017, OD-030.
- **Lesson:** a recipe that deletes symptoms while the producer keeps recreating them is not successful recovery. Exact target/authority, idempotency/reconciliation, postcondition and recurrence threshold are required.
- **Disposition:** `PRINCIPLE`, `INVARIANT`, `TEST`.
- **Current AWP alignment:** `domains/incidents-resolvers.md` and `workflows/incident-to-resolution.md` already reflect this.

## L15 — Stable operation IDs solve idempotency; incident similarity is a separate concern

- **Evidence:** OD-018, OD-019.
- **Lesson:** provider delivery ID/operation key/generation supports exact dedup. Similar incidents may be correlated but should not be silently merged solely by text, nor independently remediated solely because text differs.
- **Disposition:** `INVARIANT`, `TEST`, `IMPLEMENTATION IDEA` (semantic incident correlation implementation remains open).

## L16 — Cross-component contracts must be executable seams

- **Evidence:** OD-026.
- **Lesson:** a consumer mock that describes the endpoint the consumer wishes existed is not integration evidence. Provider ports should have authoritative fake/contract tests and at least one real adapter acceptance seam before dependent lifecycle claims are made.
- **Disposition:** `PRINCIPLE`, `TEST`.

## L17 — Task completion is semantic, not equivalent to process exit or nonempty diff

- **Evidence:** OD-027, OD-014.
- **Lesson:** already-satisfied work, no-op, provider failure, infrastructure error, candidate produced, candidate accepted and downstream delivery are different outcomes. Task acceptance is defined by required output/evidence.
- **Disposition:** `INVARIANT`, `TEST`, `UX REQUIREMENT`.
- **Current AWP alignment:** `workflows/factory-execution.md` already states agent exit alone does not complete Task.

## L18 — Current work identity must be globally discoverable outside a conversation

- **Evidence:** OD-015, OD-029.
- **Lesson:** “what did I ask, who owns it, what happened last, and what is next” is operational state. It cannot depend on remembering a terminal/chat or waiting for a plan document to land.
- **Disposition:** `PRINCIPLE`, `UX REQUIREMENT`, `OBSERVABILITY REQUIREMENT`.
- **Current AWP alignment:** Project/Plan/Work/Queue model is a direct improvement over the historical failure.

## L19 — Human attention is for authority/intent conflicts, not routine recoverable mechanics

- **Evidence:** OD-014, OD-029 and historical requests to automate incident remediation rather than relay every failure through the owner.
- **Lesson:** recoverable provider/queue/runner failures should reconcile, retry or route to a bounded resolver according to policy. Human attention is durable and explicit when a real decision/permission is required.
- **Disposition:** `PRINCIPLE`, `UX REQUIREMENT`.
- **Current AWP alignment:** consistent with Approval/Decision/Policy and Incident Resolver design.

## L20 — Safety controls must observe machine-readable full state, not presentation output

- **Evidence:** OD-028.
- **Lesson:** log truncation, pretty status and summaries may help people but cannot authorize publication, cleanup or destructive mutation. Safety gates inspect exact tree/digest/revision/resource identity.
- **Disposition:** `SECURITY REQUIREMENT`, `INVARIANT`, `TEST`.

## Legacy-specific details that should not become AWP architecture

The following are useful historical implementation details but are **`LEGACY-SPECIFIC`** rather than AWP requirements:

- tmux socket and attachment mechanics;
- `agent.slice`/systemd scope names and `TasksMax` values;
- particular Overdeck queue-file paths and lock inode rotation procedure;
- the `claudex`/systray directory layout;
- specific buildbox names/version pins;
- Overdeck's local deploy clone and historical harness registry formats;
- exact FIRE skill/playbook names.

AWP should preserve the proven invariants behind those mechanisms, not their shapes.