# Durable Execution

**Status:** Approved target-state baseline  
**First realization:** I0  
**Provider:** DBOS + PostgreSQL.

## Target State

AWP separates **workflow meaning** from **durable execution mechanics**.

```text
AWP application/domain
  defines lifecycle, authorization, IDs, policy, events
        ↓
DurableWorkflowProvider
        ↓
DBOS
  checkpoints/recovery/timers/retries mechanics
        ↓
PostgreSQL
```

DBOS is the default durable workflow provider. AWP must not create a second generic workflow engine beside it.

## Invariants From Day 1

- durable workflow ID correlates to an AWP operation/run but is not the domain identity;
- replay/retry never duplicates externally visible domain transitions;
- external side effects use idempotency/attempt identity and reconciliation;
- cancellation is explicit and persisted;
- workflow restart after control-plane crash resumes from durable state;
- user-facing state is derived from AWP domain state, not raw DBOS internals;
- long-running waits use durable timers/events rather than in-process sleeps;
- provider failures become typed lifecycle outcomes and retain provenance.

## Workflow Ownership

Examples of durable workflows:

```text
FactoryRun orchestration
AgentRun attempt dispatch/retry
trusted Git publication/merge sequence
CI request/reconciliation
Release/Deployment rollout
machine enrollment
incident resolution
scheduled Plan launch
```

Not every asynchronous helper becomes a durable workflow. A short idempotent repository read or UI query remains ordinary application code.

## Attempt Model

Logical work and concrete execution are distinct:

```text
AgentRun
  ├─ Attempt 1 (provider/account/model/workspace immutable)
  └─ Attempt 2 (fallback/retry)
```

The same pattern applies where execution attempts matter for CI, deployment or resolution. Retry reason and previous checkpoint/evidence are explicit.

## Cancellation

Cancellation has at least:

```text
requested
  -> provider cancellation attempted
  -> resources reconciled
  -> terminal cancelled / cancellation-failed-with-incident
```

Cancellation is symmetric with start/schedule where user semantics require it. WIP/evidence preservation happens before destructive cleanup.

## Idempotency and Recovery

Every workflow step interacting with an external provider defines:

- operation key;
- provider request identity/idempotency key where supported;
- retry classification;
- reconciliation query;
- terminal vs retryable errors;
- cleanup ownership;
- evidence/audit emitted on transition.

A control-plane restart at any step must resume or safely reconcile rather than restart blindly.

## Scheduling

Scheduled operations persist target time, timezone/owner intent and preflight state. Activation revalidates policy/connections/current revision rather than assuming earlier readiness is still valid.

## Increment Realization

| Increment | Durable workflow use |
|---|---|
| I0 | provider abstraction, DBOS bootstrap, core recovery/cancel/ownership tests. |
| I1 | FactoryRun/AgentRun/retry/checkpoint/publication/merge. |
| I2 | scheduled Plan launch and connection-interrupted Planning return flows where durable orchestration is needed. |
| I5 | CI orchestration/resolution. |
| I6 | deployment/rollback. |
| I7 | machine enrollment/lifecycle. |
| I8 | incident/resolver workflows. |

## Current Implementation State

DBOS selection is ratified; executable workflows are implemented with their increments.

## Deferred Realization

Alternative durable providers are not baseline scope. Re-open only if a demonstrated HA/operations constraint cannot be solved cleanly behind `DurableWorkflowProvider`.

## Temporary Dogfood Behavior

A single control-plane instance may be used initially, but restart recovery and exactly-once-ish domain transition semantics still must be tested before relying on live Factory work.

## Decisions / ADRs

See ADR 0005. Do not introduce a parallel job queue/workflow authority without a superseding Decision.