# Durable Workflow / Transaction Seam

**Status:** Approved target-state baseline  
**First realization:** I0 (rule + proof); binding for I1 FactoryRun workflows  
**Depends on:** [ADR 0004 — PostgreSQL + Drizzle](../adr/0004-postgresql-drizzle.md), [ADR 0005 — DBOS Durable Workflows](../adr/0005-dbos-durable-workflows.md), [Persistence and Transactions](persistence-and-transactions.md), [Durable Execution](durable-execution.md)

## Target State

DBOS checkpoints its own tables in the same PostgreSQL database where Drizzle owns domain state, but **the two writes are not one transaction**. A DBOS step's checkpoint is written by DBOS on its own connection/transaction; the domain mutation is written by `PostgresUnitOfWork.transaction` on the application pool. The window between "domain transaction committed" and "DBOS checkpoint durable" is real, and a crash inside it causes DBOS to replay a step whose effects already exist.

AWP does not close that window with two-phase commit. It makes replay **harmless** by making every effect-producing step idempotent against a marker that commits atomically with the effect it describes.

```text
DBOS workflow (owns: ordering, retry, recovery, timers)
   |
   |-- step "publishChangeSet"
   |     |
   |     +-- ONE Drizzle transaction:
   |           read workflow_step_outcomes (operation_id, step_name, step_key)
   |             -> found  => return recorded outcome, mutate nothing  (replay)
   |             -> absent => domain state
   |                        + business_events
   |                        + audit_records
   |                        + outbox_messages
   |                        + workflow_step_outcomes marker
   |         commit
   |     |
   |     +-- [crash window] DBOS records step checkpoint
   |
   +-- next step
```

`workflow_step_outcomes` is a new table owned by Drizzle, not by DBOS:

```text
workflow_step_outcomes(
  operation_id   text     -- MutationContext.operationId / OperationId
  step_name      text     -- stable authored step name, not a line number
  step_key       text     -- discriminator when one step runs per item
  outcome        jsonb    -- the value the step must return on replay
  recorded_at    timestamptz
  primary key (operation_id, step_name, step_key)
)
```

The marker is the seam. DBOS's checkpoint answers "did this step finish?"; the marker answers "did this step's *domain effect* happen?" — and only the second question has an authoritative answer, because only the second is written in the same transaction as the effect.

## Binding Rule

1. **Every workflow step that mutates domain state does so inside exactly one Drizzle transaction**, and that transaction also writes the step's `workflow_step_outcomes` marker together with state, business event, audit record and outbox rows.
2. **A step begins by reading its marker.** A hit means the step returns the recorded `outcome` and performs no mutation — a retry is a no-op read. Marker insertion uses the primary key as the race guard; a unique violation is treated as "another attempt won", and the step re-reads and returns that outcome.
3. **No transaction spans two steps.** A step may not open a transaction, return, and have a later step commit it. Each step is atomic in isolation; a workflow is a sequence of independently durable commits, never a distributed one.
4. **No domain mutation occurs outside a step boundary.** Workflow-body code between steps is pure orchestration: branching on returned outcomes, computing inputs, awaiting timers/events. Loading state for a decision is permitted; writing is not.
5. **External side effects live in their own step, before the step that records them.** The external call is at-least-once and carries a provider idempotency key derived from `operationId` + step name + attempt identity (per [Durable Execution > Idempotency and Recovery](durable-execution.md)). The subsequent recording step is exactly-once by rule 1.
6. **Outbox dispatch is at-least-once with consumer-side idempotency.** The dispatcher reads uncommitted-to-broker rows after commit, publishes, then marks `published`. A crash between publish and mark causes redelivery; every consumer deduplicates on the outbox message `id`. Never publish inside the domain transaction, and never treat `published=true` as proof of single delivery.
7. **`operationId` is the durable identity.** The DBOS workflow ID correlates to it but is not it, and marker rows are never keyed by a DBOS-internal identifier — the marker must remain meaningful if the workflow provider is replaced.
8. **Markers are retained as long as the workflow can be recovered or replayed**, and are pruned only by an explicit retention job, never by the step that wrote them.

### Forbidden

```text
transaction opened in step N, committed in step N+1
domain write in workflow body outside any step
domain write and DBOS checkpoint assumed atomic
step whose idempotency depends only on DBOS not replaying it
outbox publish inside the domain transaction
consumer that assumes exactly-once outbox delivery
marker written in a different transaction from the effect
non-deterministic step boundaries (marker keyed by timestamp, random ID, array index)
```

## Invariants From Day 1

- domain state, business event, audit record, outbox row and step marker of one step commit atomically or not at all;
- replay of any step produces no second externally visible domain transition;
- a step is safe to execute at least twice and observable exactly once;
- step names and `step_key` values are deterministic across replays of the same `operationId`;
- the outbox dispatcher is at-least-once and its consumers are idempotent by message `id`;
- no external network call is held open inside a domain transaction;
- crash recovery resumes from durable state; it never restarts a workflow from the beginning with fresh identity;
- DBOS internal tables are never read or written by AWP domain, application, persistence or transport code. Test code may inspect them for crash-point diagnostics only.

## Required Proof

Before any FactoryRun workflow ships, **one** integration test must exist and pass. It is a gate, not a nice-to-have.

**Name:** `durable seam: crash between step commit and DBOS checkpoint produces no duplicate domain effects`

**Shape:**

```text
1. Test starts a real PostgreSQL (testcontainer or CI service) and migrates the schema.
2. Test spawns the workflow host as a CHILD PROCESS, sharing that database.
3. Child is started with a fault hook enabled by environment:
     AWP_CRASH_AFTER_STEP_COMMIT=<stepName>
   The hook fires inside the step, AFTER the Drizzle transaction's commit
   has returned and BEFORE the step returns control to DBOS, and calls
   process.kill(process.pid, "SIGKILL") — no graceful shutdown, no flush,
   so DBOS cannot write the step checkpoint.
4. Test starts the workflow with a fixed operationId and waits for the
   child to exit with the SIGKILL signal, asserting the crash actually
   happened at the intended point: the step's marker row exists, and the
   step never returned to DBOS. Prove the second with a step-entry counter
   the child appends to on entry (a file or a test-only table), and assert
   after restart that the count went 1 -> 2 while every effect count stays
   at 1. Reading DBOS's checkpoint tables directly is permitted as test
   diagnostics only, never as the assertion of record. If the child exits
   normally, the test FAILS as inconclusive.
5. Test respawns the host WITHOUT the fault hook and lets DBOS recovery
   replay the un-checkpointed step.
6. Test waits for the workflow to reach a terminal state, then asserts,
   for that operationId, count = 1 for EACH of:
     - the domain row/state transition the step performs
     - business_events rows emitted by the step
     - audit_records rows emitted by the step
     - outbox_messages rows emitted by the step
     - workflow_step_outcomes rows for (operationId, stepName, stepKey)
   and that the replayed step returned the SAME outcome value recorded
   before the crash.
7. Separately, the test redelivers the outbox message to the consumer a
   second time and asserts the consumer's effect count stays at 1.
```

The fault hook is production-adjacent test scaffolding specified here and implemented with the workflow host; it must be inert unless the environment variable is set, and its presence must be asserted absent in production configuration.

A test that kills the process at an arbitrary time instead of at the specified point does **not** satisfy this gate: the whole risk lives in that one window, and a random kill will almost never land in it.

## Failure and Recovery Behavior

| Crash point | DBOS on restart | Domain outcome |
|---|---|---|
| Before the domain transaction commits | replays step | transaction rolled back; step runs for real, once |
| Between commit and checkpoint | replays step | marker hit; step returns recorded outcome, mutates nothing |
| After checkpoint | resumes at next step | nothing to reconcile |
| During external call, before recording step | replays external step | provider idempotency key collapses the duplicate; unknown results are reconciled via the provider's `reconcile` |
| Between outbox commit and broker publish | not DBOS's concern | dispatcher republishes; consumer deduplicates on message id |
| Between publish and `published=true` | not DBOS's concern | duplicate delivery; consumer deduplicates on message id |

Ambiguous external outcomes are never resolved by guessing: the step reconciles against the provider (`ReconcileRequest`) or transitions to a typed lifecycle failure with provenance, per [Durable Execution](durable-execution.md).

## Verify Against DBOS Docs At Implementation

The rule above is deliberately independent of DBOS API specifics, but the following are **not verified** here and must be checked against DBOS TypeScript documentation before implementation. Do not treat them as settled, and do not invent decorator signatures from this document:

- whether a DBOS step/transaction can expose a transaction handle usable by Drizzle, or whether AWP must open its own Drizzle transaction inside a plain step;
- whether DBOS shares the application connection pool or requires its own, and whether that changes connection-limit sizing;
- exact step retry policy, backoff and maximum-attempt semantics, and whether retries are configurable per step;
- the exact recovery trigger and timing after process restart, including whether recovery is automatic on init or requires an explicit call;
- whether DBOS guarantees a step is checkpointed before its return value is visible to the workflow body, and what it does with a step that crashed mid-flight;
- workflow ID assignment/idempotency semantics when a workflow is started twice with the same key;
- cancellation propagation into an in-flight step.

If any verified DBOS behavior conflicts with this document, the seam rule wins and DBOS usage is adjusted; if the conflict cannot be resolved, it requires an explicit Decision amending [ADR 0005](../adr/0005-dbos-durable-workflows.md).

## Increment Realization

| Increment | Seam realization |
|---|---|
| I0 | `workflow_step_outcomes` table, step-transaction helper in `packages/persistence`, outbox dispatcher with consumer dedupe, and the Required Proof test. |
| I1 | FactoryRun / AgentRun attempt dispatch / trusted publication workflows bound to the rule; gate blocks ship without the proof. |
| I2–I5 | CI request/reconciliation and evidence workflows reuse the same seam unchanged. |
| I6–I8 | Release/deployment rollout, machine enrollment and incident resolution reuse it. |
| I9 | marker retention/pruning policy and multi-tenant scoping of `operationId`. |

## Current Implementation State

`PostgresUnitOfWork.transaction` commits state + `business_events` + `audit_records` + `outbox_messages` atomically. The binding seam is now implemented on canonical I1 source: `workflow_step_outcomes` records application-owned `(operationId, stepName, stepKey)` completion markers; `WorkflowStepTransactionRunner` commits each domain mutation and marker in one Drizzle transaction and replays the recorded outcome; Task dispatch and ChangeSet auto-merge are decomposed into marked domain steps plus external provider steps; `OutboxDispatcher` is at-least-once and `IdempotentOutboxConsumer` commits consumer effects with durable receipt dedupe. The Required Proof is implemented in `tests/integration/dbos/durable-seam-crash.test.ts`: a real PostgreSQL + pinned DBOS child is SIGKILLed after the domain/marker commit but before the DBOS step checkpoint, then restarted under the same executor identity; handler entry reoccurs while Project/event/audit/outbox/marker counts remain exactly one and the recorded outcome is replayed. `tests/integration/outbox-delivery.test.ts` separately proves publish-before-mark redelivery with one consumer-visible effect.

The seam remains binding for future durable workflows. New workflows are not exempt merely because the first I1 realization is green; they must use the same marked-transaction/external-effect discipline and retain the crash-window proof.

## Deferred Realization

Marker retention/pruning, cross-workflow saga compensation patterns, per-step observability dashboards and any two-phase-commit or XA arrangement are deferred and trigger-only. Replacing the marker with a DBOS-native exactly-once primitive is permitted only if DBOS documents that primitive as commit-atomic with an application transaction.

## Temporary Dogfood Behavior

Dogfood may run the workflow host in the same process as the control plane and may dispatch the outbox with a simple polling loop rather than a broker. Neither shortcut relaxes any rule above: single-process co-location does not make the checkpoint atomic with the domain commit, and a polling dispatcher is still at-least-once.

## Decisions / ADRs

Governed by [ADR 0004](../adr/0004-postgresql-drizzle.md) and [ADR 0005](../adr/0005-dbos-durable-workflows.md). Any change to marker atomicity, step-boundary mutation rules, or the exactly-once claim requires an explicit Decision. Removing or weakening the Required Proof test requires the same.