# Journal v2 Spec — SQLite-backed run state, single writer

Audience: AI coding agents first. Canonical for I2 (self-verifying journal), I3 (intent log), I9 (version skew), V6 (Node single-writer). Plan context: `docs/plans/2026-07-07-design-gap-handoff.md` PART V/VI. Existing v1 journals: `runstate/<slug>.jsonl`, writers today = `src/runner.js` AND `lib/journal.sh` (the dual-writer problem this spec ends).

Storage decision (2026-07-09, supersedes the CRC-JSONL design): v2 state lives in SQLite via Node's built-in `node:sqlite` (`DatabaseSync`, zero npm deps, Node ≥ 24 verified). Crash consistency, atomic appends, and corruption detection are delegated to SQLite WAL — NEVER hand-roll CRC/torn-tail recovery. Event semantics (kinds, seq, intents, land exactly-once, migration rules) are unchanged from the prior design.

## Scope

v2 governs: slug journals (now `runstate/<slug>.db`), the improvements ledger (`runstate/improvements.db`), and any NEW runstate store. The supervisor side journal (`verb`-keyed records, `src/supervisor.js`) is OUT of scope until explicitly ported — when ported, its records become `kind: "supervisor.<verb>"`. Run LOG (`logPathForJournal` stream) is a derived human/UI stream, NOT authority — v2 additionally REQUIRES terminal run records (`run.done`, `run.paused`, `run.resumed`) be appended to the SLUG JOURNAL (today `run.done` exists only in the run log; that changes with v2 — specs reading "journal shows run.done" mean the slug journal). Liveness sidecar `runstate/<slug>.live.json` stays a separate flock file (see Module layout), NOT in the db.

## Non-negotiables

- Journal is APPEND-ONLY. NEVER rewrite, reorder, or delete a committed event. Enforced in-schema: `BEFORE UPDATE` / `BEFORE DELETE` triggers on `events` `RAISE(ABORT, 'journal is append-only')`.
- Node is the ONLY writer of `runstate/*` after migration. `lib/journal.sh` + `lib/liveness` write-paths become shims exec-ing `node src/state/cli.js <verb> …`. ALL bash READ paths of journal state also go through `src/state/cli.js` (e.g. `journal-export`) — raw file access is impossible post-migration (no jsonl exists) and MUST NOT be reintroduced via `sqlite3` CLI calls.
- Durability: db opened with `PRAGMA journal_mode=WAL; PRAGMA synchronous=FULL; PRAGMA foreign_keys=ON`. One implicit transaction per appended event; multi-event atomic groups use one explicit transaction. Append failure retries per taxonomy (`journal-append-failed`).
- Single writer enforced by existing per-slug hardlink lock, extended: lock content = `{pid, pidStartTime, host, formatVersion}`. Staleness check MUST verify pid + start-time both (pid reuse defense, I5). `pidStartTime` = field 22 of `/proc/<pid>/stat` (clock ticks since boot). SQLite's own file locking is defense-in-depth, NOT the primary mechanism.

## Schema (v2)

```sql
PRAGMA user_version = 2;
CREATE TABLE events (
  seq     INTEGER PRIMARY KEY AUTOINCREMENT,  -- ordering authority; AUTOINCREMENT forbids seq reuse
  ts      TEXT    NOT NULL,                   -- ISO-8601 UTC millis, informational only — NEVER order by ts
  kind    TEXT    NOT NULL,                   -- dot-namespaced event name
  payload TEXT    NOT NULL CHECK (json_valid(payload))  -- flat JSON object, v1-style fields
);
CREATE TRIGGER events_no_update BEFORE UPDATE ON events BEGIN SELECT RAISE(ABORT, 'journal is append-only'); END;
CREATE TRIGGER events_no_delete BEFORE DELETE ON events BEGIN SELECT RAISE(ABORT, 'journal is append-only'); END;
CREATE INDEX events_kind ON events(kind);
```

- `seq`: strictly monotonic per journal, assigned by SQLite. Ordering authority — NEVER order by `ts` (clock skew).
- `kind`: dot-namespaced event name (`<noun>.<verb-or-state>`) for every NEW kind; migrated legacy v1 kinds (e.g. `dispatch`) are exempt, grandfathered as-is. NOTE: current v1 slug-journal records are state records (`state: leased/implemented/…`); MOST lack `kind`, but some carry a legacy one (e.g. `kind: "dispatch"` alongside `state: "fix"` — see `src/runner.js`). v2 unifies per the Migration section rule: a v1 line WITHOUT `kind` becomes `kind: "task.state"`; a v1 line WITH a legacy `kind` keeps it — `state` + payload fields preserved verbatim either way. Run-log kinds keep their names when they move into the slug journal (Scope section). Post-migration consumers (resume, stats, UI) read `kind` only.
- `payload`: JSON object holding all non-envelope fields flat (v1 style, not nested).

## API (`src/state/journal.js` — sole writer)

- `open(dbPath)`: run open checks (below), return handle.
- `append(kind, payload)` → committed `seq`; `appendGroup([...])` atomic.
- `readAll()`, `readSince(seq)`, `lastSeq()` — the ONLY sanctioned read surface (control-api SSE polls `readSince`; UI watching uses `lastSeq`/`PRAGMA data_version` polling, not fs.watch).
- Export: `src/state/cli.js journal-export <slug>` prints one JSON object per line (`{"seq":…,"ts":…,"kind":…,…payload}`) — derived stream for grep/forensics/bash assertions, NEVER authority.

## Open checks + corruption (exact, run at every open)

1. `PRAGMA user_version` > 2 → fail-closed `journal corrupt: version-skew v=<n>` → escalate. NEVER best-effort read newer schemas (I9).
2. `PRAGMA integrity_check` != `ok` → fail-closed `journal corrupt: integrity` → run-fatal, resolver escalates (see `spec/RESOLVER.md`). NEVER attempt salvage/repair/`.recover`.
3. WAL crash recovery (torn last commit rolled back by SQLite on open) is automatic and internal — a rolled-back uncommitted append is NOT corruption and needs no `journal.recovered` event; the writer re-appends per taxonomy retry.

DO NOT "repair" a failing db. DO NOT export-and-rebuild to continue — a resumed run acting on a lying journal is worse than a halt.

## Intent records (I3) — non-git side effects only

Git effects stay idempotent by naming convention (branch/worktree names + reachability reconcile) — NO intent records for them. Every NON-git side effect (dep provisioning, file moves outside worktrees, notification sends, lock takeovers, snapshot writes) wraps:

```
{"kind":"intent.start","op":"provision-deps","key":"<slug>/<taskId>/provision-1", ...args}
   ... side effect executes ...
{"kind":"intent.done","key":"<slug>/<taskId>/provision-1","result":"ok"}
```

- `key`: unique per logical operation ATTEMPT; deterministic (slug/task/op/counter) so replay matches.
- Startup sweep (runner open + daemon re-attach): for every `intent.start` without matching `intent.done`: op's handler declares `redrive` (safe to re-run, idempotent op) or `rollback` (undo partial state) — declared in a registry `src/state/intents.js`, one entry per op. Unknown op in sweep → fail-closed escalate.
- Sweep outcome journals `intent.swept {key, resolution: "redriven"|"rolledback"}`.

## `land` exactly-once (I4)

`land.intent {integrationHead: <sha>}` before invoking `ship.sh land`. On re-entry with unmatched `land.intent`: check remote reachability of `integrationHead` (`git fetch` + `merge-base --is-ancestor <sha> origin/<main>` or PR-exists check per land_mode). Reachable → append `land.done {verified: "already-landed"}`, skip. Not reachable → re-run `ship.sh land` (wrapper itself is idempotent-safe per its contract). NEVER run land without an intent record.

## Migration v1 → v2

- Trigger: `runstate/<slug>.jsonl` exists and `runstate/<slug>.db` does not.
- Procedure: read every v1 line; insert as events in file order (seq 1..n; a v1 line WITHOUT `kind` gets `kind: "task.state"`; a v1 line WITH a legacy `kind` keeps it — either way `state` + all payload fields preserved verbatim in `payload`); append `journal.migrated {from: 1, to: 2, records: n}`; fsync via `synchronous=FULL` commit; rename `runstate/<slug>.jsonl` → `runstate/<slug>.jsonl.v1.bak`.
- BOTH `<slug>.jsonl` (non-bak) AND `<slug>.db` present at open → fail-closed escalate (interrupted migration; human inspects — the rename is the commit point).
- Migration runs once, before any other read/write, under the run lock. It is the ONLY code path that reads a v1 jsonl.

## Module layout + shims (V6)

- `src/state/journal.js`: open/append/read/export/migrate on `node:sqlite`. Sole journal writer.
- `src/state/liveness.js`: port of `lib/liveness` set/get/reap semantics (same sidecar path `runstate/<slug>.live.json`, same VALID_STATES `leased implemented gated reviewed committed`, same flock file). Adds `pgid` + `pidStartTime` fields (I5).
- `src/state/lock.js`: hardlink lock + pid/start-time staleness.
- `src/state/cli.js`: subcommand CLI (`journal-append`, `journal-export`, `liveness set|get|reap`, …) — the shim target.
- `lib/journal.sh`, `lib/liveness`: reduce to ≤10-line exec shims. Existing callers unchanged. Port acceptance gate: `lib/test-journal.sh` write-path behavior preserved — file-content assertions in that test are rewritten to assert via `journal-export` output (byte-level jsonl assertions cannot survive the storage change; the EXPORTED record stream must match what the old assertions demanded). Liveness tests pass UNMODIFIED (liveness storage unchanged).

## Tests (MUST exist)

1. Append/read round-trip; `UPDATE`/`DELETE` on events rejected by trigger; `appendGroup` atomic (kill mid-group → none committed).
2. Crash-kill fuzz: child process appends a scripted event stream; parent SIGKILLs it at randomized points (≥ 50 iterations) → reopen: `integrity_check` ok, events = exact committed prefix, `lastSeq()` consistent, no partial event ever visible.
3. Corruption: garble bytes mid-file in a db fixture copy → open fails closed `journal corrupt: integrity`, no auto-repair attempted.
4. Version skew: fixture with `user_version = 3` → refusal.
5. Migration: v1 fixture → db, per-record payloads byte-equal, `.v1.bak` intact; re-open idempotent (no double migration); jsonl+db both present → fail-closed.
6. Intent sweep: kill process between `intent` and `intent.done` for each registered op → sweep resolves per registry; `land.intent` already-landed path verified against a fixture remote.
7. Shim equivalence: `lib/test-journal.sh` green with read assertions routed through `journal-export` (write-path call sites unmodified); liveness tests green unmodified.