# operate-prod — working in a live production environment

Audience: AI coding agents first. Scope: env serves real users/data/money. Creating the env → `setup-prod.md`.

## Posture

**Prod is read-only to you by default.** Every mutation flows through the pipeline (PR → CI → deploy) or a pre-approved runbook. No SSH/console hand-edits, no ad-hoc `wrangler`/`gcloud`/`aws` mutations from a session. If the pipeline can't do something needed, fix the pipeline — that gap is the task.

Before ANY prod action, confirm the target: which env, which DB, which account. Print the identity check (e.g. `SELECT current_database()`, account id, worker name) in your output BEFORE the action. Wrong-target is the classic agent prod incident.

## Deploys

- Only via CI, only from the default branch/tag. Verify AFTER every deploy: health 200, smoke green, error tracker quiet for the new release. Deploy-and-walk-away is incomplete work.
- Risky change → flag it off by default, deploy dark, enable gradually. Flag flips are logged config changes, not code edits.
- Regression discovered → **roll back FIRST, diagnose second** — especially on a money path. Forward-fix under pressure is how 21-commit fix-chains happen.

## Database in prod

- **Migrations: forward-only, immutable history.** NEVER edit an applied migration file (hash-locked). Mistake → new corrective migration.
- **Breaking schema change → expand-contract:** add new column/table → dual-write → backfill → switch reads → remove old in a LATER release. Never drop-and-recreate on live data.
- **Destructive/corrective SQL runbook (mandatory sequence):** snapshot first → `SELECT count(*)` with the exact `WHERE` you'll mutate → sanity-check the count against expectation → mutate in a transaction → verify → commit. A `DELETE`/`UPDATE` whose row-count surprised you gets ROLLBACK, not a shrug.
- Long/locking operations (index builds, big backfills): batched, off-peak, `CONCURRENTLY` where available.

## Money paths

- Live provider keys: use only for designated verification transactions (minimal amount, owner-approved, refunded after). NEVER generate test traffic against live keys.
- All money mutations idempotent (idempotency keys); webhook handlers tolerate replays + out-of-order delivery. Replaying a stuck webhook is a runbook action, not an improvisation.
- Money-path bug in prod = rollback/flag-off first. Correctness beats availability here.

## Secrets + access

- Rotation via manifest + pipeline (set → runtime-verify → old key revoked LAST). Never rotate by hand mid-incident unless the leak IS the incident.
- A secret that touched a log/output is burned: rotate it, don't debate it.

## Incidents

1. **Capture before fix:** error text, failing request, timestamps, release id — verbatim, first.
2. Stop the bleeding (rollback / flag off / provider pause) before root-causing.
3. Never delete/overwrite evidence (logs, bad rows) during diagnosis — snapshot, then repair.
4. Postmortem output = a new smoke check or gate rule that makes the class recur-proof; a fix without a regression lock is half done.

## Signals

- Error tracker checked after EVERY deploy — a new error class in your release is yours, address or explicitly justify benign; never silently pass (no-ignored-signals).
- Scoreboard/smoke regression = full stop on feature work in that repo until green.