# setup-prod — creating a production environment

Audience: AI coding agents first. Scope: from "prod planned" to "prod serving, dark". Design decisions → `plan-prod.md`; touching a LIVE env → `operate-prod.md`.

## The bar for every step

**A step is done when its live observation exists — a real request answered, a real row read, a real dashboard entry seen.** Code-exists, config-written, CLI-said-Success all fail this bar. Record the observation (URL + status, query + result) in the task output.

## Setup ladder — in order, no skipping

1. **Env config as code.** `[env.production]` (or host equivalent) fully populated in the versioned config — route/domain, vars, `ENVIRONMENT=production`, worker/service name. A comment-only or empty prod block = MISSING, not partial. Idempotent: re-deploy from clean checkout reproduces the env.
2. **Prod data store.** Separate branch/instance; prod credentials held ONLY by CI + break-glass. Run the migration chain from EMPTY to current on it — if the journal cannot rebuild the schema from scratch, STOP: fix migrations first (re-baseline if needed). A prod DB seeded by hand-applied drift is a time bomb.
3. **Secrets: manifest → set → runtime-verify.** Manifest in repo (name, scope, required-by; NEVER values). Set via CI/CLI, then **verify at runtime** — call a diagnostic endpoint that proves each secret's presence + shape (prefix/length, never the value). CLI put commands have reported Success while storing garbage. Missing/extra vs manifest → fail the deploy.
4. **DNS + TLS.** Apex AND www resolve and serve; a 522/523 on apex while www works = broken, launch-blocking. Verify with real requests from outside CI.
5. **Deploy pipeline.** Deploys happen ONLY from CI (tag- or default-branch-triggered). NEVER from a laptop — laptop deploys create unreproducible prod. First pipeline run happens NOW, not at launch.
6. **Rollback BEFORE second deploy.** Prove rollback (previous-version redeploy or host rollback command) once, while stakes are zero. Untested rollback = no rollback.
7. **Health endpoint.** Public bare 200/503; detailed checks (DB, queue, providers) behind auth header. Wire it into the deploy as the post-deploy gate.
8. **Smoke suite in the pipeline.** Post-deploy job hits prod: key pages (status + doctype), health, and **negative checks** (see below). Any fail → deploy marked failed loudly. An empty smoke config passing = forbidden (exit non-zero on zero checks).
9. **Test-bypass kill, proven.** Every `/api/test/*` route, auth bypass, seed endpoint: env-gated AND smoke-verified OFF in prod (expect 404/403 by live request, re-proven every deploy). This is a standing smoke check, not a one-time audit — 12 test routes reachable in prod is a breach, not debt.
10. **Error reporting + release tagging.** Error tracker wired with release/version + sourcemaps uploaded in the deploy job. A prod without error visibility is write-only.
11. **Backups + one restore drill.** Automated snapshot/dump; restore once into a scratch branch and read a sentinel row. Backup that never restored = hope, not backup.
12. **Quota check against plan tier.** Crons, queues, CPU ceilings counted vs the host plan (read the actual plan tier — never assume). Over budget → fail setup, don't discover at deploy N.

## Cutover extras (before real users/money)

- **Live money verification:** exactly one real minimal-amount transaction (owner's card) + its refund, observed end-to-end (provider dashboard + own DB row + webhook 2xx). Live keys runtime-verified same as step 3.
- **Real auth verification:** one real login (real phone/email OTP) on prod.
- **Webhook registration:** provider→prod URLs registered + a delivery observed. Remember webhook URLs are slow-reversible (blast-radius register).

## Anti-patterns — NEVER

- Hand-edit prod config/env in a console "just to unblock" — drift the pipeline can't reproduce.
- Prod-only code branches (`if (prod) { differentLogic }`) beyond env-gated toggles.
- Secrets or their values in repo, logs, or task output.
- Marking any of the 12 rungs done from code review alone.
- Deferring rungs 6/9/11 to "after launch" — they are cheapest before users exist and mandatory after.