# Bulk Translation Robustness — Design

Date: 2026-08-23
Status: Draft v2 — rewritten after adversarial review (gpt-5.6-sol, 4 reviewers + synthesis)
Owner ask: *"I should be able to batch 100's and 1000's of posts without errors. I need a robust system, like it was before. What happened? I was able to translate 1000's of posts before."*

> **v1 → v2.** v1 argued the fix was a transient/terminal classification plugging into an existing
> browser-driven retry loop. Review proved that loop **does not close** (§2 D0). v1's billing-safety
> verdict was also false (§2 D6). This version is rebuilt around durable server-side reconciliation.
> Findings that survived review unchanged are marked **[verified]**; corrections are marked
> **[corrected v2]** with what v1 got wrong, so no one re-derives the disproven version.

---

## 0. Answering "what happened"

Batching replaced a **self-healing** per-pair flow with a **single-delivery** parent/child flow whose
only completion path is the webhook. When that delivery is lost, nothing recovers it — and the UI
renders the wreckage as translation failures. Nothing about Gemini or the translation itself got
worse; the recovery paths got removed.

Five distinct false-failure sources are live at once. Only the first was in the original bug report,
and the last two were found during review:

| # | Surfaces as | Real cause | Sev |
|---|---|---|---|
| D0 | rows stuck, then failures | **The bulk poll fallback is dead code.** Webhook is the sole delivery path. | BLOCKER |
| D1 | `Translation job is currently being finalized.` | transient `GET_LOCK` timeout written as a terminal item failure | BLOCKER |
| D2 | `HTTP 524` | indeterminate request outcome condemned as failure | MAJOR |
| D3 | `Translation polling timed out.` | client gives up after ~3 min and hard-fails every pending ref | MAJOR |
| D4 | `Timed out — no response received` | blanket 1-hour SQL sweep over `processing` jobs | MAJOR |
| D6 | *(silent)* | **paid-but-not-applied translations; retry re-bills** | BLOCKER |

---

## 1. Current architecture [verified, corrected v2]

Read from `origin/master`; the primary checkout is on a stale branch predating bulk-content.

```
browser --POST /translate-async-batch (<=20 pairs)--> WordPress
  TranslateController::translateAsyncBatch()      <-- HARD CAP 20, HTTP 400 above (:93-95, :1224-1233)
    TranslationJobDispatcher::dispatch_content_batch()
      per pair: prepare_content_job()             <-- ~6.25s synchronous WP work
      group batchable candidates by source language
      re-chunk by $limits['maxItems'] (from /health)
      POST /v1/jobs/bulk-content per chunk        <-- 30s timeout EACH (:687-702)
    <- { batch_ids, submitted[], skipped[], protected[], queued[], failed[] }
```

Backend (`origin/master`):
- `api/src/routes/jobs.ts:1239` — `POST /v1/jobs/bulk-content`. Returns 202 **after** validation,
  credit gate, a DB transaction and an awaited immediate queue-drain attempt — not instantly.
  [corrected v2: v1 said "immediately".]
- `api/src/worker.ts:923` — `translationQueue.process('bulk-content', jobConcurrency, ...)`, then
  `for (const item of data.items)` — strictly serial, one `translateStructured()` per item.
- `api/src/types/index.ts:1032` — `BULK_CONTENT_MAX_ITEMS = 20`, advertised at `health.ts:44` under
  `capabilities.bulkContentTranslation.maxItems`. [corrected v2: v1 wrote `capabilities.bulkContent`.]

### 1.1 D0 — the bulk poll fallback is dead code [BLOCKER, verified v2]

This is the regression. `TranslateJobsController.php:449-459`:

```php
$poll_results = is_array( $poll_data['results'] ?? null ) ? $poll_data['results'] : array();
if ( 'completed' === $poll_status && array() !== $poll_results ) {
    $this->completeBulkContentJob( ... );   // <-- never reached
}
```

But the backend's poll response has **no `results` key**. `translationService.getJobStatus()`
(`origin/master:./api/src/services/translationService.ts:827-845`) spreads `mapStoredTranslation(job.translation)`,
whose return type (`:87-93`) is exactly
`{ translation?, translatedTitle?, translatedExcerpt?, translatedContent?, translatedFields? }`.
The worker stores the per-item array as a JSON **string** under `translation`
(`worker.ts:1010`, `const storedTranslation = JSON.stringify(results)`), and puts `results` only in
the **webhook payload** (`:1011-1019`) — never in the polled job row.

So `$poll_results` is always empty, `completeBulkContentJob()` is never called from polling, and the
webhook is the sole delivery path for every batched translation. Consequences:

- a dropped, 500-ing, or mis-signed webhook strands the whole batch permanently;
- everything v1 built on "the poll path retries it" was false, including v1's central §3.1 claim;
- the pre-batching single-content path *did* have a working poll fallback, which is precisely why
  the owner's sweeps used to survive. **This is the answer to "what happened".**

**Fix (highest value in this document): make the fallback real.** In the bulk branch, when
`$poll_results` is empty and `$poll_data['translation']` is a JSON array string, decode it and use it
as the results array. Cheapest correct option, needs no backend deploy, restores the missing
self-healing path. Optionally also add `results` to the backend status contract for explicitness —
but the plugin must not depend on that landing first.

### 1.2 Poll-driver limits [verified]

`syncProcessingJobs()` has two shapes: untargeted (`:239`, from the plural jobs-list endpoint,
`ORDER BY updated_at ASC LIMIT 3`) and targeted (`:354`, `:1374` — the singular job GET the Generate
All client polls). The selection at `:369-386` excludes batch children via `NOT EXISTS`, so children
reconcile only through their parent. Once D0 is fixed, the targeted call is what drives per-parent
recovery during a sweep; the `LIMIT 3` path is far too slow to be the engine for hundreds of parents.

---

## 2. Defects

### D1 — Transient finalizer errors recorded as terminal [BLOCKER]

`TranslateJobsController.php:877`, inside `completeBulkContentJob()`:

```php
$finalized = $this->finalizer->finalize( $child_job, $data );
if ( is_wp_error( $finalized ) ) {
    $item_status = 'failed';
    $item_error  = sanitize_text_field( $finalized->get_error_message() );
}
```

Any `WP_Error` condemns the item — including `ipz_job_locked` (`TranslationFinalizer.php:115`), a
5-second `GET_LOCK` timeout meaning only "someone else holds it right now". The two racers are
`completeBulkContentJob()` called from the parent poll (`:451-459`) and from the bulk webhook
(`:2323-2325`). [corrected v2: v1 cited `:508` — that is the generic non-string path, and `:369-386`
excludes bulk children from it.]

Only the **item** row is failed; the child **job** row is left `processing`, so the state is also
internally inconsistent.

**Transient set is larger than v1 claimed** [corrected v2]: `ipz_acf_source_lock_failed`,
`ipz_acf_source_locked`, `ipz_acf_mutation_retry` all document retry intent, alongside
`ipz_job_locked` and `ipz_job_lock_failed`. `ipz_source_revision_commit_failed` needs a decision, not
an assumption. Enumerate every code and classify each explicitly.

**Retry state has nowhere to live** [BLOCKER, corrected v2]. v1 called this "a one-line change". It is
not. `presszone_international_job_items` (`Core/Database.php:583-599`) has `created_at` and
`completed_at` and **no attempt counter and no `updated_at`**. This requires a named migration.

### D2 — Indeterminate request outcome condemned [MAJOR]

`content-translate.js:1486` `applyGenerateAllChunkFailure()` pushes every pair in the chunk to
`progress.failed`. A 524 is **indeterminate** — WordPress may have created none, some, or all of the
rows. [corrected v2: v1 asserted the server "ran to completion and did create the jobs". Unproven.]
Likewise `20 × 6.25s ≈ 125s` is a **hypothesis** pending `timing_ms`, not an established fact.

### D3 — The client hard-fails after ~3 minutes [MAJOR, new in v2]

`content-translate.js:69` `GENERATE_ALL_MAX_STALE_POLLS = 36` at a 5s interval. At `:1647-1661`,
after ~3 minutes with no ref resolving, **every** still-pending ref is pushed to `progress.failed`
with "Translation polling timed out." A 20-item batch needs ~64s of serial Gemini work, and with many
parents queued behind each other the backend routinely exceeds three minutes of no *reconciliation*
progress. At the owner's scale this likely produces more red than the 524 does.

### D4 — Blanket 1-hour timeout [MAJOR, corrected v2]

`TranslateJobsController.php:242-255` fails any job `pending`/`processing` with
`created_at < NOW() - INTERVAL 1 HOUR`, excluding only Site Content — bulk-content parents included.
[corrected v2: it is **not** autonomous. It runs only when the plural jobs-list endpoint is called;
Generate All polls the singular endpoint, so it fires when a human opens the jobs list, not on a
timer.] Still wrong for legitimately long batches, and unpredictable.

### D5 — `Translation content could not be saved.` [MAJOR, reclassified v2]

`TranslationFinalizer.php:419-449` (existing target: falsy `update_translation()`, and separately a
read-back mismatch) and `:462-466` (new target: the two combined). [corrected v2: v1 assumed this was
a race and grouped it with the false failures. **Treat it as a genuine local persistence failure
until instrumented otherwise** — and note it is a *paid* translation being lost, which makes it D6.]

### D6 — Paid-but-not-applied, and retry re-bills [BLOCKER, new in v2]

`worker.ts:969-993,1024-1064`: the backend counts a successful Gemini item and commits the credit
deduction **before** WordPress ever applies the result. If finalization then fails (D1, D5) or the
webhook is lost (D0), the customer has **paid for a translation they did not receive**. Any recovery
that re-translates charges them twice.

v1's "double-billing safe" verdict was **false** — it reasoned only about resubmission before
completion and missed the remote-success/local-failure state entirely. Billing therefore cannot stay
out of scope.

The correct primitive: **persist the translated output on arrival, then retry local application from
the stored payload — never by re-translating.** The three states to model explicitly are:
1. remote failure — not billed — safe to retry from scratch;
2. remote success, local apply failed — **already billed** — retry local apply only;
3. submission outcome unknown — resolve by durable identity, never by blind resubmission.

---

## 3. Design

**Phase 1 is D0 + D6 + durable state.** Client UX (D2, D3) is worthless until a translation that was
paid for reliably lands.

### 3.1 Restore the recovery path (D0)

Decode `$poll_data['translation']` as the results array in the bulk branch when `results` is absent.
Guard on it being a JSON array of item shapes; ignore a structured-fields object. Test both the
current backend shape and a future one that does send `results`.

### 3.2 Persist the payload; separate delivery from application (D6)

On webhook or poll arrival, **store each item's translated payload durably before finalizing**
(`job_items.translated_content` already exists for this). Application then becomes a retry-safe local
operation against stored data. A retry never re-enters Gemini and never re-bills.

### 3.3 Durable retry state and atomic ownership (D1)

Named migration on `presszone_international_job_items`, with DB-version registration, fresh-install
schema update, and upgrade/backfill:
- `finalize_attempts INT UNSIGNED NOT NULL DEFAULT 0`
- `last_finalize_attempt_at DATETIME NULL`
- `last_finalize_error_code VARCHAR(64) NULL`

Then at `:877`: classify by code. Terminal → `failed` as today. Transient → leave `pending`, stamp
attempt state, return non-2xx **without** recording delivery so the backend outbox replays.

Two rules that keep this from degenerating:
- **A lock-busy no-op must not consume an attempt.** Otherwise contention alone exhausts the budget.
- Claim the item atomically (conditional `UPDATE ... WHERE status = 'pending'` as a lease) before
  finalizing, so two concurrent deliveries cannot both process one item.

Bound retries by **attempts and wall time**, with exhaustion producing an honest distinct message.

### 3.4 Server-side sweep state (D2, and tab-close)

There is no server-side resume today: close the tab and the sweep's remaining work is simply gone,
and in-flight parents are recovered only by whatever poll happens later. Additionally, a child
committed by `prepare_content_job()` before `submit_content_batch_chunk()` runs (`:497-523`,
`:1136-1174`) is orphaned at `processing` with no parent, no item row, and no backend id if PHP dies
in between — and replay then finds it via the active-job guard and returns `already_queued` forever.

Phase 1 therefore persists sweep intent server-side: the pair manifest plus a cursor, and child
states `prepared` / `submitted` / terminal. A scheduled reconciler submits or reconciles `prepared`
rows and resolves orphans. `already_queued` is returned only when there is submission proof or a
recoverable outbox record.

With that in place the client's role shrinks to *displaying* a sweep the server owns — which is what
makes 1000s of posts survivable across a closed laptop lid.

### 3.5 Stale policy (D3, D4)

- Remove `BULK_CONTENT_TYPE` from the blanket 1-hour update.
- Client stale polls resolve to **pending/unknown**, never `failed`.
- A scheduled reconciler decides terminality from backend state, pending item count, last real
  progress, and exhausted retries — not from how long a browser tab watched.

### 3.6 Request deadline and the 20-pair cap (corrected v2)

v1 proposed deleting the client constant and sending 100-pair chunks. **That is impossible as
written**: `TranslateController::BATCH_MAX_PAIRS = 20` (`:93-95`) rejects anything larger with HTTP
400 (`:1224-1233`).

v1's budget was also not a request bound: it stopped only the *preparation* loop, after which the
dispatcher submits singles and backend chunks serially at up to **30 seconds per POST**
(`:687-702`), plus a health lookup. Several chunks can blow well past the edge timeout with the
budget "respected".

Phase 1: **keep the 20-pair cap** and define one absolute deadline covering validation, preparation,
health lookup, every outbound submission, serialization and a response reserve — never floored above
the real PHP/FPM ceiling. Prepared-but-unsubmitted work goes to the durable outbox (§3.4), not to a
`deferred[]` bucket the client is trusted to resend.

The real cure is moving preparation off-request entirely (§5). Raising the cap is a decision to
revisit only after that.

### 3.7 Cheap pre-check on the active-job guard [verified, retained]

The guard at `TranslationJobDispatcher.php:1107` runs *after* the ACF/SEO work at `:962-1092` that is
effectively the whole ~6.25s, yet keys only on `(post_id, target_lang)` — `content_hash` (`:1096`) is
not an input. Hoist a cheap existence pre-check above `extractSourceState()`; keep the existing
post-lock check as authoritative revalidation. Pair-lock ordering must be preserved, and the early
check must not skip a pair the late check would have admitted.

Worth doing, but it is an **optimization, not a correctness fix** — it does not belong ahead of
§3.1–§3.4.

### 3.8 Client classification and counters (D2)

- Classify on HTTP status and machine error codes, not message substrings.
  `isGenerateAllBlockingError()` (`:1397-1400`) matches substrings and misses auth, forbidden,
  licensing and credits entirely.
- Indeterminate (network/timeout/5xx/524) → **unknown**, never `failed`.
- Nonce/session expiry mid-sweep must pause and re-authenticate, never condemn pairs.
- **Counter transitions must be disjoint.** `queued[]` currently increments `processed` immediately;
  making those entries pollable without changing that double-counts. Job-backed `queued` entries must
  not increment `processed` until terminal.

### 3.9 Bound the completion list

`renderGenerateAllContent()` (`content-translate.js:1244-1264`) passes the entire failure array to
`TranslationResultList` uncapped — literally the screenshot. Cap rendering (~50/bucket + "and N
more"), keep counts exact, retain only a bounded sample of details, and generate the pair
cross-product lazily rather than materializing all 12,000 pairs, chunks and maps.

### 3.10 Thread `timing_ms`

`TranslateController.php:759-770` drops the dispatcher's `timing_ms`. Add it.
`ContentBatchDispatchContractTest.php:56-59` pins the old return shape by regex and must be extended.

---

## 4. Backend — deferred out of this pass (corrected v2)

### 4.1 Serial Gemini loop
`worker.ts:952` is serial, one call per item. v1 recommended a per-job pool of 4 as "low risk" —
**withdrawn**: the worker already runs up to 8 bulk jobs per process, so 4×8 = 32 concurrent Gemini
calls per process before replicas, and `translateStructured()` has model fallback rather than an
explicit rate-limit backoff. Needs a process-wide/distributed concurrency and rate-limit design
honouring `Retry-After`, preserving one lease heartbeat, and using all-settled per-item isolation.
Choose the number from quota evidence, not by guess.

### 4.2 Incremental webhooks — **removed from this implementation pass**
v1 claimed "no plugin-side change required, just distinct delivery ids". **False.** `WebhookOutbox`
has `@@unique([job_id, event])` (`api/prisma/schema.prisma:385-400`) and the worker writes one
`bulk_content_translation.completed` row — multiple partial rows violate the constraint. And two
partial callbacks can both select the same pending item before either conditional update, so distinct
ids do not provide atomicity. Requires its own design: outbox migration, sequence/event identity,
stable ids across retries, final marker, ordering/replay rules, atomic item claims, and billing
checkpoint ordering. Revisit only after §3.1–§3.4 ship.

---

## 5. Out of scope (this pass)

- Moving the ~6.25s/pair preparation fully off-request — the real throughput cure, and the
  precondition for ever raising the 20-pair cap.
- Raising `BATCH_MAX_PAIRS`.
- Backend concurrency (§4.1) and incremental delivery (§4.2).
- The unauthenticated `GET /connect` route (separate workstream).

Billing/credits are **explicitly in scope** (D6), reversing v1.

---

## 6. Verification (corrected v2)

**The v1 E2E waiver is withdrawn.** It cited a dangling `@playwright/test` in
`plugins/multilingual-press-zone` — an unrelated tree — while this plugin has its own canonical
always-remote lane (`.claude/commands/ipz-e2e.md`, `.claude/skills/ipz-e2e.md`, `e2e-remote`,
`tests/e2e/remote-stack.sh`). A change to long-running browser orchestration, retries, auth handling,
counters and modal completion cannot ship on unit tests alone. Run the canonical lane; if it fails,
record the exact command and failure **from this worktree** rather than attributing it elsewhere.

- `php -l`, `phpcs`, `./vendor/bin/phpunit`. Fresh worktree has no `vendor/` (gitignored):
  `ln -sfn <main-checkout>/plugins/international-press-zone/vendor vendor`.
- Unit coverage: poll-body decode for both backend shapes; transient vs terminal classification;
  lock-busy not consuming an attempt; atomic item claim under two concurrent deliveries; retry
  exhaustion; migration up/backfill; disjoint counter transitions; deadline exhaustion routing work
  to the outbox.
- Remote Playwright scenarios: timeout/unknown recovery, stale polling, **tab close + reload +
  resume**, duplicate callbacks, nonce expiry, counter invariants, bounded rendering.
- dev1 is **not** smoke-tested (standing rule). Report the deploy verifier output and have the owner
  re-run Generate All; `timing_ms` from that run attributes the 6.25s.

## 7. Acceptance

A sweep of several hundred posts, **with the browser tab closed partway through**, completes with:
1. every pair reaching a correct terminal state without the tab being open;
2. zero `HTTP 524`, `Translation job is currently being finalized.`, or
   `Translation polling timed out.` entries;
3. incomplete pairs reported as unknown/pending, never failed;
4. **no pair translated or charged twice**, and no paid translation silently lost;
5. `timing_ms` present per chunk.