# Asynchronous Translation Manifest Design **Status:** implementation contract **Scope:** `plugins/international-press-zone/` and `press-zone-backend/` **Supersedes:** the dispatch and callback-delivery portions of `2026-08-23-bulk-translation-robustness-design.md` ## Goal Make Generate All and `POST /wp-json/international-press-zone/v1/translate-async-batch` asynchronous at the WordPress boundary. A request persists one durable manifest and returns `202 Accepted` without preparing translations or contacting the backend. Durable workers then prepare, batch, submit, reconcile, and finalize every intent. The release must prove at least 1,000 accepted intents, bounded worker execution, real multi-item backend payloads, crash/retry safety, callback-driven completion, aggregate accuracy, and Generate All reload recovery in Chromium and Firefox through the canonical remote k3s lane. ## Non-goals - Replacing the existing translation finalizer, field-protection rules, billing model, Bull workers, or backend queue dispatcher. - Introducing a second unfinished-feature flag. Incomplete surfaces use `ipz_unreleased`. - Making browser storage authoritative. - Running local WordPress, local builds, local profiling, or workstation test gates. - Testing or smoking dev1. ## Current problem The REST route currently invokes `TranslationJobDispatcher::dispatch_content_batch()`. That method resolves canonical sources, loads translation contexts, computes protected fields, creates local jobs, and contacts the backend before the request returns. Generate All amplifies the problem by constructing the full cross-product in the browser and sending concurrent 20-pair requests. Large requests therefore exceed edge timeouts even though the backend itself already has a durable PostgreSQL queue payload, queue leases, processing leases, and a completion webhook outbox. The missing boundary is a durable WordPress-side admission manifest and bounded worker. ## Architecture choice ### Considered approaches | Approach | Robustness | Scale and performance | Maintainability | Reversibility | Decision | |---|---|---|---|---|---| | Option-backed queue plus WP-Cron | Options are a contention and size bottleneck; weak claims and poor observability | Degrades with thousands of intents | Simple initially, difficult recovery semantics | Reversible | Rejected | | Dedicated manifest tables with WP-Cron only | Durable state and queryable retries | Scales, but progress depends on visitor/cron availability | Clear model | Reversible | Rejected as sole runner | | Dedicated manifest tables with bundled Action Scheduler, plus independent recovery wakeups | Durable claims, established scheduling, bounded actions, stale-work recovery | Handles large manifests without request-time work | Explicit ownership and testable seams | Two-way door; tables and scheduler can be replaced independently | Selected | Action Scheduler is the primary in-WordPress runner. The plugin bundles `woocommerce/action-scheduler:^3.9` as a production Composer dependency, requires its bootstrap before `plugins_loaded`, and uses group `international-press-zone-manifests`. Guaranteed progress comes from an external runner invoking the plugin WP-CLI drain command at least once per minute; Action Scheduler's async loopback is the low-latency path. WP-Cron and `spawn_cron()` are additional recovery wakeups only. Enabling `ipz_unreleased` in an environment is blocked unless the external runner and its health check are provisioned. Even when the flag is enabled, admission calls `TranslationManifestScheduler::is_ready()` and returns `503 ipz_manifest_worker_unavailable` unless Action Scheduler is loaded and the external-run timestamp is no older than 150 seconds. Deployment invokes the CLI runner once, verifies this readiness response in the non-dev1 lane, and only then enables admission. ## System boundaries ### `TranslationManifestRepository` Owns all manifest and intent SQL, immutable admission data, claims, leases, retry timing, transitions, and authoritative aggregate recomputation. Callers do not issue manifest-table SQL. Primary seams: ```text create(user_id, request_key, request_fingerprint, pairs, selection): CreateResult get_for_actor(public_id, user_id): ManifestView|not_found claim(public_id, owner_token, lease_seconds): ManifestLease|not_claimed renew(public_id, lease_token, lease_seconds): bool release(public_id, lease_token, next_attempt_at, error?): bool expand_intents(manifest_id, manifest_lease_token, limit): int claim_intents(manifest_id, manifest_lease_token, limit, now): ClaimedIntent[] record_prepared(intent_id, manifest_lease_token, intent_claim_token, local_job_id, source_language, prepared_envelope): bool persist_submission(manifest_id, manifest_lease_token, claimed_intents[], submission_draft): PersistedSubmission load_due_submission(submission_id, submission_lease_token): PersistedSubmission record_submission_accepted(submission_id, submission_lease_token, remote_job_id): bool transition_intent(intent_id, manifest_lease_token?, intent_claim_token?, expected_generation, expected_states, state, metadata): bool synchronize_local_job(local_job_id, backend_event_sequence?): AffectedManifest[] retry_failed(public_id, actor_id, intent_public_ids?): ManifestView|error recompute(manifest_id): ManifestView recover_stale(now, limit): public_id[] ``` Repository methods use compare-and-swap predicates. Every worker write carries the active manifest lease token; every claimed-intent write also carries that intent's claim token, expected generation, and expected state. Submission creation atomically fences the exact claimed set. Every submission write carries its own lease token. A stale manifest, intent, or submission owner therefore cannot mutate data after replacement. Multi-row admission, expansion, submission creation, retry, and aggregate recomputation run in explicit InnoDB transactions with `READ COMMITTED`, rollback on any error, and at most three deadlock retries with bounded jitter. ### `TranslationManifestProcessor` Orchestrates one bounded action for one manifest. It contains no direct SQL and does not own translation rules. ```text process(public_manifest_id): ProcessResult ``` One invocation: 1. Claims the manifest with a random fencing token and a 120-second lease. 2. Expands at most 100 still-unexpanded immutable request pairs into idempotent intent rows. 3. Claims pending/retry-due intents in immutable ordinal order. 4. Revalidates the initiating actor’s current object-level permission and canonical translation-group authorization. 5. Uses the dispatcher preparation seam for each intent. 6. Persists versioned prepared envelopes before network submission. 7. Claims prepared intents and groups compatible items. The dispatcher builds a pure `SubmissionDraft` from the prepared values and an injected UUID; it performs no SQL or I/O. The repository atomically persists the submission row, exact canonical secret-free request bytes/fingerprint, local parent/item rows, and linked intent IDs/generations/claim tokens before network I/O. 8. Reloads the persisted submission and passes only its stored UUID and stored canonical body to the dispatcher network seam. The sender never accepts prepared objects and never reserializes the body. It reads the stable site callback secret at send time and places it in `X-IPZ-Callback-Secret`; the API key remains in its existing authorization header. Neither secret enters the persisted body. Backend submission is create-or-return-existing under that UUID; replay with a different fingerprint is `409`. A lost response is repaired by resending the identical stored bytes, which returns the original remote job ID. 9. Records the remote identity under the submission lease and moves only the exact current-generation linked intents to `queued`. 10. Recomputes aggregate state, releases the lease, and schedules the next action when nonterminal work remains. Each invocation claims at most 8 preparation intents, stops beginning new preparation after 25 seconds, submits at most 4 backend chunks, and renews its lease between preparation and submission. These are upper bounds, not browser-facing batch sizes. Backend capacity still limits each submitted chunk. A process crash may leave an intent in `preparing`, a prepared envelope, or a `submitting` row without a recorded remote identity. Stale recovery resets expired claims. Preparation reuses the dispatcher’s idempotency and active-job detection. An `Existing` preparation outcome must link the intent to the discovered local job and map its current local state to `queued`, `processing`, or its terminal state; it is never silently skipped. Submission recovery only resends the immutable stored request with its original submission UUID. Backend create-or-return-existing semantics close the accepted-but-response-lost boundary without guessing from local rows. Manifest-created local job idempotency keys and callback refs include the immutable intent public ID plus generation (`ipz-manifest:{intent_public_id}:g{generation}`). An active job with the same current-generation key is reusable. A compatible non-manifest active job may be linked only when canonical source/target identity and source fingerprint match. A terminal prior-generation job is never reused after explicit retry; the new generation creates a new local job/ref. A current-generation terminal job may be mapped only when replaying that same generation. ### `TranslationManifestScheduler` Hides Action Scheduler and recovery-wakeup details. ```text schedule(public_manifest_id, when?): void schedule_recovery(): void register_hooks(): void ``` - Action hook: `ipz_process_translation_manifest`. - Recovery hook: `ipz_recover_translation_manifests`. - Scheduling uses Action Scheduler's unique action flag and group, but uniqueness is only an optimization: the repository lease is the concurrency authority when a running and newly queued action overlap. - Recovery runs every five minutes, scans a bounded 100 manifests, and reschedules manifests with expired leases or due retries. Concurrent enqueue/recovery races are safe because both may enqueue while only one processor can acquire the manifest lease. - Admission commits the manifest before enqueue. A crash in that gap is repaired by the independent external runner's due-manifest scan; no enqueue marker is treated as authoritative. - If Action Scheduler enqueue throws or is unavailable, admission still returns `202` after durable persistence, records the scheduling error, and requests secondary recovery wakeups. Status exposes delayed scheduling rather than losing the manifest. - `wp ipz translation-manifests run --time-budget=50 --max-manifests=20` is registered by the plugin. An environment-level k3s CronJob or system cron invokes it at least once per minute and records exit status/duration. Readiness is degraded when `last_external_run_at` is older than 150 seconds or due work is older than 3 minutes. ### `TranslationJobDispatcher` seam change Existing source resolution, field protection, translation-context creation, local-job idempotency, and backend submission remain authoritative. They are extracted behind manifest-safe seams rather than reimplemented. ```text prepare_content_intent(post_id, target_language, initiator_id, selection, intent_public_id, generation): PreparedIntent|Skipped|Protected|Existing|WP_Error create_content_submission_draft(prepared_intents, submission_uuid): SubmissionDraft|WP_Error send_persisted_content_submission(submission_uuid, canonical_request_bytes): SubmissionOutcome|WP_Error ``` `PreparedIntent` contains the local job ID, stable ref, canonical source identity, target identity, source language, target language, structured source fields, source revision/fingerprint, protocol version, and backend idempotency data. Its canonical JSON envelope has `schema_version`, SHA-256 checksum, and sorted field keys; it contains no API secret. Source text receives the same database-at-rest protection as WordPress post content and is cleared from intent/submission rows after backend acceptance, while `translation_context` remains on the existing local job. Before submission the processor rechecks the source fingerprint; a mismatch discards the envelope and returns the intent to `pending` without consuming a translation attempt. `create_content_submission_draft()` accepts arbitrary structured fields and returns canonical non-secret request bytes plus local parent/item row values, but writes nothing. Its compatibility key is `(site identity, authenticated plugin account, source language, tone/options, callback URL identity, structured protocol version, capacity bucket)`; different field-key sets may share a batch. `TranslationManifestRepository::persist_submission()` is the single atomic owner for the submission row, linked intent generations, and existing local parent/item inserts. `send_persisted_content_submission()` accepts only persisted identity/body, performs the authenticated HTTP call, and cannot recreate local rows or serialization. It obtains `Settings::get_existing_callback_secret()` at send time, rejects an absent secret, and sends it only in `X-IPZ-Callback-Secret`. Callback-secret rotation is blocked while any local job/manifest is nonterminal, so replay uses the same stable secret. `dispatch_content_batch()` remains as a compatibility wrapper for non-manifest callers and tests; the asynchronous REST route no longer invokes it. ### REST controller The existing route becomes an admission endpoint; manifest status and retry routes are added under the same namespace. #### Create `POST /wp-json/international-press-zone/v1/translate-async-batch` Request: ```json { "request_id": "client-generated UUID", "pairs": [{"post_id": 123, "target_language": "he"}], "selection": {"only_missing": true} } ``` Admission performs only: - WordPress nonce/authentication and required capability checks; - JSON shape, supported target language, eligible post type, duplicate-pair removal, and configured maximum-intent validation; - one SQL load plus WordPress post/meta cache priming for all unique post IDs, followed by exactly one `current_user_can('edit_post', id)` call per unique post (duplicate pairs reuse that result); - immutable request fingerprinting and manifest persistence. It does not resolve translation groups, collect ACF/SEO fields, prepare content, create local translation jobs, or call the backend. Admission accepts at most 2,000 canonical intents and 1 MiB of JSON. Target codes are lowercase, selection defaults are materialized, duplicate pairs are removed, pairs are sorted by `(post_id,target_language)`, and RFC 8785-style canonical JSON is hashed for replay identity. The three-second SLO is measured in k3s with 1,000 distinct posts after cache priming; timing instrumentation separately records decode, load, authorization, transaction, and enqueue. Success is `202 Accepted`: ```json { "batch_id": "public manifest UUID", "status": "preparing", "total": 1000, "status_url": "/international-press-zone/v1/translation-manifests/{batch_id}" } ``` `request_id` is mandatory and unique per initiating user. Replaying the same key and same canonical request fingerprint returns the existing manifest with `202`. Reusing the key with a different fingerprint returns `409 ipz_request_id_conflict`. The configured admission maximum is at least 2,000 intents so the required 1,000-intent proof is one request. Oversized input returns `413` without partial persistence. The public response never exposes numeric database IDs, internal paths, stack traces, callback secrets, API keys, or post content. #### Status `GET /wp-json/international-press-zone/v1/translation-manifests/{batch_id}` Only the initiating user or a site administrator with the exact site-local `manage_options` capability can read it; no network-admin or cross-site implicit override exists. Unknown and unauthorized IDs both return the same bounded-time `404`. Aggregate counts are visible to an authorized administrator, but each failure row is returned only when the reader also has `edit_post` for that row's source post. Response includes: ```json { "batch_id": "uuid", "status": "processing", "counts": { "total": 1000, "preparing": 20, "queued": 300, "processing": 100, "completed": 560, "failed": 10, "protected": 5, "skipped": 5 }, "retryable_failed": 8, "created_at": "ISO-8601", "updated_at": "ISO-8601", "completed_at": null, "failures": [], "failure_pagination": {"page": 1, "per_page": 25, "total": 10} } ``` Failure details contain public intent ID, source post ID, target language, stable error code, sanitized message, retryable flag, and attempt count. Pagination is server-side and capped at 100 rows per page. A status read may enqueue a missing recovery action after checking durable state, but it never processes intents inline. #### Selective retry `POST /wp-json/international-press-zone/v1/translation-manifests/{batch_id}/retry` Request optionally includes `intent_ids`. Omission selects all currently retryable failed intents. Only the initiating user may retry. The route rechecks `edit_post` for every selected source, validates every requested intent belongs to the manifest, atomically opens a new intent generation, recomputes counts, schedules processing, and returns `202`. A `manage_options` reader cannot retry another user's manifest. Protected, skipped, completed, nonretryable, or over-limit intents are never reset. An empty eligible set returns `409 ipz_nothing_to_retry`. ### Database schema `Database::DB_VERSION` advances and `Database::create_tables()` contains the final schema. An explicit idempotent migration creates/upgrades the tables. Plugin bootstrap performs the version check before route registration so non-admin requests cannot hit absent tables. #### `{$wpdb->prefix}ipz_translation_manifests` - `id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY` - `public_id CHAR(36) NOT NULL UNIQUE` - `user_id BIGINT UNSIGNED NOT NULL` - `request_key CHAR(36) NOT NULL` - `request_fingerprint CHAR(64) NOT NULL` - `request_payload LONGTEXT NOT NULL` - `selection LONGTEXT NOT NULL` - `status VARCHAR(20) NOT NULL` - `total`, `preparing`, `queued`, `processing`, `completed`, `failed`, `protected`, `skipped` unsigned counters; `expanded_count` is never stored and is derived in the recomputation transaction - `attempt_count INT UNSIGNED NOT NULL DEFAULT 0` - `next_attempt_at DATETIME NULL` - `lease_token CHAR(36) NULL` - `lease_expires_at DATETIME NULL` - `last_error_code VARCHAR(100) NULL` - `last_error_message TEXT NULL` - `reconciled_at`, `created_at`, `updated_at`, `completed_at` timestamps - unique `(user_id, request_key)` - indexes `(status, next_attempt_at)`, `(lease_expires_at)`, `(updated_at)` `request_payload` preserves sanitized immutable pairs in ordinal order. It exists even after intent expansion so admission replay can verify idempotency and repair can reconstruct missing intents. #### `{$wpdb->prefix}ipz_translation_manifest_intents` - `id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY` - `public_id CHAR(36) NOT NULL UNIQUE` - `manifest_id BIGINT UNSIGNED NOT NULL` - `ordinal INT UNSIGNED NOT NULL` - `requested_post_id BIGINT UNSIGNED NOT NULL` - `target_language VARCHAR(20) NOT NULL` - `status VARCHAR(20) NOT NULL` - `retryable TINYINT(1) NOT NULL DEFAULT 1` - `generation INT UNSIGNED NOT NULL DEFAULT 1` - `generation_attempt_count INT UNSIGNED NOT NULL DEFAULT 0` - `attempt_count INT UNSIGNED NOT NULL DEFAULT 0` (lifetime) - `last_backend_event_sequence BIGINT UNSIGNED NOT NULL DEFAULT 0` - `next_attempt_at DATETIME NULL` - `claim_token CHAR(36) NULL` - `claim_expires_at DATETIME NULL` - `local_job_id BIGINT UNSIGNED NULL` - `batch_parent_job_id BIGINT UNSIGNED NULL` - `submission_id BIGINT UNSIGNED NULL` - `remote_job_id VARCHAR(191) NULL` - `source_language VARCHAR(20) NULL` - `prepared_payload LONGTEXT NULL` - `error_code VARCHAR(100) NULL` - `error_message TEXT NULL` - `created_at`, `updated_at`, `completed_at` timestamps - unique `(manifest_id, ordinal)` and `(manifest_id, requested_post_id, target_language)` - indexes `(manifest_id, status, next_attempt_at)`, `(status, claim_expires_at)`, `(local_job_id)`, `(batch_parent_job_id)` #### `{$wpdb->prefix}ipz_translation_manifest_submissions` Stores one backend request before I/O: numeric/public UUID identity, manifest ID, status (`pending`, `submitting`, `accepted`, `retry_wait`, `failed`), request fingerprint, nullable canonical secret-free request body, linked intent-generation snapshot, backend job ID, lifetime attempts, due time, lease token/expiry, sanitized error, `payload_purged_at`, and timestamps. `public_id` is the backend `submissionId`. Indexes cover `(status,next_attempt_at)`, stale leases, manifest, and remote job. Before acceptance, a submission cannot change its fingerprint or body bytes. `record_submission_accepted()` atomically records the remote job ID/status and sets the body to `NULL` plus `payload_purged_at`; it retains the UUID, fingerprint, refs/hashes, and generation snapshot as immutable evidence. Purge never occurs while the remote identity is unknown. Thus an accepted-response-loss crash retains replayable bytes, while successful identity recording ends resend need and removes duplicated source text. #### `{$wpdb->prefix}ipz_translation_manifest_attempts` Append-only audit rows record manifest ID, intent ID, generation, lifetime attempt, generation attempt, transition/event, submission public ID, stable error code, sanitized error message, bounded metadata JSON without source content/secrets, and timestamp. Retry never deletes history. Intent `attempt_count` is lifetime; `generation_attempt_count` resets only when an explicit retry increments `generation`. All four tables explicitly request `ENGINE=InnoDB`. MySQL DDL is implicitly committing, so migration never promises rollback. It acquires a named database advisory lock with a 30-second timeout, inspects information schema, and executes a resumable ordered create/add/index repair plan. Every step is idempotent; after interruption, rerun detects completed steps and repairs only missing/incompatible schema. The DB version advances only after final information-schema verification confirms all tables, columns, engines, and indexes. Activation and the release migration command perform upgrades before traffic. A request observing an unavailable schema receives `503 ipz_manifest_schema_unavailable`; it never runs an unbounded migration inline. WordPress does not enforce foreign keys in plugin tables; repository deletes and retention cleanup preserve referential integrity. ### State model Intent states: - `pending`: persisted, not claimed; - `preparing`: claimed for preparation; - `prepared`: local job and durable prepared payload exist; - `queued`: backend accepted the single or bulk job; - `processing`: backend/local reconciliation reports active processing; - `completed`, `failed`, `protected`, `skipped`: terminal. Intent expansion is lazy and idempotent: `(manifest_id, ordinal)` prevents duplicates. In one `READ COMMITTED` transaction, recomputation locks the manifest row, aggregates persisted intent states, derives `expanded_count = COUNT(intents)`, synthesizes unexpanded count as `total - expanded_count`, assigns public `preparing = unexpanded + pending + preparing + prepared`, writes the seven public state counters, derives overall status, and commits. The seven public counters sum exactly to `total` before expansion, during retry reset, and across callback races. Every transition/retry transaction invokes this recomputation before commit. Overall manifest status is derived, not independently incremented: 1. `completed` when all intents are terminal and `failed = 0`; 2. `failed` when all intents are terminal and `failed > 0` (partial successes remain visible); 3. `processing` when any intent is processing; 4. `queued` when none is processing and any is queued; 5. otherwise `preparing`. Counter updates are recomputed in one aggregate query after transitions. Callbacks and retries never increment counters blindly, preventing duplicate-delivery drift. ### Retry policy Retry classification uses stable error codes. Authorization revocation, invalid post/language, protected target, invalid structured fields, and payload limits are nonretryable. Network failures, backend 429/5xx, lease loss before submission, and transient database errors are retryable. Automatic intent retries are capped at 5 attempts with `min(30 * 2^(attempt-1), 3600)` seconds plus bounded jitter. Explicit selective retry is allowed for retryable terminal failures, increments `generation`, resets `generation_attempt_count`, and starts a new automatic-attempt window. Lifetime `attempt_count` never resets, and every transition/error remains in the append-only attempt table. Scheduling failure does not consume an intent attempt. A manifest-level transient failure sets `next_attempt_at` and releases the lease. One intent failure does not block unrelated intents. A database failure before commit leaves the prior state and lease to expire; a failure after commit leaves a durable due/claimed state. Because the external runner scans due and expired records independently, recovery never depends on persisting an error about a failed persistence attempt. Stable public error codes include `authorization_revoked`, `post_not_found`, `unsupported_language`, `target_protected`, `invalid_fields`, `payload_too_large`, `backend_rate_limited`, `backend_unavailable`, `submission_conflict`, `lease_lost`, `retry_exhausted`, and `internal_error`; implementation may add only documented allowlisted codes. ## Backend bulk-content contract `POST /v1/jobs/bulk-content` requires a submission identity and extends each item with arbitrary structured fields: ```text BulkContentRequest = { submissionId: UUID, clientJobId: string, sourceLang: LanguageCode, tone?: Tone, callbackUrl: URL, items: BulkContentItem[] } BulkContentItem = { ref: string, targetLang: LanguageCode, fields: Record } ``` `TranslationJob` adds globally unique `submission_id` plus `submission_fingerprint`. The route canonicalizes the authenticated user, site identity, plugin, non-secret request fields, refs, and structured values into the fingerprint. First use creates the job and durable `queue_payload`; replay by the same authenticated user/site/plugin with the same fingerprint returns `200`/the original job identity, regardless of current job state. A different principal receives `404`, and a fingerprint mismatch receives `409 SUBMISSION_ID_CONFLICT`. The route reads `X-IPZ-Callback-Secret` only after normal API-key authentication, validates 16–256 bytes, and stores it in `TranslationJob.callback_secret` only on first creation. The header is excluded from request fingerprinting, logs, `queue_payload`, and responses. Exact replay by the same principal returns the existing job and never changes its stored callback secret. The route never creates a second job for the UUID. The WordPress submission table stores exactly the canonical secret-free body bytes used to compute this fingerprint until remote acceptance is durably recorded. Create/replay and authenticated `GET /v1/jobs/{id}` responses include `submissionId`. Every terminal webhook includes `submission_id`, `event_sequence`, backend `job_id`, `clientJobId`, and, for bulk results, each stable `ref`; WordPress can therefore resolve the persisted submission and exact intent-generation snapshot without inference. Legacy `title`, `excerpt`, and `content` aliases remain accepted during the compatibility window. The route uses the existing `mergeStructuredFields`, canonical serializer, structured character counter, exception replacement, Gemini `translateStructured`, and field restoration pipeline for every bulk item. `BulkContentItemInput`, Bull queue data, persisted `queue_payload`, worker result, stored translation, and webhook result all carry `fields: Record`. Result keys must exactly equal validated input keys; missing, added, or duplicate keys fail that item. Field keys follow the existing 64-character pattern and 100-field maximum; values and aggregate request size use existing sanitized-character capacity limits. ACF/SEO protection metadata remains WordPress-local in each translation context and is never sent as translatable text. Supplying aliases and `fields` with a duplicate core key is rejected. This end-to-end mapping permits ACF and SEO fields to share real multi-item backend jobs instead of falling back to singles. The existing PostgreSQL `queue_payload`, queue lease, stale-lease sweep, Bull enqueue, processing lease, and heartbeat remain load-bearing. No change may create a database job without a durable queue payload capable of re-enqueue. ## Backend callback durability Every terminal backend transition, including validation/provider exhaustion and every bulk-content whole-job failure branch, changes the job from a nonterminal state and creates exactly one `WebhookOutbox` row in the same PostgreSQL transaction. A guarded state transition permits only one winning terminal state; rollback leaves neither status nor outbox committed. Completion and failure events use a stable delivery ID derived from the outbox row and keep the existing HMAC-SHA256 signature over `timestamp + "." + raw_body`. `WebhookOutbox` adds durable delivery scheduling fields: - `next_attempt_at`; - `first_attempt_at`; - `last_attempt_at`; - `dead_at`; - `response_status`; - monotonic per-job `delivery_sequence`, included in the callback as `event_sequence`. The drainer claims only due rows, recovers stale `delivering` claims, and supports completion and failure events. Every unsuccessful delivery persists attempt count, sanitized error, response status, and next attempt. Backoff is capped exponential with jitter. A stable delivery ID remains unchanged while the timestamp/signature is regenerated per attempt. After the automatic delivery window is exhausted, the row becomes `dead`, is visible through `GET /v1/admin/translation-webhook-outbox`, and supports `POST /v1/admin/translation-webhook-outbox/{id}/redrive`; both require `authenticateAdmin`, never return secrets/payload source content, and audit the administrator and action. This is not the only repair path: WordPress’s bounded reconciler polls backend terminal job state and invokes the same idempotent local callback/finalization seam. Therefore a dead callback cannot silently strand a completed translation. ## Callback-to-manifest synchronization `TranslateJobsController` keeps current HMAC, source-IP, event, delivery-ID, client-job-ID, backend-job-ID, ref, and content-type checks. After an idempotent local job or child-item transition, it calls `TranslationManifestRepository::synchronize_local_job(local_job_id)`. Synchronization derives the manifest intent state from authoritative local job/item state, applies only forward-compatible transitions, and recomputes affected manifests. Rules: - the backend terminal-state transaction emits only the winning terminal event; completion and whole-job failure cannot both be committed for one backend job; - duplicate callbacks with the same delivery ID/event sequence are successful no-ops after identity validation; - an event applies only when its local job ID, backend job ID, submission ID, and intent generation all equal the intent's current linkage and `event_sequence` is greater than `last_backend_event_sequence`; - a queued/processing current generation may move to its matching terminal result; terminal states never move backward, and polling progress never overwrites a terminal state; - bulk completion applies each ref's own completed/failed result exactly once; a whole-job failure updates only still-nonterminal current-generation intents linked to that backend job; - callbacks for old retry generations or unrelated identities are acknowledged only after being recorded as stale no-ops; identity mismatches are rejected; - polling reconciliation reads the authoritative backend job status and uses the same generation-fenced synchronization method; - webhook delivery is recorded only after local finalization and manifest synchronization commit. ## Bounded reconciliation The existing `BulkContentReconciler` remains the independent backend-to-WordPress repair path and calls manifest synchronization after every local transition. The external runner invokes it after manifest processing; Action Scheduler also schedules it every five minutes. One run claims at most 20 accepted submissions/local jobs older than 60 seconds under existing local claim fencing, uses the authenticated backend `GET /v1/jobs/{remote_job_id}` endpoint, and processes for at most 20 seconds. Entries missing a remote ID are not polled: their immutable submission UUID is resent through create-or-return-existing first. Reconciliation covers active/failed manifests for the full 30-day retention window, reuses current site/API identity, applies the same generation fencing as callbacks, and advances a durable cursor so later rows cannot starve. ## Generate All UX `content-translate.js` stops chunking browser requests. Generate All constructs the intended pairs once, creates one stable `request_id`, and sends one admission request. The browser persists a bounded list of at most 20 active `{batch_id, request_id, created_at}` records using a site/user/version-namespaced guarded localStorage key following `DataStore` identity rules. This supports concurrent tabs/manifests; the newest active manifest is the default view. Server status is always authoritative. Behavior: - the modal immediately shows `preparing` after `202`; - it polls status with bounded backoff and pauses when the document is hidden; - reload/navigation restores all active manifest identities, selects the newest, and resumes one deduplicated poller per visible manifest; an ambiguous admission network failure retries the exact same canonical body and `request_id`; - the Generate All button becomes “View Generate All progress” while active; - progress displays all public counters and a determinate total; - terminal partial failure shows completed work, paginated failures, and “Retry failed” for eligible intents; - selective retry reuses the same manifest and clears terminal browser state only when no active/retryable work remains; - network polling errors do not convert server work into failed/unknown work; - `404` after identity change or retention expiry clears the stale local key with an explanatory notice. No inline CSS is introduced. Existing modal and component patterns, accessibility labels, keyboard behavior, and reduced-motion rules remain in force. ## Retention and observability - Completed manifests and intents are retained for 30 days. - A daily bounded cleanup removes terminal manifests older than retention and then their intents. - Active or failed manifests are not deleted. A terminal success is cleanup-eligible only after `reconciled_at` is set by a transaction proving every linked current-generation local job/item terminal. Cleanup uses a persisted cursor, deletes at most 100 manifests per run, and cannot repeatedly starve later rows. - Logs carry manifest public ID, intent public ID, local job ID, backend job ID, claim token suffix, attempt, transition, duration, and stable error code; they never contain API keys, callback secrets, authorization headers, or source content. - The guarded option `ipz_manifest_worker_health` stores global `last_external_run_at`, `last_scheduler_run_at`, duration, exit class, and recovery source. Status combines it with repository-derived next due time, oldest due age, stale count, and sanitized last-error metadata; it is diagnostic only, never queue authority. - Backend metrics/logs expose due, delivering, delivered, and dead outbox counts. ## Security - Admission and retry require nonce-authenticated WordPress users. - Admission checks `edit_post` for every requested post using batched loading; workers recheck current authorization before reading source fields. - Status/retry use owner-or-plugin-admin checks and return indistinguishable `404` for foreign manifests. - Canonical source and translation-group authorization remain in dispatcher preparation. - Request IDs, public manifest IDs, intent IDs, delivery IDs, claim tokens, and lease tokens use cryptographically random UUIDs. - SQL uses `$wpdb->prepare`; all state values are allowlisted. - Error codes are allowlisted lowercase snake-case strings of at most 64 bytes. Persisted and returned error messages are stripped of control characters and bounded to 500 UTF-8 bytes. - Backend callback HMAC comparison remains constant-time and timestamp bounded. - No API or callback secret is persisted in manifest, intent, submission-body, attempt, or queue payloads. The callback secret travels only in the authenticated TLS request header and remains in the existing backend job secret column for webhook signing. ## Migration and compatibility Delivery is expand–migrate–switch–contract: 1. Add tables, repository, scheduler dependency, processor, and tests while the old dispatcher wrapper remains. 2. Add backend structured-field compatibility and durable failure outbox fields. 3. Switch REST admission and Generate All behind the existing `ipz_unreleased` flag until all remote gates pass and the external runner health check is green. Scheduler, CLI drain, callback synchronization, reconciliation, and cleanup hooks remain active for admitted work even when the UI/admission flag is later disabled. 4. Enable the already-shared flag according to the release procedure; do not add a feature-specific flag. 5. Remove obsolete browser chunk/progress code and stale direct-route assertions only after replacement tests prove the manifest path. Rollback can disable `ipz_unreleased`; existing jobs, manifests, and backend queue rows remain inspectable and recoverable. Schema is not destructively contracted in this release. ## Deterministic proof All build, unit, integration, and browser proof runs remotely on debian1/2/3 and the canonical k3s/e2e-remote lane. Nothing is built, tested, profiled, or served from workstation WordPress. dev1 receives no smoke test. The deterministic harness injects `Clock`, UUID source, backend client, and scheduler collaborators into repository/processor code. Production uses real collaborators; tests use a frozen/advanced clock and deterministic UUIDs. A test-only processor hook, constructed only by the test bootstrap (never a public REST switch), throws at named boundaries: after claim, after local-job commit, after submission commit, after backend acceptance, and during callback application. The k3s stack provides a capture backend, a controllable Action Scheduler/WP-CLI runner, and database assertion helpers. Browser tests pause the runner, assert admission state/request count, advance workers explicitly, inject duplicate/reordered callbacks, then resume. Jitter is injected and fixed to zero in deterministic tests. Required evidence: 1. **Admission:** one authenticated request with 1,000 unique intents returns `202` within 3 seconds in the k3s lane, creates exactly one manifest and no intent rows during admission, and performs zero dispatcher-preparation/backend HTTP calls during the request. 2. **Idempotency:** exact request replay returns the same ID; same request ID with different content returns 409; one manifest and exactly 1,000 unique intents exist after expansion. 3. **Bounded drain and liveness:** instrumented processor calls never exceed 100 expansions, 8 preparations, 4 submissions, or the begin-new-work 25-second budget. With Action Scheduler loopback disabled, the external WP-CLI runner drains the manifest and health timestamps remain within SLO. 4. **Crash recovery:** failures injected after intent claim, after local-job creation, after submission-envelope commit, after backend acceptance, and during callback application converge after clock-driven lease expiry with one local intent, one immutable submission UUID, one backend job, and one current-generation ref. 5. **Real batching:** captured backend requests contain at least two items and include core plus ACF/SEO structured fields; no eligible structured-field intent is forced to a single solely because it has extra fields. 6. **Queue durability:** backend enqueue failure followed by dispatcher sweep processes the persisted `queue_payload` once. 7. **Callback durability:** every completion and whole-job failure branch atomically creates one outbox row, including transaction rollback assertions; transient delivery failures honor persisted due times; duplicate, reordered, old-generation, and contradictory injected callbacks do not drift counters. 8. **Repair:** simulated exhausted callback delivery is recovered by backend polling/local reconciliation and reaches the same terminal manifest state. 9. **Aggregate invariants:** after every tested transition, public counters sum to total and match intent-row aggregation. 10. **Browser:** Chromium and Firefox each assert the exact single admission body, retry the same request ID after an ambiguous response, display preparing/queued/processing/terminal counters, restore multiple active manifests after reload/navigation, and selectively retry failures through the remote k3s Playwright lane with controlled worker ticks. 11. **Distribution:** remote production build contains Action Scheduler and compiled admin assets; plugin packaging and applicable repository gates pass remotely. ## Architecture decisions - Keep repository, processor, scheduler, dispatcher, and REST responsibilities separate. Each boundary hides a different replaceable mechanism (SQL/fencing, orchestration, runner, translation semantics, HTTP authorization) and passes the deletion/seam tests. - Do not introduce repository interfaces or adapter abstractions with only one implementation; concrete classes expose narrow methods. - Collapse manifest callback handling into repository synchronization rather than a standalone callback adapter; a separate module would be organizational only. - Keep aggregate counters denormalized for inexpensive polling, but recompute them from intent rows to preserve correctness under replay. - Preserve the backend durable queue sweep and completion outbox; extend them rather than creating parallel queue/callback systems. - Keep browser persistence deliberately shallow: bounded active manifest identities only, no local progress state. - Add a durable submission table because backend acceptance is an independent crash boundary; local parent rows alone cannot close it. - Require an external WP-CLI runner for liveness; Action Scheduler, loopback, status reads, and WP-Cron remain accelerators/recovery signals, not the sole guarantee. - Add an append-only attempt table because selective retry must retain auditable history across generations.