# White-Label API & Custom Domains — Implementation Plan **Spec:** docs/specs/2026-05-30-white-label-api.md · **Slug:** white-label-api · **Wave:** 10 **Depends on:** foundation-auth-rbac, system-communications-notifications, tenant-portals ## Goal Deliver the three enterprise-tier headless capabilities of Zync: (1) custom domains that map a tenant's own DNS name (e.g. `portal.acme.com`) to their Zync customer portal via Cloudflare custom hostnames; (2) a full outbound webhook gateway that emits HMAC-signed, retryable events on every tenant state change; and (3) tenant-controlled API keys (the authoritative `tenant_api_keys` schema also consumed by the Tenant Public API). This is what makes Zync integration-ready and resellable for larger enterprise tenants. ## Architecture - **Custom domains** live in a new `tenant_domains` table. The settings UI saves a pending domain; the new `domain-verify` cron (every 15 min) polls Cloudflare for SSL+hostname activation and only then provisions the CF custom hostname via `POST /zones/{zone}/custom_hostnames`. The `zync-api` Worker (which already serves the `/portal/{slug}` content owned by **tenant-portals**) gains a host-resolution middleware: an incoming request whose `Host` header matches a `verified`/`active` `tenant_domains.domain` resolves to that tenant's slug and serves the existing portal with tenant branding (logo + brand color CSS variables — already produced by tenant-portals). - **Webhook gateway** owns `webhook_endpoints` and `webhook_deliveries`. The shared `emitWebhookEvent()` producer (exported here, called by every module's event sites) finds active subscribed endpoints and enqueues the existing `webhook.deliver` queue (declared upstream in foundation-monorepo / system-communications-notifications). The queue consumer decrypts the endpoint secret with `decryptCredential` (the same `INTEGRATION_ENCRYPTION_KEY` AES-256-GCM path used for `adapter_credentials`), computes the `X-Zync-Signature` HMAC over `${timestamp}.${body}`, POSTs, records the result in `webhook_deliveries`, and schedules exponential-backoff retries. On terminal failure it raises an in-app + email notification via `deliverNotification` (system-communications-notifications). - **API keys** own `tenant_api_keys` (authoritative schema; the Tenant Public API spec 39 consumes it, never redefines it). Keys are `zyk_live_{random32}`, shown once, stored as SHA-256 hash via `hashToken`. The internal management routes (`/api/api-keys`) live in `zync-api`; the inbound auth at `api.zync.is/v1/` middleware is owned by spec 39 and consumes this table plus the `hasScope`/`ApiScope` exports. - **Upstream consumed:** tables `tenants` (FK + tier), `users` (FK), `customer_portal_users`/portal content (tenant-portals); exports `requirePermission`, `requireTier`, `TenantTier`, `meetsMinimumTier`, `authMiddleware`, `tenantQuery`, `systemQuery`, `encryptCredential`, `decryptCredential`, `generateOpaqueToken`, `hashToken`, `timingSafeEqual`, `deliverNotification`, `createNotification`, `QUEUE`, `KV`, `RATE_LIMITER_WEBHOOK`, `ApiError`, `buildPaginated`, `clampLimit`, and the `webhook.deliver` queue. ## Tech Stack - **App/package:** `apps/zync-api` (Hono routes, cron handler, queue consumer, host-resolution middleware); `packages/db` (Drizzle schema + queries: `packages/db/src/schema/white-label.ts`, `packages/db/src/queries/{domains,webhooks,api-keys}.ts`); `packages/types` (shared `WebhookEvent`, `WebhookEventType`, `ApiScope`, `WhiteLabelTables`); `packages/auth` (api-key hashing/scope helpers reused by spec 39); `apps/zync-app` settings UI (`/settings/white-label`, `/settings/integrations/webhooks`, `/settings/api-keys`). - **Libraries:** Drizzle ORM + `@neondatabase/serverless` via Hyperdrive; Hono; Zod (request validation); React + react-query + `@zync/ui` for settings UI. - **Cloudflare bindings:** `DB` (Hyperdrive→Neon), `QUEUE` (`webhook.deliver`), `KV` (delivery idempotency / domain-verify cursor), `RATE_LIMITER_WEBHOOK`. Secrets: `INTEGRATION_ENCRYPTION_KEY` (existing), `CF_CUSTOM_HOSTNAME_API_TOKEN` (new — manual CF dashboard token, `ssl_certs:write` / `Zone:Custom Hostnames:Edit` on the zync.is zone), `CF_ZONE_ID` (zone id for custom-hostname API calls). New cron trigger `domain-verify` (`*/15 * * * *`). ## Wave Plan | Sub-wave | Tasks | Files touched | Parallelizable? | |----------|-------|---------------|-----------------| | A — schema & types | 1, 2 | `packages/db/src/schema/white-label.ts`, `packages/types/src/webhooks.ts`, `packages/types/src/api-scopes.ts` | Tasks 1 & 2 parallel | | B — crypto/token helpers | 3 | `packages/auth/src/api-keys.ts`, `packages/auth/src/webhook-sign.ts` | after A | | C — domain backend | 4, 5, 6 | `packages/db/src/queries/domains.ts`, `apps/zync-api/src/routes/domains.ts`, `apps/zync-api/src/cron/domain-verify.ts`, `apps/zync-api/src/middleware/host-resolve.ts` | 4 then 5+6 parallel | | D — webhook backend | 7, 8, 9, 10 | `packages/db/src/queries/webhooks.ts`, `apps/zync-api/src/webhooks/emit.ts`, `apps/zync-api/src/queue/webhook-deliver.ts`, `apps/zync-api/src/routes/webhooks.ts` | 7 then 8+9, then 10 | | E — API keys backend | 11, 12 | `packages/db/src/queries/api-keys.ts`, `apps/zync-api/src/routes/api-keys.ts` | after B | | F — settings UI | 13, 14, 15 | `apps/zync-app/src/modules/settings/white-label/*` | parallel after their backends | | G — wiring & event sites | 16, 17 | `apps/zync-api/src/index.ts`, `wrangler.toml`, module event call-sites | last | ## Tasks ### Task 1: Database schema — white-label tables (Drizzle) **Blocks:** 3, 4, 7, 11 · **Blocked by:** — **Files:** - Create: `packages/db/src/schema/white-label.ts` - Modify: `packages/db/src/schema/index.ts` (export new tables) - Create: `packages/db/migrations/00XX_white_label.sql` (generated via `drizzle-kit generate`) **Steps:** - [ ] Define the four tables in Drizzle matching the canonical Postgres DDL below. - [ ] Add indexes: unique on `tenant_domains.domain`; `webhook_endpoints(tenant_id, is_active)`; `webhook_deliveries(tenant_id, created_at desc)`, `webhook_deliveries(endpoint_id)`, `webhook_deliveries(status, next_retry_at)`; `tenant_api_keys(key_hash)` unique, `tenant_api_keys(tenant_id)`. - [ ] Export `tenantDomains`, `webhookEndpoints`, `webhookDeliveries`, `tenantApiKeys` from `packages/db/src/schema/index.ts`. - [ ] Generate the migration and commit the SQL. **Schema / Interfaces:** ```sql CREATE TABLE tenant_domains ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, domain TEXT NOT NULL UNIQUE, -- e.g. 'portal.acme.com' cloudflare_hostname_id TEXT, -- CF custom hostname id; null until provisioned status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','verified','active','error')), error_message TEXT, verified_at TIMESTAMPTZ, -- set when CF reports ssl+hostname active deleted_at TIMESTAMPTZ, -- soft-delete (>48h unverified, or removal) created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE webhook_endpoints ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, url TEXT NOT NULL, -- HTTPS required (validated in app layer) secret BYTEA NOT NULL, -- HMAC key, AES-256-GCM via INTEGRATION_ENCRYPTION_KEY events TEXT[] NOT NULL, -- subscribed WebhookEventType values is_active BOOLEAN NOT NULL DEFAULT true, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE webhook_deliveries ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), -- == X-Zync-Delivery header tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, endpoint_id UUID NOT NULL REFERENCES webhook_endpoints(id) ON DELETE CASCADE, event_type TEXT NOT NULL, payload JSONB NOT NULL, status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','delivered','failed','test')), response_status INTEGER, response_body TEXT, attempt INTEGER NOT NULL DEFAULT 1, delivered_at TIMESTAMPTZ, next_retry_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE tenant_api_keys ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, name TEXT NOT NULL, -- e.g. 'Zapier integration' key_prefix TEXT NOT NULL, -- first 8 chars shown in UI, e.g. 'zyk_live' key_hash TEXT NOT NULL UNIQUE, -- SHA-256 of full key scopes TEXT[] NOT NULL, -- e.g. {'customers:read','invoices:write'} last_used_at TIMESTAMPTZ, expires_at TIMESTAMPTZ, -- null = never expires created_by UUID NOT NULL REFERENCES users(id), created_at TIMESTAMPTZ NOT NULL DEFAULT now(), revoked_at TIMESTAMPTZ ); ``` **Acceptance:** - [ ] `drizzle-kit generate` produces a migration whose `up` SQL is byte-equivalent to the DDL above (UUID PKs, UUID→UUID FKs, BOOLEAN, JSONB, TEXT[] arrays, TEXT CHECK enums). - [ ] All four tables and their indexes exist after applying the migration to a Neon branch. ### Task 2: Shared types — webhook event catalog & API scopes **Blocks:** 7, 8, 11 · **Blocked by:** — **Files:** - Create: `packages/types/src/webhooks.ts` - Create: `packages/types/src/api-scopes.ts` - Modify: `packages/types/src/index.ts` (re-export) **Steps:** - [ ] Define `WebhookEventType` as a string-literal union covering the full event catalog (see below) and `WEBHOOK_EVENT_CATALOG` (grouped metadata for the multi-select UI). - [ ] Define `WebhookEvent` envelope type and `WebhookDeliveryStatus`/`DomainStatus` unions. - [ ] Define `ApiScope` union exactly as spec 39 declares, plus the enterprise-only extension scopes (`leads:write`, `campaigns:write`) as `EnterpriseApiScope`; export `ALL_API_SCOPES` (standard) and `ENTERPRISE_API_SCOPES`. - [ ] Re-export all from `packages/types/src/index.ts`. **Schema / Interfaces:** ```ts export type WebhookEventType = | 'timer.started' | 'timer.stopped' | 'timer.auto_paused' | 'lead.created' | 'lead.stage_updated' | 'proposal.viewed' | 'proposal.accepted' | 'project.created' | 'project.status_changed' | 'task.created' | 'task.assigned' | 'task.completed' | 'ticket.created' | 'ticket.replied' | 'ticket.resolved' | 'invoice.proforma_approved' | 'invoice.issued' | 'invoice.paid' | 'invoice.overdue' | 'retainer.depleted' | 'expense.submitted' | 'expense.approved' | 'payout.generated' | 'payment.completed' | 'payment.failed' | 'user.invited' | 'role.updated' | 'tenant.provisioned' | 'calendar.booking_created'; export interface WebhookEvent> { event: WebhookEventType; // e.g. 'invoice.paid' tenantId: string; // UUID data: T; timestamp: number; // unix seconds, injected at delivery time } export type WebhookDeliveryStatus = 'pending' | 'delivered' | 'failed' | 'test'; export type DomainStatus = 'pending' | 'verified' | 'active' | 'error'; export interface WebhookCatalogEntry { category: string; event: WebhookEventType; } export const WEBHOOK_EVENT_CATALOG: WebhookCatalogEntry[]; // every WebhookEventType with its category label export type ApiScope = | 'customers:read' | 'customers:write' | 'invoices:read' | 'invoices:write' | 'tasks:read' | 'tasks:write' | 'events:read'; export type EnterpriseApiScope = 'leads:write' | 'campaigns:write'; export const ALL_API_SCOPES: readonly ApiScope[]; export const ENTERPRISE_API_SCOPES: readonly EnterpriseApiScope[]; ``` **Acceptance:** - [ ] `WEBHOOK_EVENT_CATALOG` contains exactly the 29 events from the spec catalog, no more, no fewer. - [ ] `ApiScope` is identical to the spec-39 declaration (no extra members in the standard union). ### Task 3: Auth helpers — API key generation/hash + webhook HMAC signing **Blocks:** 11, 9 · **Blocked by:** 1, 2 **Files:** - Create: `packages/auth/src/api-keys.ts` - Create: `packages/auth/src/webhook-sign.ts` - Modify: `packages/auth/src/index.ts` (export) **Steps:** - [ ] `generateApiKey()` → builds `zyk_live_{random32}` using `generateOpaqueToken(32)` over the alphanumeric alphabet; returns `{ fullKey, prefix, hash }` where `prefix` = first 8 chars (`'zyk_live'`) and `hash = await hashToken(fullKey)` (SHA-256 hex). - [ ] `hasScope(keyScopes: string[], required: ApiScope): boolean` — returns true if `required` ∈ scopes, OR `required` is a `:read` scope and the matching `:write` scope ∈ scopes (write implies read). - [ ] `signWebhook(secret: string, timestamp: number, body: string): Promise` → `'sha256=' + HMAC_SHA256(secret, \`${timestamp}.${body}\`)` (hex), using `crypto.subtle` HMAC. - [ ] `verifyWebhookSignature(secret, timestamp, body, signature)` → recompute and compare via `timingSafeEqual` (reference impl for receivers / used by the test endpoint self-check). - [ ] Export all four from `packages/auth/src/index.ts`. **Schema / Interfaces:** ```ts export function generateApiKey(): Promise<{ fullKey: string; prefix: string; hash: string }>; export function hasScope(keyScopes: string[], required: ApiScope): boolean; export function signWebhook(secret: string, timestamp: number, body: string): Promise; export function verifyWebhookSignature( secret: string, timestamp: number, body: string, signature: string ): Promise; ``` **Acceptance:** - [ ] `generateApiKey()` returns a 42-char `fullKey` matching `/^zyk_live_[A-Za-z0-9]{32}$/`; `prefix === 'zyk_live'`; `hash` is 64 hex chars and re-derivable from `fullKey`. - [ ] `signWebhook` output is stable for fixed inputs and verifies via `verifyWebhookSignature`; comparison path uses `timingSafeEqual` (no `===` on the signature). ### Task 4: DB queries — domains **Blocks:** 5, 6 · **Blocked by:** 1 **Files:** - Create: `packages/db/src/queries/domains.ts` **Steps:** - [ ] `listTenantDomains(db, tenantId)` → all non-soft-deleted `tenant_domains` for tenant, scoped via `tenantQuery`. - [ ] `createTenantDomain(db, tenantId, domain)` → insert with `status='pending'`; before insert run the uniqueness guard `SELECT 1 FROM tenant_domains WHERE domain = $1 AND status IN ('verified','active') AND tenant_id <> $2 AND deleted_at IS NULL`; if found throw `DomainAlreadyVerifiedError` (maps to `409 domain_already_verified`). - [ ] `getDomainById(db, tenantId, id)` and `getDomainByHost(db, host)` (the latter uses `systemQuery` — cross-tenant lookup by exact `domain`, only `status IN ('verified','active') AND deleted_at IS NULL`). - [ ] `listPendingDomains(db)` (systemQuery; `status='pending' AND deleted_at IS NULL`) for the cron. - [ ] `setDomainVerified(db, id, hostnameId)` → `status='active', verified_at=now(), cloudflare_hostname_id=$hostnameId`. - [ ] `setDomainError(db, id, message)`; `softDeleteDomain(db, id)` → `deleted_at=now()`. **Schema / Interfaces:** ```ts export class DomainAlreadyVerifiedError extends Error {} export function listTenantDomains(db: Db, tenantId: string): Promise; export function createTenantDomain(db: Db, tenantId: string, domain: string): Promise; export function getDomainByHost(db: Db, host: string): Promise; export function listPendingDomains(db: Db): Promise; export function setDomainVerified(db: Db, id: string, hostnameId: string): Promise; export function setDomainError(db: Db, id: string, message: string): Promise; export function softDeleteDomain(db: Db, id: string): Promise; ``` **Acceptance:** - [ ] Creating a domain already `verified`/`active` under a different tenant throws `DomainAlreadyVerifiedError`. - [ ] `getDomainByHost` never returns soft-deleted or `pending`/`error` rows. ### Task 5: CF custom-hostname client + `domain-verify` cron **Blocks:** 16 · **Blocked by:** 4 **Files:** - Create: `apps/zync-api/src/lib/cf-custom-hostnames.ts` - Create: `apps/zync-api/src/cron/domain-verify.ts` **Steps:** - [ ] CF client functions over `https://api.cloudflare.com/client/v4/zones/{CF_ZONE_ID}/custom_hostnames`, authorized with `Authorization: Bearer ${env.CF_CUSTOM_HOSTNAME_API_TOKEN}`: - `createCustomHostname(env, domain)` → `POST` `{ hostname: domain, ssl: { method: 'http', type: 'dv' } }`; returns CF hostname id. - `getCustomHostname(env, hostnameId)` → `GET .../{hostnameId}`; returns `{ status, ssl: { status } }`. - `deleteCustomHostname(env, hostnameId)` → `DELETE .../{hostnameId}`. - [ ] Cron handler (scheduled `*/15 * * * *`): for each `listPendingDomains(db)`: 1. If no `cloudflare_hostname_id` yet: this is the first poll — query DNS resolution of the CNAME (resolve `domain` via DNS-over-HTTPS `https://cloudflare-dns.com/dns-query?name={domain}&type=CNAME`); if it resolves to `portal.zync.is`, call `createCustomHostname`, store the returned id (still `pending`). 2. If `cloudflare_hostname_id` set: `getCustomHostname`; transition to `active` + `verified_at` via `setDomainVerified` only when **both** `status === 'active'` AND `ssl.status === 'active'`. 3. If `created_at` older than 48h and still `pending`: `softDeleteDomain` (skip/clean any half-created hostname) and continue. 4. On CF API error: `setDomainError(db, id, message)`. - [ ] Use `CF_CUSTOM_HOSTNAME_API_TOKEN`, `CF_ZONE_ID` from env; never log the token. **Acceptance:** - [ ] A domain transitions to `active` only after CF reports both hostname `active` and SSL `active`. - [ ] A domain stuck `pending` for >48h is soft-deleted on the next cron run. - [ ] CF custom hostname is never created before DNS/verification (no pre-provisioning). ### Task 6: Host-resolution middleware (custom domain → tenant portal) **Blocks:** 16 · **Blocked by:** 4 **Files:** - Create: `apps/zync-api/src/middleware/host-resolve.ts` **Steps:** - [ ] Middleware reads the request `Host` header; if it is not a known Zync host (`zync.is`, `app.zync.is`, `api.zync.is`, `portal.zync.is`, `*.zync.is`), treat it as a candidate custom domain. - [ ] `getDomainByHost(db, host)`; if found, set `c.set('portalTenantId', row.tenant_id)` and `c.set('isCustomDomain', true)` and route the request to the existing tenant-portals portal handler (serve `/portal/{slug}` content for that tenant with its branding CSS variables — branding is produced by tenant-portals). - [ ] If no match for a non-Zync host: return `404` (unknown domain). - [ ] Cache `host → tenantId` lookups in `KV` with a short TTL (e.g. 300s) keyed `domain:{host}` to avoid a DB hit per request; invalidate on domain removal (Task 4 `softDeleteDomain` callers delete the KV key). **Acceptance:** - [ ] A request with `Host: portal.acme.com` (active domain) serves Acme's portal with Acme branding, no slug in the URL. - [ ] A request with an unknown non-Zync host returns 404. ### Task 7: DB queries — webhook endpoints & deliveries **Blocks:** 8, 9, 10 · **Blocked by:** 1, 2, 3 **Files:** - Create: `packages/db/src/queries/webhooks.ts` **Steps:** - [ ] `listEndpoints(db, tenantId)`; `getEndpoint(db, tenantId, id)`; (all `tenantQuery`-scoped). - [ ] `createEndpoint(db, tenantId, { url, events, secret })` → encrypt the raw HMAC secret with `encryptCredential(env, secret)` (AES-256-GCM via `INTEGRATION_ENCRYPTION_KEY`), store the ciphertext in `secret BYTEA`; insert `is_active=true`. - [ ] `updateEndpoint(db, tenantId, id, patch)` (url/events/is_active; `updated_at=now()`); `deleteEndpoint(db, tenantId, id)`. - [ ] `listActiveEndpointsForEvent(db, tenantId, eventType)` → `is_active = true AND $eventType = ANY(events)` (systemQuery scoped to the given tenantId — called by the emitter). - [ ] `insertDelivery(db, { id, tenantId, endpointId, eventType, payload, status })` (id is pre-generated so it can be used as `X-Zync-Delivery`). - [ ] `markDeliverySucceeded(db, id, responseStatus, responseBody)` → `status='delivered', delivered_at=now()`, clears `next_retry_at`. - [ ] `scheduleDeliveryRetry(db, id, attempt, nextRetryAt, responseStatus, responseBody)`; `markDeliveryFailed(db, id, responseStatus, responseBody)`. - [ ] `listDeliveries(db, tenantId, filters)` (event_type, status, date range; cursor paginated via `clampLimit`/`buildPaginated`); `getDelivery(db, tenantId, id)`. - [ ] `getEndpointSecret(db, env, endpointId)` → load + `decryptCredential(env, row.secret)` returning the raw HMAC key (used by the consumer; never returned to UI). **Schema / Interfaces:** ```ts export function createEndpoint( db: Db, env: Env, tenantId: string, input: { url: string; events: WebhookEventType[]; secret: string } ): Promise; export function listActiveEndpointsForEvent( db: Db, tenantId: string, eventType: WebhookEventType ): Promise; export function insertDelivery(db: Db, row: { id: string; tenantId: string; endpointId: string; eventType: WebhookEventType; payload: unknown; status: WebhookDeliveryStatus; }): Promise; export function getEndpointSecret(db: Db, env: Env, endpointId: string): Promise; ``` **Acceptance:** - [ ] The plaintext HMAC secret is never persisted — only `encryptCredential` ciphertext lands in `webhook_endpoints.secret`. - [ ] `listActiveEndpointsForEvent` matches only active endpoints subscribed to the exact event type. ### Task 8: Webhook event emitter (`emitWebhookEvent`) **Blocks:** 17 · **Blocked by:** 7 **Files:** - Create: `apps/zync-api/src/webhooks/emit.ts` - Modify: `packages/types/src/index.ts` (export `EmitWebhookEvent` signature if shared) **Steps:** - [ ] `emitWebhookEvent(env, tenantId, eventType, data)`: 1. `listActiveEndpointsForEvent(db, tenantId, eventType)`. 2. For each endpoint: pre-generate a delivery id (`crypto.randomUUID()`), `insertDelivery(... status:'pending')`, then enqueue the existing `webhook.deliver` queue with `{ deliveryId, endpointId, tenantId, eventType, payload: data }`. 3. Fire-and-forget friendly (return without awaiting HTTP) — emitting must not block the originating event handler (e.g. invoice.paid). - [ ] No-op cleanly (no throw) when the tenant has zero matching endpoints. **Schema / Interfaces:** ```ts export async function emitWebhookEvent( env: Env, tenantId: string, eventType: WebhookEventType, data: Record ): Promise; ``` **Acceptance:** - [ ] Emitting an event with 2 matching endpoints inserts 2 `pending` deliveries and enqueues 2 jobs; each delivery has a distinct UUID id. - [ ] Emitting with no matching endpoints performs zero enqueues and does not throw. ### Task 9: `webhook.deliver` queue consumer **Blocks:** 16 · **Blocked by:** 3, 7 **Files:** - Create: `apps/zync-api/src/queue/webhook-deliver.ts` **Steps:** - [ ] Consumer handles each `{ deliveryId, endpointId, tenantId, eventType, payload }` message: 1. Load endpoint; if endpoint is gone or `is_active=false` → mark delivery `failed` and ack. 2. `timestamp = Math.floor(Date.now()/1000)` (injected at delivery time, not enqueue time). 3. Build body JSON `{ event: eventType, tenantId, data: payload, timestamp }`. 4. `secret = await getEndpointSecret(db, env, endpointId)`; `signature = await signWebhook(secret, timestamp, body)`. 5. `fetch(endpoint.url, { method:'POST', headers })` with headers: `Content-Type: application/json`, `X-Zync-Signature: {signature}`, `X-Zync-Timestamp: {timestamp}`, `X-Zync-Event: {eventType}`, `X-Zync-Delivery: {deliveryId}`. Use a fetch timeout (~10s). 6. On 2xx: `markDeliverySucceeded(...)`. 7. On non-2xx / timeout / network error: if `attempt < 5`, `scheduleDeliveryRetry` with backoff `[1m,5m,30m,2h,24h][attempt-1]` and re-enqueue with `{ ...msg, delaySeconds }` (and incremented attempt); else `markDeliveryFailed` and emit an in-app + email notification to tenant admins via `deliverNotification` (type `webhook_delivery_failed`). - [ ] Retries reuse the **same** `deliveryId` (idempotent). A "redeliver" (Task 10) supplies a **new** id. - [ ] Never log the decrypted secret or full signature material. **Schema / Interfaces:** ```ts interface WebhookDeliverMessage { deliveryId: string; endpointId: string; tenantId: string; eventType: WebhookEventType; payload: Record; attempt?: number; } export async function handleWebhookDeliver(batch: MessageBatch, env: Env): Promise; ``` **Acceptance:** - [ ] A 200 response marks delivery `delivered` with `delivered_at` set and `response_status=200`. - [ ] 5 consecutive failures end in `status='failed'` and a `webhook_delivery_failed` notification to tenant admins; backoff intervals follow `1m,5m,30m,2h,24h`. - [ ] `X-Zync-Timestamp` is included verbatim in the HMAC and reflects delivery time. ### Task 10: Webhook routes (endpoints + deliveries + test/redeliver) **Blocks:** 14 · **Blocked by:** 7, 8 **Files:** - Create: `apps/zync-api/src/routes/webhooks.ts` - Modify: `apps/zync-api/src/index.ts` (mount router) **Steps:** - [ ] All routes use `authMiddleware`. Read routes require `requirePermission('settings:read')`; mutating routes require `requirePermission('settings:write')`. Gate endpoint creation/usage behind `requireTier(TenantTier.WHITE_LABEL)` (full webhook set is White-Label per the entitlement matrix). - [ ] Zod-validate all bodies (HTTPS URL required; events ⊆ `WebhookEventType`). - [ ] `GET /api/webhooks/endpoints` → `listEndpoints` (secret never returned; expose `is_active`, `events`, `url`, `created_at`, last delivery summary). - [ ] `POST /api/webhooks/endpoints` → auto-generate secret (`generateOpaqueToken(32)`), `createEndpoint` (stores encrypted), return the **raw secret once** in the response body (shown-once UX). - [ ] `PATCH /api/webhooks/endpoints/:id` → `updateEndpoint` (url/events/is_active). - [ ] `DELETE /api/webhooks/endpoints/:id` → `deleteEndpoint`. - [ ] `POST /api/webhooks/endpoints/:id/test` → insert a `test`-status delivery + enqueue `webhook.deliver` with a synthetic `{ event:'tenant.provisioned'-style test payload }`; returns the delivery id. - [ ] `GET /api/webhooks/deliveries` → `listDeliveries` with filters (event, status, from/to date), cursor-paginated. - [ ] `POST /api/webhooks/deliveries/:id/redeliver` → only for `failed` deliveries; create a **new** delivery row (new id) referencing the same endpoint+payload and enqueue it. **Schema / Interfaces:** ``` GET /api/webhooks/endpoints POST /api/webhooks/endpoints PATCH /api/webhooks/endpoints/:id DELETE /api/webhooks/endpoints/:id POST /api/webhooks/endpoints/:id/test GET /api/webhooks/deliveries POST /api/webhooks/deliveries/:id/redeliver ``` **Acceptance:** - [ ] `POST /api/webhooks/endpoints` returns the secret exactly once; a subsequent `GET` never includes it. - [ ] Non-White-Label tenants are rejected at endpoint creation with a 402 upgrade response. - [ ] Redeliver of a failed delivery creates a new delivery id (distinct from the original). ### Task 11: DB queries — API keys **Blocks:** 12 · **Blocked by:** 1, 2, 3 **Files:** - Create: `packages/db/src/queries/api-keys.ts` **Steps:** - [ ] `listApiKeys(db, tenantId)` → returns prefix, name, scopes, `last_used_at`, `expires_at`, `created_at`, `revoked_at` (never the hash or full key); `tenantQuery`-scoped. - [ ] `countActiveApiKeys(db, tenantId)` → `revoked_at IS NULL` count (for tier cap enforcement: business=5, enterprise/white_label=20). - [ ] `createApiKey(db, tenantId, { name, scopes, expiresAt, createdBy, prefix, hash })` → insert; returns the row (full key returned by the route layer, not stored). - [ ] `revokeApiKey(db, tenantId, id)` → `revoked_at=now()`. - [ ] `findApiKeyByHash(db, hash)` → `systemQuery`; `key_hash=$hash AND revoked_at IS NULL` (consumed by spec-39 `api.zync.is/v1` auth middleware). - [ ] `touchApiKeyLastUsed(db, id)` → `last_used_at=now()` (fire-and-forget helper for spec 39). **Schema / Interfaces:** ```ts export function listApiKeys(db: Db, tenantId: string): Promise; export function countActiveApiKeys(db: Db, tenantId: string): Promise; export function createApiKey(db: Db, tenantId: string, input: { name: string; scopes: string[]; expiresAt: Date | null; createdBy: string; prefix: string; hash: string; }): Promise; export function revokeApiKey(db: Db, tenantId: string, id: string): Promise; export function findApiKeyByHash(db: Db, hash: string): Promise; ``` **Acceptance:** - [ ] `listApiKeys` output type contains no `key_hash` field. - [ ] `findApiKeyByHash` excludes revoked keys. ### Task 12: API key routes **Blocks:** 15 · **Blocked by:** 11 **Files:** - Create: `apps/zync-api/src/routes/api-keys.ts` - Modify: `apps/zync-api/src/index.ts` (mount router) **Steps:** - [ ] All routes use `authMiddleware`. Per the permissions table, create/revoke require `requirePermission('settings:write')` **AND** OWNER role (check `session.role === 'OWNER'` / `RoleId.OWNER`); list requires `requirePermission('settings:read')`. - [ ] Enforce tier gate `requireTier(TenantTier.BUSINESS)` for key creation; reject Freelancer with `403 { error: 'tier_required', minimum_tier: 'business' }`. - [ ] Enforce per-tier key caps via `countActiveApiKeys` (business=5, enterprise/white_label=20); exceeding returns `409 { error: 'key_limit_reached' }`. - [ ] Validate requested scopes (Zod): standard `ApiScope` always allowed; `EnterpriseApiScope` (`leads:write`, `campaigns:write`) only allowed for enterprise/white_label tiers (else `422 invalid_scope`). - [ ] `GET /api/api-keys` → `listApiKeys` (prefix + scopes, no full key). - [ ] `POST /api/api-keys` → `generateApiKey()`, `createApiKey` with `createdBy = session.userId`; return `{ ...keyMeta, key: fullKey }` exactly once. - [ ] `DELETE /api/api-keys/:id` → `revokeApiKey`. **Schema / Interfaces:** ``` GET /api/api-keys POST /api/api-keys -> 201 { id, name, key_prefix, scopes, expires_at, key } (key shown once) DELETE /api/api-keys/:id -> 204 ``` **Acceptance:** - [ ] Freelancer tenant `POST /api/api-keys` → 403 `tier_required` (minimum_tier `business`). - [ ] Non-OWNER with `settings:write` → 403 on create and revoke. - [ ] Full key returned only in the create response; `GET` returns prefix only; business tenant blocked at the 6th active key. ### Task 13: Custom-domain settings UI (`/settings/white-label`) **Blocks:** — · **Blocked by:** 5, 6 **Files:** - Create: `apps/zync-app/src/modules/settings/white-label/DomainsPage.tsx` - Create: `apps/zync-app/src/modules/settings/white-label/useDomains.ts` **Steps:** - [ ] react-query hooks over `GET/POST/DELETE /api/settings/domains`. - [ ] Add-domain form (HTTPS hostname input, Zod-validated); on save, show the CNAME instruction card: `{domain} CNAME portal.zync.is` (copyable). - [ ] Domains table: domain, status badge (`pending`/`verified`/`active`/`error` with `aria-label` text equivalents, not color-only), error message, created date, Remove action (confirm dialog → `DELETE /api/settings/domains/:id`). - [ ] Gate the whole page behind `useTierGate(TenantTier.WHITE_LABEL)` (custom domain is White-Label per entitlement matrix); show upgrade prompt otherwise. - [ ] Components from `@zync/ui` (`Card`, `Table`, `Badge`, `Button`, `Dialog`, `Input`, `Form`); RTL-safe (logical CSS props), `prefers-reduced-motion` respected on any status-polling animation; status conveyed by text + icon, never color alone. **Acceptance:** - [ ] Status is announced to assistive tech via text (not color-only); page is fully usable in RTL Hebrew. - [ ] Non-White-Label tenants see the upgrade prompt, not the form. ### Task 14: Webhooks settings UI (`/settings/integrations/webhooks`) **Blocks:** — · **Blocked by:** 10 **Files:** - Create: `apps/zync-app/src/modules/settings/white-label/WebhooksPage.tsx` - Create: `apps/zync-app/src/modules/settings/white-label/useWebhooks.ts` - Create: `apps/zync-app/src/modules/settings/white-label/DeliveryLogPage.tsx` **Steps:** - [ ] Endpoints table: URL, subscribed events (chips), status (active toggle), last delivery; from `GET /api/webhooks/endpoints`. - [ ] "Add endpoint" form: HTTPS URL, events multi-select sourced from `WEBHOOK_EVENT_CATALOG` (grouped by category), active toggle. On create, surface the auto-generated secret in a shown-once dialog with copy + "you won't see this again" warning. - [ ] Edit (url/events/toggle) via `PATCH`; delete via `DELETE` with confirm. "Send test event" button → `POST .../:id/test`. - [ ] Delivery log page: filter by event/status/date; table of deliveries (event, status badge, response status, attempt, timestamp). "Redeliver" button on failed rows → `POST /api/webhooks/deliveries/:id/redeliver`. - [ ] `@zync/ui` components; RTL-safe; status by text+icon; `prefers-reduced-motion`; gate behind `useTierGate(TenantTier.WHITE_LABEL)`. **Acceptance:** - [ ] The webhook secret is displayed exactly once at creation and never re-fetched. - [ ] Multi-select offers exactly the 29 catalog events grouped by category. - [ ] "Redeliver" appears only on failed deliveries. ### Task 15: API-keys settings UI (`/settings/api-keys`) **Blocks:** — · **Blocked by:** 12 **Files:** - Create: `apps/zync-app/src/modules/settings/white-label/ApiKeysPage.tsx` - Create: `apps/zync-app/src/modules/settings/white-label/useApiKeys.ts` **Steps:** - [ ] Keys table: name, prefix (`zyk_live…`), scopes (chips), last used, expires, created; from `GET /api/api-keys`. Revoke action (confirm → `DELETE`). - [ ] "Create key" form: name, scope multi-select (standard `ApiScope`; enterprise-only scopes shown only for enterprise/white_label tier), optional expiry date. On create, show the full key once in a copy dialog with a strong "shown only once" warning. - [ ] Tier gate: for Freelancer, replace the create form with an upgrade prompt (`useTierGate(TenantTier.BUSINESS)`). For non-OWNER users, disable create/revoke with an explanatory tooltip. - [ ] Show remaining-key-quota hint based on tier cap (5 / 20). - [ ] `@zync/ui` components; RTL-safe; reduced-motion respected. **Acceptance:** - [ ] Freelancer sees an upgrade prompt instead of the create form. - [ ] The full API key is shown exactly once; the list shows prefix only. - [ ] Non-OWNER users cannot create or revoke keys from the UI. ### Task 16: Wiring — routes, cron, queue consumer, host middleware, bindings **Blocks:** — · **Blocked by:** 5, 6, 9, 10, 12 **Files:** - Modify: `apps/zync-api/src/index.ts` (mount `domains`, `webhooks`, `api-keys` routers; register host-resolution middleware before portal routes; export `scheduled` + `queue` handlers) - Modify: `apps/zync-api/wrangler.toml` - Modify: `apps/zync-api/src/routes/domains.ts` (mount `/api/settings/domains` — list/create/delete; delete must call `deleteCustomHostname` **before** `softDeleteDomain` and KV-invalidate the host cache) **Steps:** - [ ] Mount routers: `/api/settings/domains` (Task 5/6 backed), `/api/webhooks/*` (Task 10), `/api/api-keys` (Task 12). - [ ] Register the host-resolution middleware (Task 6) early in the chain so custom-domain requests reach the tenant-portals portal handler. - [ ] Add the `scheduled(event, env, ctx)` handler dispatching `domain-verify` on the `*/15 * * * *` trigger; add the `queue(batch, env, ctx)` handler routing `zync-jobs` messages with `type:'webhook.deliver'` to `handleWebhookDeliver`. - [ ] In `wrangler.toml`: add `[triggers] crons = ["*/15 * * * *"]` (merge with existing); ensure `[[queues.consumers]] queue = "zync-jobs"` and `[[queues.producers]] queue = "zync-jobs", binding = "QUEUE"` (no dotted CF queue name — producers send `{ type:'webhook.deliver', ... }`); declare secrets `CF_CUSTOM_HOSTNAME_API_TOKEN`, `CF_ZONE_ID` (and confirm existing `INTEGRATION_ENCRYPTION_KEY`); confirm `KV`, `RATE_LIMITER_WEBHOOK` bindings present. - [ ] `DELETE /api/settings/domains/:id`: synchronous removal — call `deleteCustomHostname(env, hostnameId)` first (prevents CNAME-squatting window), then `softDeleteDomain`, then delete the `domain:{host}` KV cache key. **Acceptance:** - [ ] `wrangler deploy --dry-run` resolves all bindings, the cron trigger, and the queue consumer with no missing-binding errors. - [ ] Domain deletion unprovisions the CF hostname before the DB row is soft-deleted. ### Task 17: Emit webhook events at source state-change sites **Blocks:** — · **Blocked by:** 8 **Files:** - Modify: invoice status handler (`invoice.proforma_approved`, `invoice.issued`, `invoice.paid`, `invoice.overdue`, `retainer.depleted`) - Modify: time-management timer handlers (`timer.started`, `timer.stopped`, `timer.auto_paused`) - Modify: leads/proposals handlers (`lead.created`, `lead.stage_updated`, `proposal.viewed`, `proposal.accepted`) - Modify: projects handler (`project.created`, `project.status_changed`) - Modify: tasks handlers (`task.created`, `task.assigned`, `task.completed`) - Modify: support/tickets handlers (`ticket.created`, `ticket.replied`, `ticket.resolved`) - Modify: expenses handlers (`expense.submitted`, `expense.approved`) - Modify: payouts handler (`payout.generated`) - Modify: billing handlers (`payment.completed`, `payment.failed`) - Modify: users/roles handlers (`user.invited`, `role.updated`) - Modify: tenant provisioning (`tenant.provisioned`) - Modify: calendar booking handler (`calendar.booking_created`) **Steps:** - [ ] At each state-change site, after the transaction commits, call `emitWebhookEvent(env, tenantId, '', data)` with a stable, snake_case-keyed `data` payload (entity id + the minimal fields the event implies). - [ ] Wrap emission so a webhook failure never rolls back or fails the originating operation (emit is fire-and-forget; the queue owns retry). - [ ] For modules not yet built at this wave, leave a documented integration note in that module's plan; emit calls are added when the owning module lands (the emitter is the stable contract). **Acceptance:** - [ ] Paying an invoice (`TAX_ISSUED → PAID`) emits `invoice.paid`, enqueuing deliveries to all subscribed active endpoints. - [ ] An emit failure does not affect the originating operation's success. - [ ] Every one of the 29 catalog events has exactly one emit call-site (or a documented deferred note in the owning module's plan).