# White-Label API & Custom Domains

**Date:** 2026-05-30  
**Status:** Draft  
**Depends on:** `foundation-auth-rbac`, `tenant-portals`, `system-communications-notifications`  
**Referenced by:** `audit-compliance`

---

## Overview

Enterprise tier features: (1) custom domains for the customer portal (CNAME → Workers routing); (2) full outbound webhook system (all system events); (3) API key management for tenant-controlled machine-to-machine access. The custom domain + webhook gateway is what makes Zync headless-capable for larger enterprise tenants.

---

## Custom Domains

Enterprise tenants can map their own domain (e.g. `portal.acme.com`) to their Zync customer portal.

### How It Works

1. Tenant enters domain in `/settings/white-label` → saved to `tenant_domains.domain`
2. System shows CNAME instruction: `portal.acme.com CNAME portal.zync.is`
3. Tenant adds CNAME in their DNS
4. System verifies DNS resolution (polls until CNAME resolves, max 48h)
5. Cloudflare custom hostname provisioned via CF API: `POST /zones/{zoneId}/custom_hostnames`
6. TLS certificate auto-provisioned by Cloudflare (Universal SSL)
7. Workers route: incoming request with `Host: portal.acme.com` → lookup `tenant_domains` → resolve `tenantSlug` → serve portal with tenant branding

### Data Model

```sql
tenant_domains (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  domain TEXT NOT NULL UNIQUE,           -- e.g. 'portal.acme.com'
  cloudflare_hostname_id TEXT,           -- CF custom hostname ID (for cert management)
  verified_at TIMESTAMPTZ,               -- null = pending DNS verification
  status TEXT DEFAULT 'PENDING',         -- 'PENDING' | 'ACTIVE' | 'ERROR'
  error_message TEXT,
  created_at TIMESTAMPTZ DEFAULT now()
)
```

### Domain Verification Security Lifecycle

Custom hostnames in Cloudflare are provisioned **only after** DNS verification completes. Pre-provisioning before verification would allow a dangling CNAME.

**Provisioning flow:**
1. Tenant adds domain → `tenant_domains` record created with `status = 'pending'`; CF custom hostname is **not yet created**
2. DNS instructions shown to tenant: add CNAME `{domain}` → `portal.zync.is`
3. `domain-verify` cron (every 15 min): calls CF API `GET /zones/{zone}/custom_hostnames/{hostnameId}` for all `pending` records
4. Transition to `verified` only when CF returns **both** `ssl.status = 'active'` AND `status = 'active'`
5. CF custom hostname created at this point via `POST /zones/{zone}/custom_hostnames`
6. Domains remaining `pending` for > 48 hours: soft-delete `tenant_domains` record; skip hostname creation

**Removal (must be synchronous):**
- On domain removal (tenant-initiated or admin): call `DELETE /zones/{zone}/custom_hostnames/{hostnameId}` **before** soft-deleting the DB record
- This prevents the window where the CNAME is live but unassigned (squatting opportunity)

**Uniqueness enforcement:**
- Before creating a new `tenant_domains` record: check `SELECT 1 FROM tenant_domains WHERE domain = $1 AND status = 'verified' AND tenant_id != $2`
- If found: return `409 domain_already_verified` — a tenant cannot claim a domain actively verified by another tenant

**Required CF API permissions:**
- `domain-verify` cron requires `CF_API_TOKEN` with scope: `Zone:Custom Hostnames:Edit` (zync.is zone only)

---

## Webhook Gateway

Full outbound webhook system. Tenants configure webhook endpoints; system emits signed events on every state change.

### Webhook Endpoints Management (`/settings/integrations/webhooks`)

Table: URL, Events subscribed, Secret, Status, Last delivery.

"Add endpoint" form:
- URL (HTTPS required)
- Secret (auto-generated; HMAC-SHA256 signing key; shown once)
- Events: multi-select from full event catalog
- Active toggle

Stored in `webhook_endpoints` table.

### Data Model

```sql
webhook_endpoints (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  url TEXT NOT NULL,
  secret TEXT NOT NULL,                  -- HMAC signing key (AES-256-GCM encrypted via INTEGRATION_ENCRYPTION_KEY; never shown again after creation)
  events TEXT[] NOT NULL,                -- subscribed event types
  is_active BOOLEAN DEFAULT true,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
)

webhook_deliveries (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  endpoint_id UUID NOT NULL,
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT DEFAULT 'pending',         -- 'pending' | 'delivered' | 'failed' | 'test'
  response_status INTEGER,
  response_body TEXT,
  attempt INTEGER DEFAULT 1,
  delivered_at TIMESTAMPTZ,
  next_retry_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ DEFAULT now()
)
```

### Event Delivery

On any system event (invoice.paid, lead.created, etc.):
1. Find all active `webhook_endpoints` for this tenant subscribed to this event
2. For each: enqueue `webhook.deliver` job with `{ endpointId, eventType, payload }`
3. Consumer: HTTP POST to endpoint URL with:
   - Body: `{ event: "invoice.paid", tenantId, data: {...}, timestamp }`
   - Headers:
     - `X-Zync-Signature: sha256=HMAC_SHA256(secret, "${timestamp}.${body}")`
     - `X-Zync-Timestamp: {unix_seconds}` — seconds since epoch (integer); included in signature
     - `X-Zync-Event: invoice.paid`
     - `X-Zync-Delivery: {deliveryId}`
4. On 2xx: status → `delivered`
5. On non-2xx / timeout: retry with exponential backoff (1m, 5m, 30m, 2h, 24h)
6. After 5 failures: status → `failed`, notify tenant admin

Secret is HMAC key — stored AES-256-GCM encrypted via `INTEGRATION_ENCRYPTION_KEY` (same as adapter credentials). Retrieved and decrypted per delivery to compute signature. If compromised: rotate secret (regenerate + update endpoint).

### Signature verification (receiver-side)

Receivers should verify the signature using this pattern:

```ts
// Recipient's verification code (reference implementation)
function verifyZyncWebhook(
  body: string,
  signature: string,  // from X-Zync-Signature header: "sha256=..."
  timestamp: string,  // from X-Zync-Timestamp header
  secret: string
): boolean {
  const expectedSig = 'sha256=' + hmacSHA256(secret, `${timestamp}.${body}`)
  return timingSafeEqual(signature, expectedSig)
}
```

### Replay protection

The `X-Zync-Timestamp` header enables receivers to reject replayed requests:

```ts
// Receiver middleware (recommended guard):
const MAX_AGE_SECONDS = 300  // 5 minutes

function isReplayAttack(timestampHeader: string): boolean {
  const ts = parseInt(timestampHeader, 10)
  if (isNaN(ts)) return true
  return Math.abs(Date.now() / 1000 - ts) > MAX_AGE_SECONDS
}
```

Receivers should:
1. Check `|now - X-Zync-Timestamp| <= 300s` — reject if outside window
2. Store seen `X-Zync-Delivery` IDs for the same 5-minute window (e.g. in Redis/KV) — reject if already processed
3. Only after passing both checks: verify HMAC signature

**Delivery ID uniqueness:** `X-Zync-Delivery` is a UUID generated per delivery attempt. Retries of the same event reuse the same delivery ID (so replay protection at the delivery-ID level is safe — retries are idempotent by design). A "Redeliver" action via the Zync UI generates a **new** delivery ID to distinguish from the original delivery.

### Worker-side timestamp injection

The `X-Zync-Timestamp` value is injected by the queue consumer at delivery time (not at event enqueue time). This ensures the timestamp reflects the actual delivery moment, not the event creation time. The timestamp is included in the HMAC computation to bind signature and timestamp together — a signature without the timestamp would be valid indefinitely.

### Full Event Catalog

| Category | Events |
|----------|--------|
| Time | `timer.started`, `timer.stopped`, `timer.auto_paused` |
| CRM | `lead.created`, `lead.stage_updated`, `proposal.viewed`, `proposal.accepted` |
| Projects | `project.created`, `project.status_changed` |
| Tasks | `task.created`, `task.assigned`, `task.completed` |
| Support | `ticket.created`, `ticket.replied`, `ticket.resolved` |
| Finance | `invoice.proforma_approved`, `invoice.issued`, `invoice.paid`, `invoice.overdue`, `retainer.depleted` |
| Expenses | `expense.submitted`, `expense.approved` |
| Payouts | `payout.generated` |
| Billing | `payment.completed`, `payment.failed` |
| Users | `user.invited`, `role.updated` |
| Tenant | `tenant.provisioned` |
| Calendar | `calendar.booking_created` |

### Delivery Logs

`GET /api/webhooks/deliveries` — list recent deliveries per endpoint. Filter by event, status, date. "Redeliver" button → re-enqueues failed delivery.

---

## API Keys (Machine-to-Machine)

**Business+ tenants** can create API keys for external integrations without exposing staff credentials. This `tenant_api_keys` table is the authoritative schema for all API key operations — shared with the Tenant Public API (`2026-05-31-tenant-public-api`, spec 39).

**Tier split:**
- `business` / `enterprise` / `white_label` — can create keys; tier gate enforced at `api.zync.is/v1/` middleware (spec 39)
- `freelancer` — cannot create keys (403 `tier_required`)

Enterprise and White-label tenants can additionally use keys with broader internal scopes (e.g. `leads:write`, `campaigns:write`) for reseller automation workflows beyond spec 39's standard public API surface.

```sql
tenant_api_keys (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  name TEXT NOT NULL,                    -- descriptive (e.g. "Zapier integration")
  key_prefix TEXT NOT NULL,              -- first 8 chars shown in UI: e.g. "zyk_live"
  key_hash TEXT NOT NULL,                -- SHA-256 of full key (for lookup)
  scopes TEXT[] NOT NULL,                -- e.g. ['customers:read', 'invoices:write']
  last_used_at TIMESTAMPTZ,
  expires_at TIMESTAMPTZ,                -- null = never expires
  created_by UUID NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  revoked_at TIMESTAMPTZ
)
```

Key format: `zyk_live_{random32}` (32 random alphanumeric chars; total length 42). Example: `zyk_live_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6`. Shown once on creation. Stored as SHA-256 hash only.

API key auth: `Authorization: Bearer zyk_live_...` → hash lookup → check scopes → resolve tenant.

Standard public API scopes (`customers:read/write`, `invoices:read/write`, `tasks:read/write`, `events:read`) are defined in spec 39. Enterprise-only scopes (e.g. `leads:write`) are enforced at the endpoint level.

---

## Permissions

| Action | Required permission |
|--------|-------------------|
| View webhook settings | `settings:read` |
| Create/edit webhooks | `settings:write` |
| View delivery logs | `settings:read` |
| Create API keys | `settings:write` + OWNER role required |
| Revoke API keys | `settings:write` + OWNER role required |
| Configure custom domain | `settings:write` (Enterprise tier) |

---

## API Endpoints

```
-- Custom domains
GET    /api/settings/domains             → list tenant's domains
POST   /api/settings/domains             → add custom domain
DELETE /api/settings/domains/:id         → remove domain + unprovision CF hostname

-- Webhooks
GET    /api/webhooks/endpoints           → list configured endpoints
POST   /api/webhooks/endpoints           → add endpoint
PATCH  /api/webhooks/endpoints/:id       → update (url, events, toggle)
DELETE /api/webhooks/endpoints/:id       → delete
POST   /api/webhooks/endpoints/:id/test  → send test event

GET    /api/webhooks/deliveries          → delivery log (filterable)
POST   /api/webhooks/deliveries/:id/redeliver → redeliver failed delivery

-- API keys
GET    /api/api-keys                     → list (prefix + scopes, no full key)
POST   /api/api-keys                     → create (returns full key once)
DELETE /api/api-keys/:id                 → revoke
```

---

## Foundation Deltas

**New cron:** `domain-verify` — every 15 minutes, checks DNS + provisions CF custom hostnames.

**Async delivery:** producers enqueue `{ type: 'webhook.deliver', ... }` on the shared `zync-jobs` queue (`QUEUE` binding); the zync-jobs consumer routes by message type. There is no dedicated `webhook.deliver` Cloudflare queue — CF queue names cannot contain dots. *Rationale: align spec with deployable CF queue naming and existing zync-jobs type-routing.*

**New secret:** `CF_CUSTOM_HOSTNAME_API_TOKEN` — scoped CF API token with `ssl_certs:write` for custom hostname provisioning (not in wrangler OAuth scope; must be created manually in CF dashboard and stored as Worker secret).

---

## Architecture Decisions

| Decision | Choice | Reason |
|----------|--------|--------|
| Custom hostnames via CF API | Not Cloudflare for SaaS (Orange-to-Orange) | Tenants bring their own domain on any registrar; CF custom hostnames on portal zone handles TLS automatically without requiring tenant to be on Cloudflare |
| Webhook secret encrypted | AES-256-GCM via `INTEGRATION_ENCRYPTION_KEY` (not hashed) | Must be *recoverable* — Zync signs outbound deliveries using the raw key. Hashing would throw the key away, making HMAC impossible. Encrypted + decrypt-per-send is the correct pattern (same as adapter credentials). DB compromise → rotate `INTEGRATION_ENCRYPTION_KEY`. |
| Exponential backoff with 5 retries | Not unlimited | Prevents overwhelming failing endpoints; 24h ceiling gives ~1 day before admin alert |
| API keys SHA-256 hashed | Not encrypted | Hash is one-way; no decryption possible even with DB access; key shown once on creation by design |
| Scoped API keys | Explicit scope array | Least-privilege; external integrations get only what they need; mirrors RBAC permission strings |
| Webhook delivery via queue | Not sync in event handler | Event handler (e.g. invoice.paid) must return fast; delivery can take 5-30s; queue handles async + retry + DLQ |