# Translate Press Zone - Comprehensive Backend API Plan

# Skill: React Admin Dashboard

## Identity
- **Skill ID**: `admin-dashboard-react`
- **Domain**: Admin Panel Frontend, React UI
- **Technologies**: React 18, TypeScript, Vite, TanStack Query, Zustand
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- React component development for admin panel
- Admin dashboard UI/UX
- State management with Zustand
- Data fetching with TanStack Query
- Admin authentication flow
- Charts and analytics visualization
- User management interface
- System settings UI

**File patterns:**
- `admin-panel/src/**/*.tsx`
- `admin-panel/src/**/*.ts`
- `admin-panel/src/components/**/*`
- `admin-panel/src/pages/**/*`
- `admin-panel/src/hooks/**/*`

## Core Patterns

### 1. Project Structure

```
admin-panel/
├── src/
│   ├── components/       # Reusable UI components
│   ├── pages/            # Page components
│   ├── hooks/            # Custom React hooks
│   ├── stores/           # Zustand state stores
│   ├── services/         # API service layer
│   ├── types/            # TypeScript types
│   ├── utils/            # Utility functions
│   ├── App.tsx           # Root component
│   └── main.tsx          # Entry point
├── public/
├── index.html
├── vite.config.ts
├── tsconfig.json
└── package.json
```

### 2. Authentication Store (Zustand)

```typescript
// stores/authStore.ts
import { create } from 'zustand';
import { persist } from 'zustand/middleware';

interface User {
  id: string;
  email: string;
  role: 'admin' | 'support';
}

interface AuthState {
  user: User | null;
  isAuthenticated: boolean;
  login: (email: string, password: string) => Promise<void>;
  logout: () => Promise<void>;
  refreshToken: () => Promise<void>;
}

export const useAuthStore = create<AuthState>()(
  persist(
    (set) => ({
      user: null,
      isAuthenticated: false,
      
      login: async (email: string, password: string) => {
        const response = await fetch('/api/auth/login', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ email, password }),
          credentials: 'include' // Important for cookies
        });
        
        if (!response.ok) {
          throw new Error('Login failed');
        }
        
        const data = await response.json();
        set({ user: data.user, isAuthenticated: true });
      },
      
      logout: async () => {
        await fetch('/api/auth/logout', {
          method: 'POST',
          credentials: 'include'
        });
        set({ user: null, isAuthenticated: false });
      },
      
      refreshToken: async () => {
        const response = await fetch('/api/auth/refresh', {
          method: 'POST',
          credentials: 'include'
        });
        
        if (!response.ok) {
          set({ user: null, isAuthenticated: false });
          throw new Error('Token refresh failed');
        }
      }
    }),
    {
      name: 'auth-storage',
      partialize: (state) => ({ user: state.user, isAuthenticated: state.isAuthenticated })
    }
  )
);
```

### 3. API Service Layer

```typescript
// services/api.ts
class ApiError extends Error {
  constructor(public status: number, message: string) {
    super(message);
    this.name = 'ApiError';
  }
}

async function fetchApi<T>(url: string, options?: RequestInit): Promise<T> {
  const response = await fetch(`/api${url}`, {
    ...options,
    credentials: 'include',
    headers: {
      'Content-Type': 'application/json',
      ...options?.headers
    }
  });
  
  if (response.status === 401) {
    // Token expired, try to refresh
    const refreshed = await useAuthStore.getState().refreshToken();
    if (refreshed) {
      // Retry original request
      return fetchApi<T>(url, options);
    } else {
      throw new ApiError(401, 'Unauthorized');
    }
  }
  
  if (!response.ok) {
    const error = await response.json().catch(() => ({ error: 'Unknown error' }));
    throw new ApiError(response.status, error.error || 'Request failed');
  }
  
  return response.json();
}

// Analytics service
export const analyticsService = {
  getDashboardMetrics: () =>
    fetchApi<DashboardMetrics>('/admin/analytics'),
    
  getUserStats: (params: { startDate: string; endDate: string }) =>
    fetchApi<UserStats>(`/admin/analytics/users?${new URLSearchParams(params)}`)
};

// Users service
export const usersService = {
  getUsers: (page: number, limit: number) =>
    fetchApi<PaginatedUsers>(`/admin/users?page=${page}&limit=${limit}`),
    
  getUser: (id: string) =>
    fetchApi<User>(`/admin/users/${id}`),
    
  updateUser: (id: string, data: Partial<User>) =>
    fetchApi<User>(`/admin/users/${id}`, {
      method: 'PUT',
      body: JSON.stringify(data)
    }),
    
  allocateCredits: (userId: string, amount: number, description: string) =>
    fetchApi(`/admin/credits`, {
      method: 'POST',
      body: JSON.stringify({ user_id: userId, amount, description })
    })
};
```

### 4. Data Fetching with TanStack Query

```typescript
// hooks/useUsers.ts
import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query';
import { usersService } from '../services/api';

export function useUsers(page: number, limit: number = 20) {
  return useQuery({
    queryKey: ['users', page, limit],
    queryFn: () => usersService.getUsers(page, limit),
    staleTime: 30000, // 30 seconds
    gcTime: 300000 // 5 minutes (formerly cacheTime)
  });
}

export function useUser(id: string) {
  return useQuery({
    queryKey: ['user', id],
    queryFn: () => usersService.getUser(id),
    enabled: !!id // Only run if ID exists
  });
}

export function useUpdateUser() {
  const queryClient = useQueryClient();
  
  return useMutation({
    mutationFn: ({ id, data }: { id: string; data: Partial<User> }) =>
      usersService.updateUser(id, data),
      
    onSuccess: (updatedUser) => {
      // Update user list cache
      queryClient.invalidateQueries({ queryKey: ['users'] });
      
      // Update specific user cache
      queryClient.setQueryData(['user', updatedUser.id], updatedUser);
    }
  });
}
```

### 5. Dashboard Page Component

```typescript
// pages/Dashboard.tsx
import { useQuery } from '@tanstack/react-query';
import { analyticsService } from '../services/api';
import { MetricCard } from '../components/MetricCard';
import { RevenueChart } from '../components/RevenueChart';

export function Dashboard() {
  const { data: metrics, isLoading, error } = useQuery({
    queryKey: ['dashboard-metrics'],
    queryFn: analyticsService.getDashboardMetrics,
    refetchInterval: 60000 // Refresh every minute
  });
  
  if (isLoading) {
    return <LoadingSpinner />;
  }
  
  if (error) {
    return <ErrorMessage error={error} />;
  }
  
  return (
    <div className="dashboard">
      <h1>Dashboard</h1>
      
      <div className="metrics-grid">
        <MetricCard
          title="Total Users"
          value={metrics.totalUsers}
          change={metrics.usersChange}
          icon={<UsersIcon />}
        />
        <MetricCard
          title="Monthly Revenue"
          value={`$${metrics.revenue.toFixed(2)}`}
          change={metrics.revenueChange}
          icon={<DollarIcon />}
        />
        <MetricCard
          title="Active Jobs"
          value={metrics.activeJobs}
          icon={<JobsIcon />}
        />
        <MetricCard
          title="Tokens Processed"
          value={metrics.tokensProcessed.toLocaleString()}
          icon={<TokensIcon />}
        />
      </div>
      
      <div className="charts">
        <RevenueChart data={metrics.revenueHistory} />
        <UsageChart data={metrics.usageHistory} />
      </div>
    </div>
  );
}
```

### 6. User Management Page

```typescript
// pages/Users.tsx
import { useState } from 'react';
import { useUsers } from '../hooks/useUsers';
import { UserTable } from '../components/UserTable';
import { Pagination } from '../components/Pagination';

export function Users() {
  const [page, setPage] = useState(1);
  const { data, isLoading, error } = useUsers(page, 20);
  
  if (isLoading) return <LoadingSpinner />;
  if (error) return <ErrorMessage error={error} />;
  
  return (
    <div className="users-page">
      <div className="page-header">
        <h1>Users</h1>
        <SearchInput onSearch={(query) => {/* Filter users */}} />
      </div>
      
      <UserTable users={data.users} />
      
      <Pagination
        currentPage={page}
        totalPages={data.pagination.pages}
        onPageChange={setPage}
      />
    </div>
  );
}

// components/UserTable.tsx
interface UserTableProps {
  users: User[];
}

export function UserTable({ users }: UserTableProps) {
  const updateUser = useUpdateUser();
  
  const handleSuspend = async (userId: string) => {
    if (!confirm('Suspend this user?')) return;
    
    await updateUser.mutateAsync({
      id: userId,
      data: { status: 'suspended' }
    });
  };
  
  return (
    <table className="user-table">
      <thead>
        <tr>
          <th>Email</th>
          <th>Plan</th>
          <th>Credits</th>
          <th>Status</th>
          <th>Actions</th>
        </tr>
      </thead>
      <tbody>
        {users.map(user => (
          <tr key={user.id}>
            <td>{user.email}</td>
            <td>{user.subscription?.plan_tier || 'None'}</td>
            <td>{user.credits?.toLocaleString()}</td>
            <td>
              <StatusBadge status={user.status} />
            </td>
            <td>
              <button onClick={() => handleSuspend(user.id)}>
                Suspend
              </button>
              <button onClick={() => {/* Navigate to user detail */}}>
                View
              </button>
            </td>
          </tr>
        ))}
      </tbody>
    </table>
  );
}
```

### 7. Protected Routes

```typescript
// components/ProtectedRoute.tsx
import { Navigate } from 'react-router-dom';
import { useAuthStore } from '../stores/authStore';

interface ProtectedRouteProps {
  children: React.ReactNode;
  requiredRole?: 'admin' | 'support';
}

export function ProtectedRoute({ children, requiredRole }: ProtectedRouteProps) {
  const { isAuthenticated, user } = useAuthStore();
  
  if (!isAuthenticated) {
    return <Navigate to="/login" replace />;
  }
  
  if (requiredRole && user?.role !== requiredRole) {
    return <Navigate to="/unauthorized" replace />;
  }
  
  return <>{children}</>;
}

// App.tsx
import { BrowserRouter, Routes, Route } from 'react-router-dom';

function App() {
  return (
    <BrowserRouter>
      <Routes>
        <Route path="/login" element={<LoginPage />} />
        
        <Route
          path="/"
          element={
            <ProtectedRoute>
              <Layout />
            </ProtectedRoute>
          }
        >
          <Route index element={<Dashboard />} />
          <Route path="users" element={<Users />} />
          <Route path="users/:id" element={<UserDetail />} />
          <Route path="jobs" element={<Jobs />} />
          <Route
            path="settings"
            element={
              <ProtectedRoute requiredRole="admin">
                <Settings />
              </ProtectedRoute>
            }
          />
        </Route>
      </Routes>
    </BrowserRouter>
  );
}
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Fetching data in `useEffect` | Use TanStack Query hooks |
| Storing server state in component state | Use TanStack Query for server data |
| Not handling loading/error states | Always show loaders and error messages |
| Prop drilling | Use Zustand for global state |
| Inline styles | Use CSS modules or styled-components |
| Not memoizing expensive computations | Use `useMemo` and `useCallback` |
| Mutating state directly | Use immutable updates |
| No TypeScript types | Define interfaces for all data |
| Hardcoding API URLs | Use environment variables |
| Not using React Router for navigation | Use `<Link>` and `useNavigate()` |

## Integration with Other Skills

**Often combined with:**
- `api-endpoint-creation` - Consuming backend APIs
- `authentication-security` - JWT token handling
- `error-handling-logging` - Error reporting to backend

## Environment Variables Required

```bash
# .env
VITE_API_BASE_URL=http://localhost:3000
VITE_APP_NAME="TranslatePressZone Admin"
```

## Quick Reference

### TanStack Query Configuration

```typescript
// main.tsx
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';

const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      staleTime: 60000, // 1 minute
      gcTime: 300000, // 5 minutes
      retry: 1,
      refetchOnWindowFocus: false
    }
  }
});

ReactDOM.createRoot(document.getElementById('root')!).render(
  <React.StrictMode>
    <QueryClientProvider client={queryClient}>
      <App />
    </QueryClientProvider>
  </React.StrictMode>
);
```

### Common Hooks

| Hook | Purpose |
|------|---------|
| `useQuery` | Fetch and cache data |
| `useMutation` | Create/update/delete data |
| `useQueryClient` | Access query cache |
| `useInfiniteQuery` | Infinite scroll/pagination |

## Validation Checklist

- [ ] All API calls use service layer
- [ ] TanStack Query for all server data
- [ ] Loading states handled
- [ ] Error states handled with user-friendly messages
- [ ] Protected routes implemented
- [ ] JWT token refresh on 401
- [ ] Zustand for global UI state (not server data)
- [ ] TypeScript types for all data
- [ ] Environment variables for API URLs
- [ ] React Router for navigation
- [ ] Responsive design (mobile-friendly)
- [ ] Accessibility (ARIA labels, keyboard nav)

## Build Configuration

```typescript
// vite.config.ts
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  server: {
    port: 5173,
    proxy: {
      '/api': {
        target: 'http://localhost:3000',
        changeOrigin: true
      }
    }
  }
});
```
# Skill: API Endpoint Creation (Express + Zod)

## Identity
- **Skill ID**: `api-endpoint-creation`
- **Domain**: RESTful API Endpoints
- **Technologies**: Express.js, Zod Validation, TypeScript
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill
- Creating REST API endpoints
- Request validation with Zod
- Error handling
- Files matching: `press-zone-backend/api/src/routes/**/*.ts`, `press-zone-backend/api/src/controllers/**/*.ts`

## Core Patterns

### Express Router Setup
```typescript
import express from 'express';
import { authenticateApiKey } from '../middleware/auth';
import { validateRequest } from '../middleware/validation';
import { translationSchema } from '../schemas/translation';

const router = express.Router();

router.post('/translate', 
  authenticateApiKey,
  validateRequest(translationSchema),
  async (req, res) => {
    try {
      const { source_lang, target_lang, content, model } = req.body;
      
      // Business logic here
      const result = await translateService.process({
        userId: req.user.id,
        sourceLang: source_lang,
        targetLang: target_lang,
        content,
        model
      });
      
      res.json({ success: true, data: result });
    } catch (error) {
      res.status(500).json({ success: false, error: error.message });
    }
  }
);

export default router;
```

### Zod Schema Validation
```typescript
import { z } from 'zod';

export const translationSchema = z.object({
  source_lang: z.string().length(2),
  target_lang: z.string().length(2),
  content: z.string().min(1).max(50000),
  model: z.enum(['4b', '27b']).default('4b'),
  tone: z.enum(['neutral', 'formal', 'casual']).optional(),
  callback_url: z.string().url().optional()
});

export type TranslationRequest = z.infer<typeof translationSchema>;
```

### Validation Middleware
```typescript
import { Request, Response, NextFunction } from 'express';
import { ZodSchema } from 'zod';

export const validateRequest = (schema: ZodSchema) => {
  return (req: Request, res: Response, next: NextFunction) => {
    try {
      schema.parse(req.body);
      next();
    } catch (error) {
      res.status(400).json({
        success: false,
        error: 'Validation failed',
        details: error.errors
      });
    }
  };
};
```

## Validation Checklist
- [ ] All endpoints have authentication middleware
- [ ] Request bodies validated with Zod schemas
- [ ] Error responses use consistent format
- [ ] Rate limiting applied
- [ ] CORS configured properly
# Skill: Authentication & Security

## Identity
- **Skill ID**: `authentication-security`
- **Domain**: Authentication, Authorization, Security
- **Technologies**: JWT, bcrypt, SHA-256, API Keys, Rate Limiting
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- User registration and login endpoints
- API key generation and validation
- JWT token management (access/refresh)
- Password hashing and verification
- Rate limiting implementation
- Session management
- Security headers and middleware
- HMAC signature verification for webhooks

**File patterns:**
- `api/src/middleware/auth*.ts`
- `api/src/routes/auth*.ts`
- `api/src/utils/security*.ts`
- `api/src/middleware/rate-limit*.ts`

## Core Patterns

### 1. JWT Authentication (Admin Panel)

```typescript
import jwt from 'jsonwebtoken';
import { Response } from 'express';

// Token generation
interface TokenPayload {
  userId: string;
  email: string;
}

function generateTokens(payload: TokenPayload) {
  const accessToken = jwt.sign(
    payload,
    process.env.JWT_ACCESS_SECRET!,
    { expiresIn: '15m' } // Short-lived access token
  );

  const refreshToken = jwt.sign(
    payload,
    process.env.JWT_REFRESH_SECRET!,
    { expiresIn: '7d' } // Long-lived refresh token
  );

  return { accessToken, refreshToken };
}

// Set secure httpOnly cookies
function setAuthCookies(res: Response, accessToken: string, refreshToken: string) {
  res.cookie('access_token', accessToken, {
    httpOnly: true,
    secure: process.env.NODE_ENV === 'production',
    sameSite: 'strict',
    maxAge: 15 * 60 * 1000 // 15 minutes
  });

  res.cookie('refresh_token', refreshToken, {
    httpOnly: true,
    secure: process.env.NODE_ENV === 'production',
    sameSite: 'strict',
    maxAge: 7 * 24 * 60 * 60 * 1000 // 7 days
  });
}

// Token verification middleware
import { Request, Response, NextFunction } from 'express';

interface AuthRequest extends Request {
  user?: TokenPayload;
}

async function requireAuth(req: AuthRequest, res: Response, next: NextFunction) {
  try {
    const token = req.cookies.access_token;
    
    if (!token) {
      return res.status(401).json({ error: 'Authentication required' });
    }

    const decoded = jwt.verify(token, process.env.JWT_ACCESS_SECRET!) as TokenPayload;
    req.user = decoded;
    next();
  } catch (error) {
    if (error instanceof jwt.TokenExpiredError) {
      return res.status(401).json({ error: 'Token expired' });
    }
    return res.status(401).json({ error: 'Invalid token' });
  }
}

// Refresh token endpoint
router.post('/auth/refresh', async (req, res) => {
  try {
    const refreshToken = req.cookies.refresh_token;
    
    if (!refreshToken) {
      return res.status(401).json({ error: 'Refresh token required' });
    }

    const decoded = jwt.verify(refreshToken, process.env.JWT_REFRESH_SECRET!) as TokenPayload;
    
    // Generate new tokens
    const tokens = generateTokens({ userId: decoded.userId, email: decoded.email });
    setAuthCookies(res, tokens.accessToken, tokens.refreshToken);
    
    res.json({ success: true });
  } catch (error) {
    res.status(401).json({ error: 'Invalid refresh token' });
  }
});
```

### 2. API Key Authentication (WordPress Plugin)

```typescript
import crypto from 'crypto';
import { prisma } from '../lib/prisma';

// Generate API key
async function generateApiKey(userId: string, name: string) {
  // Format: sk_live_{32 random chars}
  const randomBytes = crypto.randomBytes(24);
  const apiKey = `sk_live_${randomBytes.toString('hex')}`;
  
  // Store SHA-256 hash in database
  const keyHash = crypto
    .createHash('sha256')
    .update(apiKey)
    .digest('hex');
  
  const prefix = apiKey.substring(0, 8); // Store prefix for display
  
  await prisma.apiKey.create({
    data: {
      user_id: userId,
      key_hash: keyHash,
      prefix,
      name,
      is_active: true
    }
  });
  
  // Return plain text key ONCE (never stored)
  return apiKey;
}

// Validate API key middleware
async function requireApiKey(req: AuthRequest, res: Response, next: NextFunction) {
  try {
    const authHeader = req.headers.authorization;
    
    if (!authHeader || !authHeader.startsWith('Bearer ')) {
      return res.status(401).json({ error: 'API key required' });
    }
    
    const apiKey = authHeader.substring(7);
    
    // Hash the provided key
    const keyHash = crypto
      .createHash('sha256')
      .update(apiKey)
      .digest('hex');
    
    // Find matching key
    const key = await prisma.apiKey.findUnique({
      where: { key_hash: keyHash },
      include: {
        user: {
          select: {
            id: true,
            email: true,
            status: true
          }
        }
      }
    });
    
    if (!key || !key.is_active) {
      return res.status(401).json({ error: 'Invalid API key' });
    }
    
    if (key.user.status !== 'active') {
      return res.status(403).json({ error: 'Account suspended' });
    }
    
    // Update last_used_at
    await prisma.apiKey.update({
      where: { id: key.id },
      data: { last_used_at: new Date() }
    });
    
    // Attach user to request
    req.user = {
      userId: key.user.id,
      email: key.user.email
    };
    
    next();
  } catch (error) {
    res.status(500).json({ error: 'Authentication failed' });
  }
}
```

### 3. Password Hashing (bcrypt)

```typescript
import bcrypt from 'bcrypt';

const SALT_ROUNDS = 12; // Recommended for production

// Hash password during registration
async function hashPassword(plainPassword: string): Promise<string> {
  return bcrypt.hash(plainPassword, SALT_ROUNDS);
}

// Verify password during login
async function verifyPassword(plainPassword: string, hashedPassword: string): Promise<boolean> {
  return bcrypt.compare(plainPassword, hashedPassword);
}

// Registration endpoint
router.post('/auth/register', async (req, res) => {
  const { email, password } = req.body;
  
  // Password strength validation
  if (password.length < 8) {
    return res.status(400).json({ error: 'Password must be at least 8 characters' });
  }
  
  // Check if email exists
  const existingUser = await prisma.user.findUnique({
    where: { email }
  });
  
  if (existingUser) {
    return res.status(409).json({ error: 'Email already registered' });
  }
  
  // Hash password
  const passwordHash = await hashPassword(password);
  
  // Create user
  const user = await prisma.user.create({
    data: {
      email,
      password_hash: passwordHash
    }
  });
  
  res.status(201).json({
    id: user.id,
    email: user.email
  });
});

// Login endpoint
router.post('/auth/login', async (req, res) => {
  const { email, password } = req.body;
  
  const user = await prisma.user.findUnique({
    where: { email }
  });
  
  if (!user) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }
  
  const isValid = await verifyPassword(password, user.password_hash);
  
  if (!isValid) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }
  
  if (user.status !== 'active') {
    return res.status(403).json({ error: 'Account suspended' });
  }
  
  // Generate tokens
  const tokens = generateTokens({ userId: user.id, email: user.email });
  setAuthCookies(res, tokens.accessToken, tokens.refreshToken);
  
  res.json({
    success: true,
    user: {
      id: user.id,
      email: user.email
    }
  });
});
```

### 4. Rate Limiting

```typescript
import rateLimit from 'express-rate-limit';
import RedisStore from 'rate-limit-redis';
import Redis from 'ioredis';

const redis = new Redis({
  host: process.env.REDIS_HOST,
  port: parseInt(process.env.REDIS_PORT || '6379')
});

// Global rate limit
const globalLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 100, // 100 requests per minute
  message: 'Too many requests, please try again later',
  standardHeaders: true,
  legacyHeaders: false
});

// API key rate limit (per user)
const apiKeyLimiter = rateLimit({
  store: new RedisStore({
    client: redis,
    prefix: 'rl:api:'
  }),
  windowMs: 60 * 60 * 1000, // 1 hour
  max: 1000, // 1000 requests per hour per key
  keyGenerator: (req: AuthRequest) => req.user?.userId || req.ip,
  message: 'API rate limit exceeded',
  standardHeaders: true,
  legacyHeaders: false
});

// Auth endpoint rate limit (prevent brute force)
const authLimiter = rateLimit({
  store: new RedisStore({
    client: redis,
    prefix: 'rl:auth:'
  }),
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 5, // 5 login attempts per 15 minutes
  skipSuccessfulRequests: true, // Don't count successful logins
  message: 'Too many login attempts, please try again later'
});

// Apply middleware
app.use('/api', globalLimiter);
app.use('/api', requireApiKey, apiKeyLimiter);
app.use('/auth/login', authLimiter);
```

### 5. HMAC Signature Verification (Webhooks)

```typescript
import crypto from 'crypto';

// Generate HMAC signature for outgoing webhooks
function generateWebhookSignature(payload: string, secret: string): string {
  return crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
}

// Verify HMAC signature for incoming webhooks (e.g., PayPal)
function verifyWebhookSignature(
  payload: string,
  signature: string,
  secret: string
): boolean {
  const expectedSignature = generateWebhookSignature(payload, secret);
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  );
}

// Webhook middleware
function verifyWebhook(secret: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const signature = req.headers['x-webhook-signature'] as string;
    const timestamp = req.headers['x-webhook-timestamp'] as string;
    
    if (!signature || !timestamp) {
      return res.status(401).json({ error: 'Missing signature headers' });
    }
    
    // Verify timestamp (prevent replay attacks)
    const requestTime = parseInt(timestamp);
    const currentTime = Date.now();
    const timeDiff = Math.abs(currentTime - requestTime);
    
    if (timeDiff > 5 * 60 * 1000) { // 5 minute window
      return res.status(401).json({ error: 'Request timestamp too old' });
    }
    
    // Verify signature
    const payload = JSON.stringify(req.body);
    const isValid = verifyWebhookSignature(payload, signature, secret);
    
    if (!isValid) {
      return res.status(401).json({ error: 'Invalid signature' });
    }
    
    next();
  };
}

// Usage
router.post(
  '/webhooks/paypal',
  verifyWebhook(process.env.PAYPAL_WEBHOOK_SECRET!),
  async (req, res) => {
    // Process PayPal webhook
    res.sendStatus(200);
  }
);
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Storing API keys in plain text | Hash with SHA-256 before storing |
| Long-lived JWT access tokens (>1 hour) | Use 15-minute access tokens + refresh tokens |
| `bcrypt` rounds < 10 | Use 12 rounds minimum for production |
| No rate limiting on auth endpoints | Apply strict rate limits to prevent brute force |
| Trusting `X-Forwarded-For` header | Use `req.ip` or validated proxy settings |
| Comparing secrets with `===` | Use `crypto.timingSafeEqual()` to prevent timing attacks |
| No expiry on password reset tokens | Set 15-minute expiry on reset tokens |
| Allowing weak passwords | Enforce minimum 8 characters + complexity rules |
| Not validating JWT algorithm | Specify algorithm explicitly: `jwt.verify(token, secret, { algorithms: ['HS256'] })` |
| Storing JWT in localStorage | Use httpOnly cookies for web clients |

## Integration with Other Skills

**Often combined with:**
- `api-endpoint-creation` - Protecting endpoints with auth middleware
- `error-handling-logging` - Logging failed auth attempts
- `database-schema-design` - User, ApiKey, AuditLog tables

**Depends on:**
- `database-operations` (from WordPress skills) - Database queries via Prisma

## Environment Variables Required

```bash
# JWT Secrets (generate with: openssl rand -base64 32)
JWT_ACCESS_SECRET="32-char-random-string"
JWT_REFRESH_SECRET="different-32-char-random-string"

# Redis (for rate limiting)
REDIS_HOST="localhost"
REDIS_PORT="6379"

# PayPal Webhook Secret
PAYPAL_WEBHOOK_SECRET="paypal-webhook-id"

# Node Environment
NODE_ENV="production"  # Affects cookie security
```

## Quick Reference

### Password Requirements
- Minimum length: 8 characters
- bcrypt rounds: 12
- No maximum length (bcrypt handles truncation)

### JWT Configuration
- **Access Token**: 15 minutes
- **Refresh Token**: 7 days
- **Algorithm**: HS256
- **Storage**: httpOnly cookies (web), secure storage (mobile)

### API Key Format
```
sk_live_{48 hex characters}
Example: sk_live_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6
```

### Rate Limits
| Endpoint | Limit | Window |
|----------|-------|--------|
| Global API | 100 requests | 1 minute |
| Per API key | 1000 requests | 1 hour |
| Login attempts | 5 attempts | 15 minutes |
| Registration | 3 attempts | 1 hour |

### HTTP Status Codes
- `401 Unauthorized` - Missing/invalid credentials
- `403 Forbidden` - Valid credentials, insufficient permissions
- `429 Too Many Requests` - Rate limit exceeded

## Validation Checklist

Before completing authentication work:

- [ ] All passwords hashed with bcrypt (12+ rounds)
- [ ] API keys hashed with SHA-256 before storage
- [ ] JWT access tokens expire within 15 minutes
- [ ] Refresh tokens stored in httpOnly cookies
- [ ] Rate limiting applied to all public endpoints
- [ ] Auth endpoints have strict rate limits (5/15min)
- [ ] Webhook signatures verified with HMAC-SHA256
- [ ] Timestamp validation prevents replay attacks (5min window)
- [ ] Account status checked on every auth request
- [ ] Failed login attempts logged to AuditLog table
- [ ] API key `last_used_at` updated on each request
- [ ] No sensitive data logged (passwords, tokens, keys)
- [ ] Security headers set (`Helmet.js` middleware)
- [ ] CORS configured correctly for admin panel
- [ ] Environment variables validated on startup

## Security Headers (Helmet.js)

```typescript
import helmet from 'helmet';

app.use(helmet({
  contentSecurityPolicy: {
    directives: {
      defaultSrc: ["'self'"],
      styleSrc: ["'self'", "'unsafe-inline'"],
      scriptSrc: ["'self'"],
      imgSrc: ["'self'", "data:", "https:"]
    }
  },
  hsts: {
    maxAge: 31536000,
    includeSubDomains: true,
    preload: true
  }
}));
```

## Common Security Vulnerabilities to Prevent

1. **SQL Injection**: Use Prisma ORM exclusively (no raw SQL)
2. **XSS**: Sanitize all user inputs, escape output
3. **CSRF**: Use SameSite cookies + CSRF tokens for state-changing operations
4. **Timing Attacks**: Use `crypto.timingSafeEqual()` for secret comparison
5. **Brute Force**: Rate limit auth endpoints aggressively
6. **Session Fixation**: Regenerate session tokens after login
7. **JWT Confusion**: Always specify algorithm in `jwt.verify()`
8. **Credential Stuffing**: Monitor for unusual login patterns
# Skill: Database Schema Design (Prisma + PostgreSQL)

## Identity
- **Skill ID**: `database-schema-design`
- **Domain**: Backend Database Schema & Migrations
- **Technologies**: Prisma ORM, PostgreSQL, Database Migrations
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill
- Task involves database schema design
- Creating or modifying Prisma models
- Database migrations
- Index optimization
- Files matching: `press-zone-backend/api/prisma/**/*.prisma`, migration files

## Core Patterns

### Prisma Model Definition
```prisma
model TranslationJob {
  id                  String                @id @default(uuid()) @db.Uuid
  user_id             String                @db.Uuid
  client_job_id       String?               // Optional field
  status              TranslationJobStatus  @default(pending)
  source_lang         String                @db.Char(2)
  target_lang         String                @db.Char(2)
  content             String                @db.Text
  content_hash        String                // SHA-256 for deduplication
  translation         String?               @db.Text
  tokens_used         Int                   @default(0)
  cost                Decimal               @default(0) @db.Decimal(10, 4)
  created_at          DateTime              @default(now())
  updated_at          DateTime              @updatedAt
  
  // Relations
  user                User                 @relation(fields: [user_id], references: [id], onDelete: Cascade)
  
  // Indexes for performance
  @@index([user_id, status, created_at])
  @@index([content_hash])
  @@index([created_at])
  @@map("translation_jobs")
}
```

### Enum Definitions
```prisma
enum TranslationJobStatus {
  pending
  processing
  completed
  failed
  cancelled
}

enum PlanTier {
  starter
  professional
  enterprise
}
```

### Relations & Foreign Keys
```prisma
model User {
  id          String   @id @default(uuid()) @db.Uuid
  email       String   @unique
  created_at  DateTime @default(now())
  
  // One-to-many
  api_keys             ApiKey[]
  translation_jobs     TranslationJob[]
  
  // One-to-one
  subscription         Subscription?
  
  @@map("users")
}

model ApiKey {
  id         String   @id @default(uuid()) @db.Uuid
  user_id    String   @db.Uuid
  key_hash   String   @unique
  
  // Cascade delete when user is deleted
  user User @relation(fields: [user_id], references: [id], onDelete: Cascade)
  
  @@index([user_id])
  @@map("api_keys")
}
```

### Index Strategy
```prisma
model TranslationJob {
  // ...fields...
  
  // Composite index for common queries
  @@index([user_id, status, created_at])
  
  // Individual indexes
  @@index([content_hash])
  @@index([created_at])
  @@index([client_job_id])
}
```

### Migrations
```bash
# Generate migration from schema changes
npx prisma migrate dev --name add_tone_field

# Apply migrations to production
npx prisma migrate deploy

# Generate Prisma Client
npx prisma generate

# Reset database (DEV ONLY)
npx prisma migrate reset
```

## Anti-Patterns (Forbidden)

| Mistake | Fix |
|---------|-----|
| Missing `@@map()` for tables | Always specify table name |
| No indexes on foreign keys | Add `@@index([user_id])` |
| Missing `onDelete` cascade | Specify `onDelete: Cascade` or `SetNull` |
| Using `Int` for IDs | Use `String @db.Uuid` for UUIDs |
| No `updated_at` field | Add `updated_at DateTime @updatedAt` |
| Missing unique constraints | Use `@unique` for email, keys |
| No database column types | Specify `@db.Text`, `@db.Uuid`, etc. |

## PostgreSQL Best Practices

### Data Types
| Prisma Type | PostgreSQL Type | Use For |
|-------------|-----------------|---------|
| `String @db.Uuid` | UUID | IDs |
| `String @db.Text` | TEXT | Long content |
| `String @db.VarChar(255)` | VARCHAR | Short strings |
| `Decimal @db.Decimal(10, 2)` | DECIMAL | Money |
| `Int` | INTEGER | Counts, numeric IDs |
| `DateTime` | TIMESTAMP | Dates/times |
| `Boolean` | BOOLEAN | Flags |
| `Json` | JSONB | Flexible data |

### Performance Indexes
```prisma
// Single column index
@@index([email])

// Composite index (order matters!)
@@index([user_id, status, created_at])

// Unique constraint
@unique
@@unique([user_id, site_url])

// Full-text search (raw SQL needed)
@@index([content], type: Gin)
```

## Integration with Other Skills
- **Often combined with**: `api-endpoint-creation`, `authentication-security`
- **For migrations**: Coordinate with `deployment-dockerization`
- **For queries**: Load `queue-management` for background jobs

## Quick Reference

### Common Prisma Commands
```bash
# Schema validation
npx prisma validate

# View database
npx prisma studio

# Format schema
npx prisma format

# Pull schema from existing DB
npx prisma db pull

# Push schema without migration
npx prisma db push
```

### Cascade Delete Options
| Option | Behavior |
|--------|----------|
| `Cascade` | Delete related records |
| `SetNull` | Set foreign key to NULL |
| `Restrict` | Prevent deletion |
| `NoAction` | Database default |

## Validation Checklist
- [ ] All models have `@@map("table_name")`
- [ ] IDs use `@default(uuid()) @db.Uuid`
- [ ] Foreign keys have `onDelete` behavior
- [ ] Indexes on foreign keys and query fields
- [ ] `@unique` on emails, API keys
- [ ] `updated_at` fields use `@updatedAt`
- [ ] Enums defined for fixed values
- [ ] Column types specified (`@db.Text`, `@db.Uuid`)
- [ ] Relations properly defined both sides
# Skill: Deployment & Dockerization

## Identity
- **Skill ID**: `deployment-dockerization`
- **Domain**: DevOps, Containerization, CI/CD
- **Technologies**: Docker, Docker Compose, Kubernetes, GitHub Actions
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Docker containerization
- Docker Compose orchestration
- Production deployment
- CI/CD pipeline setup
- Kubernetes deployment
- Environment configuration
- Health checks and monitoring

**File patterns:**
- `Dockerfile`
- `docker-compose.yml`
- `.github/workflows/*.yml`
- `k8s/**/*.yaml`

## Core Patterns

### 1. API Server Dockerfile

```dockerfile
# api/Dockerfile
FROM node:20-alpine AS base

# Install dependencies only when needed
FROM base AS deps
WORKDIR /app

# Copy package files
COPY package.json package-lock.json ./
RUN npm ci --only=production

# Build stage
FROM base AS builder
WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci

COPY . .
COPY prisma ./prisma

# Generate Prisma client
RUN npx prisma generate

# Build TypeScript
RUN npm run build

# Production image
FROM base AS runner
WORKDIR /app

ENV NODE_ENV=production

# Create non-root user
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nodejs

# Copy necessary files
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/prisma ./prisma
COPY package.json ./

USER nodejs

EXPOSE 3000

CMD ["node", "dist/index.js"]
```

### 2. Worker Dockerfile

```dockerfile
# worker/Dockerfile
FROM node:20-alpine

WORKDIR /app

# Install dependencies
COPY package.json package-lock.json ./
RUN npm ci --only=production

# Copy application code
COPY dist ./dist
COPY prisma ./prisma

# Generate Prisma client
RUN npx prisma generate

ENV NODE_ENV=production

CMD ["node", "dist/worker.js"]
```

### 3. Docker Compose (Development)

```yaml
# docker-compose.yml
version: '3.8'

services:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: translatepresszone
      POSTGRES_USER: tpz
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U tpz"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5

  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://tpz:password@postgres:5432/translatepresszone
      REDIS_HOST: redis
      REDIS_PORT: 6379
      NODE_ENV: development
    env_file:
      - .env
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    volumes:
      - ./api/src:/app/src  # Hot reload in dev
    command: npm run dev

  worker:
    build:
      context: ./api
      dockerfile: Dockerfile
    environment:
      DATABASE_URL: postgresql://tpz:password@postgres:5432/translatepresszone
      REDIS_HOST: redis
      REDIS_PORT: 6379
      NODE_ENV: development
    env_file:
      - .env
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    command: npm run worker

  admin:
    build:
      context: ./admin-panel
      dockerfile: Dockerfile
    ports:
      - "5173:5173"
    environment:
      VITE_API_BASE_URL: http://localhost:3000
    volumes:
      - ./admin-panel/src:/app/src  # Hot reload
    command: npm run dev

volumes:
  postgres_data:
  redis_data:
```

### 4. Docker Compose (Production)

```yaml
# docker-compose.prod.yml
version: '3.8'

services:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./backups:/backups
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    restart: unless-stopped

  api:
    image: translatepresszone/api:latest
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
      REDIS_HOST: redis
      NODE_ENV: production
    env_file:
      - .env.production
    depends_on:
      - postgres
      - redis
    restart: unless-stopped
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1'
          memory: 1G

  worker:
    image: translatepresszone/worker:latest
    environment:
      DATABASE_URL: postgresql://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
      REDIS_HOST: redis
      NODE_ENV: production
    env_file:
      - .env.production
    depends_on:
      - postgres
      - redis
    restart: unless-stopped
    deploy:
      replicas: 3

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/ssl:/etc/nginx/ssl
      - admin_dist:/usr/share/nginx/html
    depends_on:
      - api
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:
  admin_dist:
```

### 5. Nginx Configuration

```nginx
# nginx/nginx.conf
upstream api {
    least_conn;
    server api:3000 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name api.translatepresszone.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.translatepresszone.com;
    
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;
    
    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    
    # API proxy
    location /api {
        proxy_pass http://api;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
    
    # Rate limiting
    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
    limit_req zone=api_limit burst=20 nodelay;
}

# Admin panel
server {
    listen 443 ssl http2;
    server_name admin.translatepresszone.com;
    
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;
    
    root /usr/share/nginx/html;
    index index.html;
    
    location / {
        try_files $uri $uri/ /index.html;
    }
    
    # Cache static assets
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}
```

### 6. Kubernetes Deployment

```yaml
# k8s/api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: translatepresszone
    component: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: translatepresszone
      component: api
  template:
    metadata:
      labels:
        app: translatepresszone
        component: api
    spec:
      containers:
      - name: api
        image: translatepresszone/api:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
        - name: REDIS_HOST
          value: "redis-service"
        envFrom:
        - secretRef:
            name: api-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  selector:
    app: translatepresszone
    component: api
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000
  type: LoadBalancer
```

### 7. GitHub Actions CI/CD

```yaml
# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '20'
        cache: 'npm'
    
    - name: Install dependencies
      run: npm ci
    
    - name: Run tests
      run: npm test
      env:
        DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test

  build-and-push:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Log in to Container Registry
      uses: docker/login-action@v2
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    
    - name: Build and push Docker image
      uses: docker/build-push-action@v4
      with:
        context: ./api
        push: true
        tags: |
          ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/api:latest
          ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}/api:${{ github.sha }}

  deploy:
    needs: build-and-push
    runs-on: ubuntu-latest
    
    steps:
    - name: Deploy to production
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.DEPLOY_HOST }}
        username: ${{ secrets.DEPLOY_USER }}
        key: ${{ secrets.DEPLOY_KEY }}
        script: |
          cd /app/translatepresszone
          docker-compose -f docker-compose.prod.yml pull
          docker-compose -f docker-compose.prod.yml up -d
          docker system prune -f
```

### 8. Database Migrations (Production)

```bash
# scripts/migrate.sh
#!/bin/bash
set -e

echo "Running database migrations..."

# Run migrations
npx prisma migrate deploy

echo "Migrations completed successfully"
```

### 9. Health Check Endpoints

```typescript
// routes/health.ts
import { Router } from 'express';
import { prisma } from '../lib/prisma';
import Redis from 'ioredis';

const router = Router();

// Basic health check
router.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

// Detailed health check
router.get('/health/detailed', async (req, res) => {
  const health = {
    status: 'ok',
    timestamp: new Date().toISOString(),
    services: {
      database: 'unknown',
      redis: 'unknown'
    }
  };
  
  // Check database
  try {
    await prisma.$queryRaw`SELECT 1`;
    health.services.database = 'healthy';
  } catch (error) {
    health.services.database = 'unhealthy';
    health.status = 'degraded';
  }
  
  // Check Redis
  try {
    const redis = new Redis({
      host: process.env.REDIS_HOST,
      port: parseInt(process.env.REDIS_PORT || '6379')
    });
    await redis.ping();
    redis.disconnect();
    health.services.redis = 'healthy';
  } catch (error) {
    health.services.redis = 'unhealthy';
    health.status = 'degraded';
  }
  
  const statusCode = health.status === 'ok' ? 200 : 503;
  res.status(statusCode).json(health);
});

export default router;
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Running as root in containers | Create non-root user |
| Hardcoding secrets in Dockerfile | Use environment variables |
| No health checks | Implement `/health` endpoint |
| Not using multi-stage builds | Use multi-stage for smaller images |
| Storing data in containers | Use volumes for persistence |
| No resource limits | Set CPU/memory limits |
| Not handling graceful shutdown | Listen for SIGTERM |
| Building on every start | Build once, deploy image |
| No CI/CD pipeline | Automate testing and deployment |
| Missing database backups | Implement backup strategy |

## Integration with Other Skills

**Often combined with:**
- All skills (deployment affects entire system)
- `error-handling-logging` - Centralized logging in production
- `authentication-security` - Secret management

## Environment Variables Required

```bash
# .env.production
NODE_ENV=production
PORT=3000

# Database
DATABASE_URL=postgresql://user:pass@postgres:5432/db

# Redis
REDIS_HOST=redis
REDIS_PORT=6379

# JWT
JWT_ACCESS_SECRET=xxx
JWT_REFRESH_SECRET=xxx

# External Services
GEMINI_API_KEY=xxx
GEMINI_MODEL=gemini-3-flash-preview
PAYPAL_CLIENT_ID=xxx
PAYPAL_CLIENT_SECRET=xxx
```

## Quick Reference

### Docker Commands

```bash
# Build image
docker build -t translatepresszone/api:latest ./api

# Run container
docker run -p 3000:3000 --env-file .env translatepresszone/api:latest

# View logs
docker logs -f container_id

# Execute command in container
docker exec -it container_id sh
```

### Docker Compose Commands

```bash
# Start all services
docker-compose up -d

# View logs
docker-compose logs -f api

# Stop services
docker-compose down

# Rebuild specific service
docker-compose up -d --build api
```

### Kubernetes Commands

```bash
# Apply configuration
kubectl apply -f k8s/

# View pods
kubectl get pods

# View logs
kubectl logs pod-name

# Scale deployment
kubectl scale deployment api --replicas=5

# Rollback deployment
kubectl rollout undo deployment api
```

## Validation Checklist

- [ ] Dockerfile uses multi-stage build
- [ ] Non-root user created in Dockerfile
- [ ] `.dockerignore` file present
- [ ] Health check endpoint implemented
- [ ] Database migrations automated
- [ ] Environment variables externalized
- [ ] Secrets managed securely (not in code)
- [ ] Resource limits set (CPU, memory)
- [ ] Graceful shutdown implemented
- [ ] Logging to stdout (not files in containers)
- [ ] Docker Compose for local development
- [ ] Production compose file separate
- [ ] Nginx configured with SSL
- [ ] CI/CD pipeline automated
- [ ] Database backups scheduled

## Production Deployment Checklist

- [ ] SSL certificates configured
- [ ] Domain DNS configured
- [ ] Load balancer setup
- [ ] Database backups automated
- [ ] Monitoring and alerting configured
- [ ] Log aggregation setup
- [ ] Security headers configured
- [ ] Rate limiting enabled
- [ ] Auto-scaling configured
- [ ] Disaster recovery plan documented
# Skill: Error Handling & Logging

## Identity
- **Skill ID**: `error-handling-logging`
- **Domain**: Error Management, Structured Logging, Monitoring
- **Technologies**: Winston, Error Classes, Audit Logging
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Winston logger configuration
- Custom error classes
- Error middleware
- Audit log implementation
- Monitoring and alerting
- Debug logging
- Production error tracking

**File patterns:**
- `api/src/middleware/error*.ts`
- `api/src/utils/logger*.ts`
- `api/src/utils/errors*.ts`

## Core Patterns

### 1. Winston Logger Setup

```typescript
// utils/logger.ts
import winston from 'winston';
import path from 'path';

// Custom log format
const logFormat = winston.format.combine(
  winston.format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss' }),
  winston.format.errors({ stack: true }),
  winston.format.splat(),
  winston.format.json()
);

// Console format (development)
const consoleFormat = winston.format.combine(
  winston.format.colorize(),
  winston.format.timestamp({ format: 'HH:mm:ss' }),
  winston.format.printf(({ timestamp, level, message, ...meta }) => {
    let msg = `${timestamp} [${level}]: ${message}`;
    if (Object.keys(meta).length > 0) {
      msg += ` ${JSON.stringify(meta)}`;
    }
    return msg;
  })
);

// Create logger instance
export const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: logFormat,
  defaultMeta: { service: 'translate-press-zone-api' },
  transports: [
    // Error logs
    new winston.transports.File({
      filename: path.join('logs', 'error.log'),
      level: 'error',
      maxsize: 5242880, // 5MB
      maxFiles: 5
    }),
    
    // Combined logs
    new winston.transports.File({
      filename: path.join('logs', 'combined.log'),
      maxsize: 5242880,
      maxFiles: 5
    })
  ]
});

// Console transport for development
if (process.env.NODE_ENV !== 'production') {
  logger.add(new winston.transports.Console({
    format: consoleFormat
  }));
}

// Stream for Morgan HTTP logging
export const httpLogStream = {
  write: (message: string) => {
    logger.http(message.trim());
  }
};
```

### 2. Custom Error Classes

```typescript
// utils/errors.ts

// Base API Error
export class ApiError extends Error {
  constructor(
    public statusCode: number,
    message: string,
    public isOperational: boolean = true
  ) {
    super(message);
    this.name = this.constructor.name;
    Error.captureStackTrace(this, this.constructor);
  }
}

// Specific error types
export class BadRequestError extends ApiError {
  constructor(message: string = 'Bad request') {
    super(400, message);
  }
}

export class UnauthorizedError extends ApiError {
  constructor(message: string = 'Unauthorized') {
    super(401, message);
  }
}

export class ForbiddenError extends ApiError {
  constructor(message: string = 'Forbidden') {
    super(403, message);
  }
}

export class NotFoundError extends ApiError {
  constructor(message: string = 'Resource not found') {
    super(404, message);
  }
}

export class ConflictError extends ApiError {
  constructor(message: string = 'Resource conflict') {
    super(409, message);
  }
}

export class PaymentRequiredError extends ApiError {
  constructor(message: string = 'Insufficient credits') {
    super(402, message);
  }
}

export class TooManyRequestsError extends ApiError {
  constructor(message: string = 'Rate limit exceeded') {
    super(429, message);
  }
}

export class InternalServerError extends ApiError {
  constructor(message: string = 'Internal server error') {
    super(500, message, false); // Not operational
  }
}

// Validation error with field details
export class ValidationError extends ApiError {
  constructor(
    public fields: Record<string, string>,
    message: string = 'Validation failed'
  ) {
    super(400, message);
  }
}
```

### 3. Error Handling Middleware

```typescript
// middleware/errorHandler.ts
import { Request, Response, NextFunction } from 'express';
import { ApiError } from '../utils/errors';
import { logger } from '../utils/logger';

export function errorHandler(
  err: Error,
  req: Request,
  res: Response,
  next: NextFunction
) {
  // Default to 500 server error
  let statusCode = 500;
  let message = 'Internal server error';
  let isOperational = false;
  
  if (err instanceof ApiError) {
    statusCode = err.statusCode;
    message = err.message;
    isOperational = err.isOperational;
  }
  
  // Log error
  const logData = {
    error: {
      message: err.message,
      stack: err.stack,
      statusCode
    },
    request: {
      method: req.method,
      url: req.url,
      ip: req.ip,
      userId: (req as any).user?.userId
    }
  };
  
  if (isOperational) {
    logger.warn('Operational error', logData);
  } else {
    logger.error('Non-operational error', logData);
    
    // In production, don't expose internal errors
    if (process.env.NODE_ENV === 'production') {
      message = 'An unexpected error occurred';
    }
  }
  
  // Send error response
  res.status(statusCode).json({
    error: message,
    ...(err instanceof ValidationError && { fields: err.fields }),
    ...(process.env.NODE_ENV !== 'production' && { stack: err.stack })
  });
}

// 404 handler
export function notFoundHandler(req: Request, res: Response) {
  res.status(404).json({
    error: 'Endpoint not found',
    path: req.path
  });
}

// Async error wrapper
export function asyncHandler(
  fn: (req: Request, res: Response, next: NextFunction) => Promise<any>
) {
  return (req: Request, res: Response, next: NextFunction) => {
    Promise.resolve(fn(req, res, next)).catch(next);
  };
}
```

### 4. HTTP Request Logging

```typescript
// middleware/httpLogger.ts
import morgan from 'morgan';
import { httpLogStream } from '../utils/logger';

// Custom Morgan format
morgan.token('user-id', (req: any) => req.user?.userId || 'anonymous');

export const httpLogger = morgan(
  ':method :url :status :response-time ms - :user-id',
  { stream: httpLogStream }
);

// Skip logging health checks
export const httpLoggerWithSkip = morgan(
  ':method :url :status :response-time ms - :user-id',
  {
    stream: httpLogStream,
    skip: (req) => req.url === '/health'
  }
);
```

### 5. Audit Logging

```typescript
// utils/auditLog.ts
import { prisma } from '../lib/prisma';
import { Request } from 'express';

interface AuditLogData {
  userId?: string;
  action: string;
  resourceType?: string;
  resourceId?: string;
  details?: any;
  req: Request;
}

export async function createAuditLog(data: AuditLogData): Promise<void> {
  try {
    await prisma.auditLog.create({
      data: {
        user_id: data.userId || null,
        action: data.action,
        resource_type: data.resourceType || null,
        resource_id: data.resourceId || null,
        ip_address: data.req.ip,
        user_agent: data.req.headers['user-agent'] || null,
        details: data.details || null
      }
    });
  } catch (error) {
    // Don't let audit log failure break the request
    logger.error('Failed to create audit log', { error, data });
  }
}

// Middleware to automatically audit specific actions
export function auditAction(action: string, resourceType?: string) {
  return async (req: any, res: Response, next: NextFunction) => {
    // Store original send function
    const originalSend = res.send;
    
    // Override send to capture success
    res.send = function (data: any) {
      if (res.statusCode >= 200 && res.statusCode < 300) {
        createAuditLog({
          userId: req.user?.userId,
          action,
          resourceType,
          resourceId: req.params.id,
          details: { method: req.method, body: req.body },
          req
        });
      }
      
      return originalSend.call(this, data);
    };
    
    next();
  };
}
```

### 6. Usage in Routes

```typescript
// Example: User management endpoint
import { asyncHandler } from '../middleware/errorHandler';
import { NotFoundError, BadRequestError } from '../utils/errors';
import { createAuditLog } from '../utils/auditLog';
import { logger } from '../utils/logger';

router.get('/users/:id', requireAuth, asyncHandler(async (req, res) => {
  const { id } = req.params;
  
  logger.info('Fetching user', { userId: id });
  
  const user = await prisma.user.findUnique({
    where: { id }
  });
  
  if (!user) {
    throw new NotFoundError('User not found');
  }
  
  res.json(user);
}));

router.put('/users/:id', requireAuth, asyncHandler(async (req, res) => {
  const { id } = req.params;
  const { status } = req.body;
  
  if (!['active', 'suspended'].includes(status)) {
    throw new BadRequestError('Invalid status value');
  }
  
  const user = await prisma.user.update({
    where: { id },
    data: { status }
  });
  
  // Audit log
  await createAuditLog({
    userId: (req as any).user.userId,
    action: 'user.update',
    resourceType: 'user',
    resourceId: id,
    details: { status },
    req
  });
  
  logger.info('User updated', { userId: id, status });
  
  res.json(user);
}));
```

### 7. Application Setup

```typescript
// app.ts
import express from 'express';
import { httpLoggerWithSkip } from './middleware/httpLogger';
import { errorHandler, notFoundHandler } from './middleware/errorHandler';
import { logger } from './utils/logger';

const app = express();

// Middleware
app.use(express.json());
app.use(httpLoggerWithSkip);

// Routes
app.use('/api', apiRoutes);

// Health check
app.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});

// 404 handler
app.use(notFoundHandler);

// Error handler (must be last)
app.use(errorHandler);

// Graceful shutdown
process.on('SIGTERM', () => {
  logger.info('SIGTERM received, shutting down gracefully');
  server.close(() => {
    logger.info('Server closed');
    process.exit(0);
  });
});

// Unhandled rejection
process.on('unhandledRejection', (reason, promise) => {
  logger.error('Unhandled Rejection', { reason, promise });
  process.exit(1);
});

// Uncaught exception
process.on('uncaughtException', (error) => {
  logger.error('Uncaught Exception', { error });
  process.exit(1);
});

export default app;
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Using `console.log()` in production | Use Winston logger |
| Not catching async errors | Wrap with `asyncHandler()` |
| Exposing error stack traces in production | Hide in production, show in development |
| Generic "Error occurred" messages | Provide specific, actionable error messages |
| Not logging request context | Include user ID, IP, request details |
| Throwing strings instead of Error objects | Always throw Error instances |
| Not distinguishing operational vs programmer errors | Use `isOperational` flag |
| Logging sensitive data (passwords, tokens) | Sanitize logs |
| Not setting log rotation | Use maxFiles and maxSize |
| Letting audit log failures break requests | Catch and log audit errors separately |

## Integration with Other Skills

**Often combined with:**
- All skills (logging is cross-cutting)
- `authentication-security` - Log auth failures
- `payment-integration` - Audit payment events
- `api-endpoint-creation` - Error responses

## Environment Variables Required

```bash
# Logging Configuration
LOG_LEVEL="info"  # debug, info, warn, error
NODE_ENV="production"  # affects error verbosity
```

## Quick Reference

### Log Levels

| Level | When to Use |
|-------|-------------|
| `error` | Application errors, exceptions |
| `warn` | Operational errors (bad requests, auth failures) |
| `info` | Important business events (user created, payment completed) |
| `http` | HTTP requests/responses |
| `debug` | Detailed debugging information |

### Common Log Patterns

```typescript
// Info log
logger.info('User logged in', { userId: user.id, email: user.email });

// Warning log
logger.warn('Invalid API key attempt', { key: keyPrefix, ip: req.ip });

// Error log
logger.error('Database connection failed', { error: err.message, stack: err.stack });

// Debug log
logger.debug('Processing translation job', { jobId, tokens: estimatedTokens });
```

## Validation Checklist

- [ ] Winston logger configured with file transports
- [ ] Log rotation enabled (maxFiles, maxSize)
- [ ] Custom error classes created
- [ ] Error handler middleware implemented
- [ ] `asyncHandler` wraps all async routes
- [ ] HTTP request logging enabled
- [ ] Audit log for sensitive actions
- [ ] No `console.log()` in production code
- [ ] Error messages user-friendly
- [ ] Stack traces hidden in production
- [ ] Sensitive data (passwords, tokens) not logged
- [ ] Unhandled rejection handler registered
- [ ] Graceful shutdown implemented
- [ ] Log aggregation setup (optional: ELK, CloudWatch)

## Production Monitoring

### Recommended Services

- **Error Tracking**: Sentry, Rollbar
- **Log Aggregation**: ELK Stack, CloudWatch, Datadog
- **APM**: New Relic, AppDynamics

### Sentry Integration

```typescript
import * as Sentry from '@sentry/node';

if (process.env.NODE_ENV === 'production') {
  Sentry.init({
    dsn: process.env.SENTRY_DSN,
    environment: process.env.NODE_ENV,
    tracesSampleRate: 0.1
  });
  
  app.use(Sentry.Handlers.requestHandler());
  app.use(Sentry.Handlers.errorHandler());
}
```
# Skill: Translation Service Integration (Google Gemini API)

## Identity
- **Skill ID**: `ml-service-integration`
- **Domain**: AI Translation, Google Gemini API
- **Technologies**: Google Gemini API, TypeScript, Node.js
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Google Gemini API client implementation
- Translation service integration
- HTML content preservation during translation
- Token counting and billing calculation
- Translation quality optimization
- Error handling and retries
- Dynamic configuration management

**File patterns:**
- `api/src/services/gemini*.ts`
- `api/src/services/translation*.ts`
- `api/src/config/index.ts`

## Core Patterns
## Core Patterns

### 1. Gemini Client Setup

```typescript
// api/src/services/geminiClient.ts
import { GoogleGenerativeAI, GenerativeModel } from '@google/generative-ai';
import { getGeminiConfig, onConfigChange } from '../config';
import { logger } from '../utils/logger';

class GeminiClient {
  private client: GoogleGenerativeAI | null = null;
  private model: GenerativeModel | null = null;
  private readonly timeout = 60000; // 60 seconds

  constructor() {
    this.initializeClient();
    
    // Subscribe to config changes
    onConfigChange('gemini', () => {
      this.refreshConfig();
    });
  }

  private initializeClient(): void {
    const geminiConfig = getGeminiConfig();
    this.client = new GoogleGenerativeAI(geminiConfig.apiKey);
    this.model = this.client.getGenerativeModel({ 
      model: 'gemini-3-flash-preview' 
    });
    
    logger.info('Gemini client initialized');
  }

  refreshConfig(): void {
    this.initializeClient();
  }
}

export const geminiClient = new GeminiClient();
```

### 2. HTML Content Preservation

```typescript
// Extract and restore HTML tags
private extractHtmlTags(content: string): { 
  text: string; 
  tags: Map<string, string> 
} {
  const tags = new Map<string, string>();
  let counter = 0;

  const htmlTagRegex = /<\/?[a-zA-Z][a-zA-Z0-9-]*(?:\s+[^>]*)?>/g;

  const text = content.replace(htmlTagRegex, (match) => {
    const placeholder = `__TAG_${counter}__`;
    tags.set(placeholder, match);
    counter++;
    return placeholder;
  });

  return { text, tags };
}

private restoreHtmlTags(text: string, tags: Map<string, string>): string {
  let result = text;
  tags.forEach((tag, placeholder) => {
    result = result.replace(placeholder, tag);
  });
  return result;
}
```

### 3. Translation Implementation

```typescript
async translate(
  content: string,
  sourceLang: string,
  targetLang: string,
  model: Model,
  tone: Tone = Tone.NEUTRAL
): Promise<GeminiTranslationResponse> {
  if (!this.client || !this.model) {
    throw new Error('Gemini client not initialized');
  }

  const startTime = Date.now();

  // Extract HTML tags
  const { text: cleanText, tags } = this.extractHtmlTags(content);

  // Build translation prompt
  const prompt = `Translate the following text from ${sourceLang} to ${targetLang}.

${this.getToneInstruction(tone)}

IMPORTANT RULES:
1. Preserve all placeholders in the format __TAG_N__ exactly as they appear
2. Do not translate the placeholders themselves
3. Only translate the actual text content
4. Maintain the exact position and format of placeholders
5. Output ONLY the translated text

Text to translate:
${cleanText}`;

  // Count input tokens
  const tokenCountResult = await this.model.countTokens(prompt);
  const inputTokens = tokenCountResult.totalTokens;

  // Generate translation with timeout
  const result = await Promise.race([
    this.model.generateContent(prompt),
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('Translation timeout')), this.timeout)
    ),
  ]);

  const translatedText = result.response.text();
  
  // Restore HTML tags
  const finalTranslation = this.restoreHtmlTags(translatedText, tags);

  // Count output tokens
  const outputTokenCountResult = await this.model.countTokens(translatedText);
  const outputTokens = outputTokenCountResult.totalTokens;

  const totalTokens = inputTokens + outputTokens;
  const processingTime = Date.now() - startTime;

  return {
    translation: finalTranslation,
    tokens_used: totalTokens,
    processing_time_ms: processingTime,
    model,
  };
}
```

### 4. Tone Instructions

```typescript
private getToneInstruction(tone: Tone): string {
  const toneInstructions = {
    [Tone.FORMAL]: 'Use formal, professional language.',
    [Tone.CASUAL]: 'Use casual, conversational language.',
    [Tone.NEUTRAL]: 'Use neutral, standard language.',
  };

  return toneInstructions[tone] || toneInstructions[Tone.NEUTRAL];
}
```

### 5. Language Name Mapping

```typescript
private getLanguageName(code: string): string {
  const languageMap: Record<string, string> = {
    'en': 'English',
    'es': 'Spanish',
    'fr': 'French',
    'de': 'German',
    'it': 'Italian',
    'pt': 'Portuguese',
    'nl': 'Dutch',
    'pl': 'Polish',
    'ru': 'Russian',
    'ja': 'Japanese',
    'ko': 'Korean',
    'zh': 'Chinese',
    'ar': 'Arabic',
    'hi': 'Hindi',
    'tr': 'Turkish',
    'vi': 'Vietnamese',
    'th': 'Thai',
    'id': 'Indonesian',
  };

  return languageMap[code] || code.toUpperCase();
}
```

### 6. Health Check

```typescript
async healthCheck(): Promise<boolean> {
  try {
    if (!this.client || !this.model) {
      return false;
    }

    const testPrompt = 'Translate "hello" to Spanish';
    const result = await Promise.race([
      this.model.generateContent(testPrompt),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('Health check timeout')), 5000)
      ),
    ]);

    return !!result.response.text();
  } catch (error) {
    logger.warn('Gemini health check failed', { error });
    return false;
  }
}
```

### 7. Token Estimation

```typescript
// api/src/utils/tokenCalculation.ts
export function estimateTokens(content: string): number {
  // Strip HTML for more accurate estimate
  const plainText = content.replace(/<[^>]*>/g, ' ');
  
  // Rough estimation: 1 token ≈ 4 characters
  const charCount = plainText.length;
  const estimatedInputTokens = Math.ceil(charCount / 4);
  
  // Output typically 1.2-1.5x input for translation
  const estimatedTotal = Math.ceil(estimatedInputTokens * 1.5);
  
  return estimatedTotal;
}
```

### 8. Dynamic Configuration

```typescript
// api/src/config/index.ts
import { prisma } from '../db';
import { EventEmitter } from 'events';

const configEmitter = new EventEmitter();

interface GeminiConfig {
  apiKey: string;
  model: string;
}

let cachedGeminiConfig: GeminiConfig | null = null;

export function getGeminiConfig(): GeminiConfig {
  if (cachedGeminiConfig) {
    return cachedGeminiConfig;
  }

  // Load from database or environment
  const apiKey = process.env.GEMINI_API_KEY;
  const model = process.env.GEMINI_MODEL || 'gemini-3-flash-preview';

  if (!apiKey) {
    throw new Error('GEMINI_API_KEY not configured');
  }

  cachedGeminiConfig = { apiKey, model };
  return cachedGeminiConfig;
}

export function refreshGeminiConfig(): void {
  cachedGeminiConfig = null;
  configEmitter.emit('gemini');
}

export function onConfigChange(
  event: 'gemini' | 'all',
  callback: () => void
): void {
  configEmitter.on(event, callback);
}
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Not preserving HTML structure | Extract and restore HTML tags with placeholders |
| Hardcoding API key in code | Use environment variables and dynamic config |
| Not handling timeouts | Set appropriate timeouts (60s recommended) |
| Not counting tokens accurately | Count both input and output tokens |
| Ignoring API errors | Handle specific error cases with user-friendly messages |
| Not validating language codes | Validate against supported languages |
| Sending unescaped HTML directly | Use placeholder system for HTML preservation |
| Exposing API keys in logs | Sanitize sensitive data before logging |
| Not implementing health checks | Add health check endpoint for monitoring |
| Blocking on translation | Use async/await properly with timeouts |

## Integration with Other Skills

**Often combined with:**
- `queue-management` - Async translation processing
- `api-endpoint-creation` - Translation endpoints
- `webhook-implementation` - Delivery of results

**Depends on:**
- `database-schema-design` - TranslationJob table
- `error-handling-logging` - Proper error handling

## Environment Variables Required

```bash
# Google Gemini API Configuration
GEMINI_API_KEY="your-google-gemini-api-key"
GEMINI_MODEL="gemini-3-flash-preview"
```

## Quick Reference

### Supported Languages

```typescript
const SUPPORTED_LANGUAGES = {
  'en': 'English',
  'es': 'Spanish',
  'fr': 'French',
  'de': 'German',
  'it': 'Italian',
  'pt': 'Portuguese',
  'ja': 'Japanese',
  'zh': 'Chinese',
  'ar': 'Arabic',
  'ru': 'Russian',
  'ko': 'Korean',
  'nl': 'Dutch',
  'pl': 'Polish',
  'tr': 'Turkish',
  'vi': 'Vietnamese',
  'th': 'Thai',
  'id': 'Indonesian',
  'hi': 'Hindi',
};
```

### Response Times

| Content Size | Expected Time |
|--------------|---------------|
| Short (< 100 words) | ~1-2s |
| Medium (100-500 words) | ~2-4s |
| Long (500-2000 words) | ~4-8s |

### Token Pricing

- **Gemini 3 Flash Preview**: ~$0.002 per 1K tokens (combined input + output)

### Typical Token Counts

| Content Type | Tokens (estimate) |
|--------------|-------------------|
| Short paragraph (100 words) | ~150 tokens |
| Blog post (500 words) | ~750 tokens |
| Long article (2000 words) | ~3000 tokens |

## Validation Checklist

- [ ] Gemini API key configured in environment
- [ ] geminiClient properly initialized
- [ ] HTML structure preservation tested
- [ ] Token counting accurate
- [ ] Timeout handling implemented
- [ ] Language validation in place
- [ ] Tone parameter working (neutral/formal/casual)
- [ ] Error responses properly formatted
- [ ] Health check endpoint functional
- [ ] Dynamic config refresh working
- [ ] All supported languages tested
- [ ] Cost calculation accurate

## Error Handling

### Common Error Cases

```typescript
try {
  const translation = await geminiClient.translate(...);
} catch (error) {
  if (error.message.includes('timeout')) {
    // Handle timeout - suggest async translation
    throw new Error('Translation timed out. Please use async translation.');
  } else if (error.message.includes('API key')) {
    // Handle auth error
    throw new Error('Gemini API authentication failed.');
  } else if (error.message.includes('quota') || error.message.includes('rate limit')) {
    // Handle rate limit
    throw new Error('API rate limit exceeded. Please try again later.');
  } else {
    // Generic error
    throw new Error(`Translation failed: ${error.message}`);
  }
}
```

## Performance Optimization

### Caching Strategy

```typescript
// Cache translations for identical content
import NodeCache from 'node-cache';

const translationCache = new NodeCache({ 
  stdTTL: 3600, // 1 hour
  maxKeys: 10000 
});

const cacheKey = `${sourceLang}:${targetLang}:${contentHash}`;
const cached = translationCache.get(cacheKey);

if (cached) {
  return cached;
}

// Perform translation...
translationCache.set(cacheKey, result);
```

### Batch Processing

```typescript
// Process multiple translations efficiently
async function batchTranslate(requests: TranslationRequest[]): Promise<TranslationResult[]> {
  const results = await Promise.all(
    requests.map(req => 
      geminiClient.translate(
        req.content,
        req.sourceLang,
        req.targetLang,
        req.model,
        req.tone
      )
    )
  );
  
  return results;
}
```

## Monitoring

### Log Translation Metrics

```typescript
logger.info('Gemini translation successful', {
  inputTokens,
  outputTokens,
  totalTokens,
  processingTimeMs: processingTime,
  sourceLang,
  targetLang,
  contentLength: content.length,
});
```

### Track API Usage

```typescript
// Track token usage for billing
await prisma.translationJob.update({
  where: { id: jobId },
  data: {
    tokens_used: totalTokens,
    processing_time_ms: processingTime,
    status: 'completed',
  },
});
```

# Skill: PayPal Payment Integration

## Identity
- **Skill ID**: `payment-integration`
- **Domain**: Payment Processing, Subscription Management
- **Technologies**: PayPal Subscriptions API, Webhooks
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- PayPal subscription creation
- Payment webhook processing
- Subscription plan management
- Billing cycle handling
- Payment history tracking
- Refund processing
- Dispute handling

**File patterns:**
- `api/src/services/paypal*.ts`
- `api/src/routes/subscriptions*.ts`
- `api/src/routes/payments*.ts`
- `api/src/routes/webhooks/paypal*.ts`

## Core Patterns

### 1. PayPal SDK Setup

```typescript
import axios from 'axios';

// PayPal configuration
const PAYPAL_BASE_URL = process.env.PAYPAL_MODE === 'live'
  ? 'https://api.paypal.com'
  : 'https://api.sandbox.paypal.com';

const PAYPAL_CLIENT_ID = process.env.PAYPAL_CLIENT_ID!;
const PAYPAL_CLIENT_SECRET = process.env.PAYPAL_CLIENT_SECRET!;

// Get access token
async function getPayPalAccessToken(): Promise<string> {
  const auth = Buffer.from(`${PAYPAL_CLIENT_ID}:${PAYPAL_CLIENT_SECRET}`).toString('base64');
  
  const response = await axios.post(
    `${PAYPAL_BASE_URL}/v1/oauth2/token`,
    'grant_type=client_credentials',
    {
      headers: {
        'Authorization': `Basic ${auth}`,
        'Content-Type': 'application/x-www-form-urlencoded'
      }
    }
  );
  
  return response.data.access_token;
}

// Cached token with refresh
let cachedToken: { token: string; expiresAt: number } | null = null;

async function getAccessToken(): Promise<string> {
  const now = Date.now();
  
  if (cachedToken && cachedToken.expiresAt > now) {
    return cachedToken.token;
  }
  
  const token = await getPayPalAccessToken();
  cachedToken = {
    token,
    expiresAt: now + 3600000 // 1 hour
  };
  
  return token;
}
```

### 2. Create Subscription Plans (One-Time Setup)

```typescript
// Subscription plan configuration
interface PlanConfig {
  id: string;
  name: string;
  price: string;
  credits: number;
  billingCycle: 'monthly' | 'annual';
}

const PLANS: PlanConfig[] = [
  {
    id: 'starter-monthly',
    name: 'Starter Plan',
    price: '9.00',
    credits: 100000,
    billingCycle: 'monthly'
  },
  {
    id: 'professional-monthly',
    name: 'Professional Plan',
    price: '29.00',
    credits: 500000,
    billingCycle: 'monthly'
  },
  {
    id: 'enterprise-monthly',
    name: 'Enterprise Plan',
    price: '99.00',
    credits: 2000000,
    billingCycle: 'monthly'
  }
];

// Create plan in PayPal (run once during setup)
async function createPayPalPlan(config: PlanConfig): Promise<string> {
  const token = await getAccessToken();
  
  const productResponse = await axios.post(
    `${PAYPAL_BASE_URL}/v1/catalogs/products`,
    {
      name: config.name,
      description: `${config.name} - ${config.credits.toLocaleString()} tokens/month`,
      type: 'SERVICE',
      category: 'SOFTWARE'
    },
    {
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      }
    }
  );
  
  const productId = productResponse.data.id;
  
  const planResponse = await axios.post(
    `${PAYPAL_BASE_URL}/v1/billing/plans`,
    {
      product_id: productId,
      name: config.name,
      description: `${config.name} subscription`,
      billing_cycles: [
        {
          frequency: {
            interval_unit: config.billingCycle === 'monthly' ? 'MONTH' : 'YEAR',
            interval_count: 1
          },
          tenure_type: 'REGULAR',
          sequence: 1,
          total_cycles: 0, // Infinite
          pricing_scheme: {
            fixed_price: {
              value: config.price,
              currency_code: 'USD'
            }
          }
        }
      ],
      payment_preferences: {
        auto_bill_outstanding: true,
        setup_fee_failure_action: 'CANCEL',
        payment_failure_threshold: 3
      }
    },
    {
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      }
    }
  );
  
  return planResponse.data.id;
}

// Store plan IDs in database
interface StoredPlan {
  tier: 'starter' | 'professional' | 'enterprise';
  billing_cycle: 'monthly' | 'annual';
  paypal_plan_id: string;
  price: number;
  credits: number;
}

// Get plan ID from database
async function getPayPalPlanId(
  tier: 'starter' | 'professional' | 'enterprise',
  billingCycle: 'monthly' | 'annual'
): Promise<string> {
  // In production, fetch from database or config
  const plans = {
    'starter-monthly': process.env.PAYPAL_PLAN_STARTER_MONTHLY!,
    'professional-monthly': process.env.PAYPAL_PLAN_PROFESSIONAL_MONTHLY!,
    'enterprise-monthly': process.env.PAYPAL_PLAN_ENTERPRISE_MONTHLY!
  };
  
  return plans[`${tier}-${billingCycle}`];
}
```

### 3. Create Subscription (User Checkout)

```typescript
import { z } from 'zod';

const CreateSubscriptionSchema = z.object({
  plan_tier: z.enum(['starter', 'professional', 'enterprise']),
  billing_cycle: z.enum(['monthly', 'annual']),
  return_url: z.string().url(),
  cancel_url: z.string().url()
});

// Create subscription endpoint
router.post('/subscriptions', requireAuth, async (req: AuthRequest, res) => {
  try {
    const data = CreateSubscriptionSchema.parse(req.body);
    const userId = req.user!.userId;
    
    // Check if user already has active subscription
    const existing = await prisma.subscription.findFirst({
      where: {
        user_id: userId,
        status: 'active'
      }
    });
    
    if (existing) {
      return res.status(409).json({ error: 'Active subscription already exists' });
    }
    
    const token = await getAccessToken();
    const planId = await getPayPalPlanId(data.plan_tier, data.billing_cycle);
    
    // Create PayPal subscription
    const response = await axios.post(
      `${PAYPAL_BASE_URL}/v1/billing/subscriptions`,
      {
        plan_id: planId,
        custom_id: userId, // Store user ID for webhook handling
        application_context: {
          brand_name: 'TranslatePressZone',
          locale: 'en-US',
          shipping_preference: 'NO_SHIPPING',
          user_action: 'SUBSCRIBE_NOW',
          return_url: data.return_url,
          cancel_url: data.cancel_url
        }
      },
      {
        headers: {
          'Authorization': `Bearer ${token}`,
          'Content-Type': 'application/json'
        }
      }
    );
    
    const subscriptionId = response.data.id;
    const approvalUrl = response.data.links.find((link: any) => link.rel === 'approve')?.href;
    
    // Create pending subscription in database
    await prisma.subscription.create({
      data: {
        user_id: userId,
        plan_tier: data.plan_tier,
        billing_cycle: data.billing_cycle,
        status: 'pending',
        paypal_subscription_id: subscriptionId,
        current_period_start: new Date(),
        current_period_end: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000) // Placeholder
      }
    });
    
    res.json({
      subscription_id: subscriptionId,
      approval_url: approvalUrl
    });
    
  } catch (error) {
    res.status(500).json({ error: 'Failed to create subscription' });
  }
});
```

### 4. Handle Subscription Webhooks

```typescript
// See webhook-implementation skill for signature verification

// Subscription activated
async function handleSubscriptionActivated(event: any) {
  const subscriptionId = event.resource.id;
  const billingInfo = event.resource.billing_info;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: {
      status: 'active',
      current_period_start: new Date(billingInfo.last_payment.time),
      current_period_end: new Date(billingInfo.next_billing_time)
    }
  });
  
  // Allocate initial credits
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId },
    include: { user: true }
  });
  
  if (subscription) {
    const planCredits = {
      starter: 100000,
      professional: 500000,
      enterprise: 2000000
    };
    
    const credits = planCredits[subscription.plan_tier];
    
    await prisma.creditTransaction.create({
      data: {
        user_id: subscription.user_id,
        type: 'allocation',
        amount: credits,
        balance_after: credits,
        description: `Initial allocation for ${subscription.plan_tier} plan`
      }
    });
  }
}

// Payment completed
async function handlePaymentCompleted(event: any) {
  const paymentId = event.resource.id;
  const amount = parseFloat(event.resource.amount.total);
  const subscriptionId = event.resource.billing_agreement_id;
  
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId }
  });
  
  if (!subscription) {
    console.error(`Subscription not found: ${subscriptionId}`);
    return;
  }
  
  // Record payment
  await prisma.payment.create({
    data: {
      user_id: subscription.user_id,
      paypal_payment_id: paymentId,
      amount,
      currency: 'USD',
      status: 'completed',
      type: 'subscription_payment',
      subscription_id: subscription.id
    }
  });
  
  // Extend subscription period
  const nextPeriodEnd = new Date(subscription.current_period_end);
  nextPeriodEnd.setMonth(nextPeriodEnd.getMonth() + 1);
  
  await prisma.subscription.update({
    where: { id: subscription.id },
    data: {
      current_period_start: subscription.current_period_end,
      current_period_end: nextPeriodEnd
    }
  });
}

// Subscription cancelled
async function handleSubscriptionCancelled(event: any) {
  const subscriptionId = event.resource.id;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: {
      status: 'cancelled',
      cancel_at_period_end: true
    }
  });
  
  // User retains credits until period end
}

// Subscription suspended (payment failed)
async function handleSubscriptionSuspended(event: any) {
  const subscriptionId = event.resource.id;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: { status: 'suspended' }
  });
  
  // Suspend user account
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId }
  });
  
  if (subscription) {
    await prisma.user.update({
      where: { id: subscription.user_id },
      data: { status: 'suspended' }
    });
  }
}
```

### 5. Cancel Subscription

```typescript
// User-initiated cancellation
router.delete('/subscriptions/:id', requireAuth, async (req: AuthRequest, res) => {
  const { id } = req.params;
  const userId = req.user!.userId;
  
  const subscription = await prisma.subscription.findFirst({
    where: {
      id,
      user_id: userId,
      status: 'active'
    }
  });
  
  if (!subscription) {
    return res.status(404).json({ error: 'Active subscription not found' });
  }
  
  try {
    const token = await getAccessToken();
    
    // Cancel via PayPal API
    await axios.post(
      `${PAYPAL_BASE_URL}/v1/billing/subscriptions/${subscription.paypal_subscription_id}/cancel`,
      {
        reason: 'User requested cancellation'
      },
      {
        headers: {
          'Authorization': `Bearer ${token}`,
          'Content-Type': 'application/json'
        }
      }
    );
    
    // Update local database
    await prisma.subscription.update({
      where: { id },
      data: {
        status: 'cancelled',
        cancel_at_period_end: true
      }
    });
    
    res.json({
      success: true,
      message: 'Subscription will be cancelled at period end',
      period_end: subscription.current_period_end
    });
    
  } catch (error) {
    res.status(500).json({ error: 'Failed to cancel subscription' });
  }
});
```

### 6. Payment History

```typescript
// Get payment history
router.get('/payments', requireAuth, async (req: AuthRequest, res) => {
  const userId = req.user!.userId;
  const page = parseInt(req.query.page as string) || 1;
  const limit = 20;
  const skip = (page - 1) * limit;
  
  const [payments, total] = await Promise.all([
    prisma.payment.findMany({
      where: { user_id: userId },
      orderBy: { created_at: 'desc' },
      skip,
      take: limit
    }),
    prisma.payment.count({ where: { user_id: userId } })
  ]);
  
  res.json({
    payments: payments.map(p => ({
      id: p.id,
      amount: p.amount,
      currency: p.currency,
      status: p.status,
      type: p.type,
      date: p.created_at
    })),
    pagination: {
      page,
      limit,
      total,
      pages: Math.ceil(total / limit)
    }
  });
});
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Storing PayPal credentials in code | Use environment variables |
| Not caching access tokens | Cache token for 1 hour |
| Trusting webhook data without verification | Always verify PayPal signatures |
| Double-charging users | Check for duplicate payment IDs |
| Not handling suspended subscriptions | Suspend user account access |
| Allowing negative credit balances | Check balance before deducting |
| Not logging payment events | Log all transactions to audit log |
| Hardcoding plan IDs in code | Store in database or env vars |
| Not handling refunds | Implement refund webhook handler |
| Immediate account termination on cancel | Allow access until period end |

## Integration with Other Skills

**Often combined with:**
- `webhook-implementation` - Processing PayPal webhooks
- `authentication-security` - Protecting payment endpoints
- `queue-management` - Scheduled credit allocation
- `error-handling-logging` - Logging payment events

**Depends on:**
- `database-schema-design` - Subscription, Payment, CreditTransaction tables

## Environment Variables Required

```bash
# PayPal Configuration
PAYPAL_MODE="sandbox"  # or "live"
PAYPAL_CLIENT_ID="your-client-id"
PAYPAL_CLIENT_SECRET="your-client-secret"

# PayPal Webhook
PAYPAL_WEBHOOK_ID="webhook-id-from-dashboard"
PAYPAL_WEBHOOK_SECRET="webhook-secret"

# Plan IDs (from PayPal Dashboard)
PAYPAL_PLAN_STARTER_MONTHLY="P-xxx"
PAYPAL_PLAN_PROFESSIONAL_MONTHLY="P-xxx"
PAYPAL_PLAN_ENTERPRISE_MONTHLY="P-xxx"
```

## Quick Reference

### Subscription Tiers

| Tier | Monthly Price | Credits | Annual Discount |
|------|---------------|---------|-----------------|
| Starter | $9 | 100K tokens | 15% |
| Professional | $29 | 500K tokens | 15% |
| Enterprise | $99 | 2M tokens | 15% |

### PayPal Webhook Events

| Event | Action |
|-------|--------|
| `BILLING.SUBSCRIPTION.CREATED` | Store subscription ID |
| `BILLING.SUBSCRIPTION.ACTIVATED` | Activate account, allocate credits |
| `PAYMENT.SALE.COMPLETED` | Record payment, extend period |
| `BILLING.SUBSCRIPTION.CANCELLED` | Mark for cancellation at period end |
| `BILLING.SUBSCRIPTION.SUSPENDED` | Suspend account |
| `BILLING.SUBSCRIPTION.UPDATED` | Sync plan changes |
| `PAYMENT.SALE.REFUNDED` | Process refund, deduct credits |

### Subscription Statuses

- `pending` - Created, awaiting user approval
- `active` - Active subscription with valid payment
- `suspended` - Payment failed, awaiting resolution
- `cancelled` - User cancelled, access until period end
- `past_due` - Payment overdue

## Validation Checklist

Before completing payment integration:

- [ ] PayPal credentials stored securely in env vars
- [ ] Access token caching implemented (1 hour)
- [ ] All webhook events handled properly
- [ ] Webhook signature verification working
- [ ] Duplicate payment prevention implemented
- [ ] Subscription status synced correctly
- [ ] Credits allocated on activation
- [ ] Credits retained until period end on cancellation
- [ ] Payment history endpoint implemented
- [ ] Refund handling implemented
- [ ] Suspended accounts blocked from API access
- [ ] Audit logs for all payment events
- [ ] Proper error handling for PayPal API failures
- [ ] Test mode with sandbox credentials
- [ ] Production plan IDs configured

## Testing

### Sandbox Testing
```bash
# PayPal Sandbox Credentials
# https://developer.paypal.com/dashboard/

# Test Credit Cards:
# Visa: 4111111111111111
# Mastercard: 5555555555554444
# Amex: 378282246310005

# Test accounts created in sandbox
```

### Manual Testing Flow
1. Create subscription via API
2. Complete payment in PayPal sandbox
3. Verify webhook received and processed
4. Check database for subscription activation
5. Verify credits allocated
6. Test cancellation flow
7. Verify access retained until period end

## Monitoring

### Key Metrics
- Subscription creation rate
- Payment success rate
- Churn rate (cancellations)
- Failed payment rate
- Average revenue per user (ARPU)

### Alerts
- Failed webhook deliveries from PayPal
- Payment failures exceeding threshold
- Unusual subscription cancellation spike
- Duplicate payment detections
# Skill: Queue Management with Bull

## Identity
- **Skill ID**: `queue-management`
- **Domain**: Asynchronous Job Processing, Background Workers
- **Technologies**: Bull, BullMQ, Redis, Job Queues
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Translation job processing
- Webhook delivery with retry logic
- Background email sending
- Periodic credit allocation
- Scheduled cleanup tasks
- Job status monitoring
- Queue metrics and monitoring

**File patterns:**
- `api/src/queues/**/*.ts`
- `api/src/workers/**/*.ts`
- `api/src/jobs/**/*.ts`
- `api/src/services/job-processor*.ts`

## Core Patterns

### 1. Queue Setup and Configuration

```typescript
import Queue from 'bull';
import Redis from 'ioredis';

// Redis connection
const redisOptions = {
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  maxRetriesPerRequest: null,
  enableReadyCheck: false
};

// Create translation job queue
export const translationQueue = new Queue('translation-jobs', {
  redis: redisOptions,
  defaultJobOptions: {
    attempts: 3, // Retry up to 3 times
    backoff: {
      type: 'exponential',
      delay: 2000 // Start with 2 seconds, then 4s, 8s
    },
    removeOnComplete: 100, // Keep last 100 completed jobs
    removeOnFail: false // Keep failed jobs for debugging
  }
});

// Webhook delivery queue
export const webhookQueue = new Queue('webhook-deliveries', {
  redis: redisOptions,
  defaultJobOptions: {
    attempts: 5, // More retries for webhooks
    backoff: {
      type: 'exponential',
      delay: 5000 // Start with 5 seconds
    },
    timeout: 30000 // 30 second timeout per attempt
  }
});

// Credit allocation queue (scheduled)
export const creditQueue = new Queue('credit-allocation', {
  redis: redisOptions,
  defaultJobOptions: {
    attempts: 1, // No retries for scheduled jobs
    removeOnComplete: 10
  }
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  await translationQueue.close();
  await webhookQueue.close();
  await creditQueue.close();
  process.exit(0);
});
```

### 2. Adding Jobs to Queue

```typescript
import { translationQueue } from './queues';
import { z } from 'zod';

// Job data schema
const TranslationJobData = z.object({
  jobId: z.string().uuid(),
  userId: z.string().uuid(),
  sourceLang: z.string().length(2),
  targetLang: z.string().length(2),
  content: z.string(),
  model: z.enum(['4b', '27b']),
  tone: z.enum(['neutral', 'formal', 'casual']),
  callbackUrl: z.string().url().optional(),
  callbackSecret: z.string().optional()
});

type TranslationJobData = z.infer<typeof TranslationJobData>;

// Add job to queue
async function queueTranslationJob(data: TranslationJobData) {
  // Validate data
  const validatedData = TranslationJobData.parse(data);
  
  // Add to queue with priority
  const job = await translationQueue.add(validatedData, {
    jobId: validatedData.jobId, // Custom job ID for deduplication
    priority: validatedData.model === '27b' ? 1 : 2, // Premium gets priority
    timeout: validatedData.model === '27b' ? 600000 : 300000 // 10min vs 5min
  });
  
  return job;
}

// Endpoint to submit job
router.post('/jobs', requireApiKey, async (req: AuthRequest, res) => {
  try {
    const userId = req.user!.userId;
    
    // Check credits first
    const user = await prisma.user.findUnique({
      where: { id: userId },
      include: {
        credit_transactions: {
          orderBy: { created_at: 'desc' },
          take: 1
        }
      }
    });
    
    const currentBalance = user.credit_transactions[0]?.balance_after || 0;
    const estimatedCost = estimateTokens(req.body.content) * getCostPerToken(req.body.model);
    
    if (currentBalance < estimatedCost) {
      return res.status(402).json({ error: 'Insufficient credits' });
    }
    
    // Create job in database
    const job = await prisma.translationJob.create({
      data: {
        user_id: userId,
        client_job_id: req.body.job_id,
        status: 'pending',
        source_lang: req.body.source_lang,
        target_lang: req.body.target_lang,
        model: req.body.model,
        tone: req.body.tone || 'neutral',
        content: req.body.content,
        content_hash: createHash('sha256').update(req.body.content).digest('hex'),
        callback_url: req.body.callback_url,
        callback_secret: req.body.callback_secret
      }
    });
    
    // Add to queue
    await queueTranslationJob({
      jobId: job.id,
      userId: job.user_id,
      sourceLang: job.source_lang,
      targetLang: job.target_lang,
      content: job.content,
      model: job.model as '4b' | '27b',
      tone: job.tone as 'neutral' | 'formal' | 'casual',
      callbackUrl: job.callback_url || undefined,
      callbackSecret: job.callback_secret || undefined
    });
    
    res.status(202).json({
      success: true,
      job_id: job.id,
      client_job_id: job.client_job_id,
      status: 'pending',
      estimated_tokens: estimateTokens(req.body.content)
    });
  } catch (error) {
    res.status(500).json({ error: 'Failed to queue job' });
  }
});
```

### 3. Worker Processing

```typescript
import { Job } from 'bull';
import { translationQueue } from './queues';
import { prisma } from '../lib/prisma';
import axios from 'axios';

interface TranslationResult {
  translation: string;
  tokensUsed: number;
  processingTimeMs: number;
}

// Process translation jobs
translationQueue.process(5, async (job: Job<TranslationJobData>) => {
  const startTime = Date.now();
  
  try {
    // Update job status to processing
    await prisma.translationJob.update({
      where: { id: job.data.jobId },
      data: { status: 'processing' }
    });

    // Call Gemini API translation service
    const response = await geminiClient.translate(
      job.data.content,
      job.data.sourceLang,
      job.data.targetLang,
      job.data.model,
      job.data.tone
    );

    // Update job with translation result
    await prisma.translationJob.update({
      where: { id: job.data.jobId },
      data: {
        timeout: job.opts.timeout
      }
    );
    
    const processingTime = Date.now() - startTime;
    const { translation, tokensUsed } = response.data;
    
    // Calculate cost
    const costPerToken = job.data.model === '4b' ? 0.0000005 : 0.000002;
    const cost = tokensUsed * costPerToken;
    
    // Update job in database
    await prisma.translationJob.update({
      where: { id: job.data.jobId },
      data: {
        status: 'completed',
        translation,
        tokens_used: tokensUsed,
        cost,
        processing_time_ms: processingTime,
        completed_at: new Date()
      }
    });
    
    // Deduct credits
    const lastTransaction = await prisma.creditTransaction.findFirst({
      where: { user_id: job.data.userId },
      orderBy: { created_at: 'desc' }
    });
    
    const currentBalance = lastTransaction?.balance_after || 0;
    
    await prisma.creditTransaction.create({
      data: {
        user_id: job.data.userId,
        type: 'deduction',
        amount: -tokensUsed,
        balance_after: currentBalance - tokensUsed,
        description: `Translation job ${job.data.jobId}`,
        related_job_id: job.data.jobId
      }
    });
    
    // Queue webhook if callback URL provided
    if (job.data.callbackUrl) {
      await webhookQueue.add({
        jobId: job.data.jobId,
        url: job.data.callbackUrl,
        secret: job.data.callbackSecret,
        payload: {
          job_id: job.data.jobId,
          client_job_id: await getClientJobId(job.data.jobId),
          status: 'completed',
          translation,
          tokens_used: tokensUsed,
          cost_usd: cost
        }
      });
    }
    
    return { success: true, tokensUsed, cost };
    
  } catch (error) {
    // Update job status to failed
    await prisma.translationJob.update({
      where: { id: job.data.jobId },
      data: {
        status: 'failed',
        error_message: error.message
      }
    });
    
    // Refund credits if deducted
    const lastTransaction = await prisma.creditTransaction.findFirst({
      where: {
        user_id: job.data.userId,
        related_job_id: job.data.jobId,
        type: 'deduction'
      }
    });
    
    if (lastTransaction) {
      await prisma.creditTransaction.create({
        data: {
          user_id: job.data.userId,
          type: 'refund',
          amount: Math.abs(lastTransaction.amount),
          balance_after: lastTransaction.balance_after + Math.abs(lastTransaction.amount),
          description: `Refund for failed job ${job.data.jobId}`,
          related_job_id: job.data.jobId
        }
      });
    }
    
    throw error; // Re-throw for Bull retry logic
  }
});

// Helper function
async function getClientJobId(jobId: string): Promise<string | null> {
  const job = await prisma.translationJob.findUnique({
    where: { id: jobId },
    select: { client_job_id: true }
  });
  return job?.client_job_id || null;
}
```

### 4. Webhook Delivery Worker

```typescript
import { Job } from 'bull';
import { webhookQueue } from './queues';
import { prisma } from '../lib/prisma';
import axios from 'axios';
import crypto from 'crypto';

interface WebhookJobData {
  jobId: string;
  url: string;
  secret?: string;
  payload: any;
}

// Process webhook deliveries
webhookQueue.process(10, async (job: Job<WebhookJobData>) => {
  const attemptNumber = job.attemptsMade + 1;
  
  try {
    // Generate HMAC signature
    const timestamp = Date.now().toString();
    const payloadString = JSON.stringify(job.data.payload);
    
    let signature: string | undefined;
    if (job.data.secret) {
      signature = crypto
        .createHmac('sha256', job.data.secret)
        .update(payloadString)
        .digest('hex');
    }
    
    // Send webhook
    const response = await axios.post(job.data.url, job.data.payload, {
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': signature || '',
        'X-Webhook-Timestamp': timestamp,
        'User-Agent': 'TranslatePressZone/1.0'
      },
      timeout: 30000,
      validateStatus: (status) => status >= 200 && status < 300
    });
    
    // Log successful delivery
    await prisma.webhookDelivery.create({
      data: {
        job_id: job.data.jobId,
        attempt_number: attemptNumber,
        success: true,
        http_status: response.status,
        response_body: JSON.stringify(response.data).substring(0, 1000)
      }
    });
    
    return { success: true };
    
  } catch (error) {
    const httpStatus = error.response?.status || null;
    const errorMessage = error.message;
    const responseBody = error.response?.data 
      ? JSON.stringify(error.response.data).substring(0, 1000)
      : null;
    
    // Log failed delivery
    await prisma.webhookDelivery.create({
      data: {
        job_id: job.data.jobId,
        attempt_number: attemptNumber,
        success: false,
        http_status: httpStatus,
        response_body: responseBody,
        error_message: errorMessage
      }
    });
    
    // Don't retry on 4xx errors (client errors)
    if (httpStatus && httpStatus >= 400 && httpStatus < 500) {
      console.error(`Webhook failed with client error ${httpStatus}, not retrying`);
      return { success: false, permanent: true };
    }
    
    throw error; // Re-throw for Bull retry logic (5xx errors, network issues)
  }
});
```

### 5. Scheduled Jobs (Cron-style)

```typescript
import { creditQueue } from './queues';

// Monthly credit allocation
creditQueue.add(
  'monthly-credit-allocation',
  {},
  {
    repeat: {
      cron: '0 0 1 * *' // First day of month at midnight
    },
    jobId: 'monthly-credit-allocation' // Prevent duplicates
  }
);

// Process scheduled credit allocation
creditQueue.process('monthly-credit-allocation', async (job) => {
  // Get all active subscriptions
  const subscriptions = await prisma.subscription.findMany({
    where: {
      status: 'active',
      current_period_end: {
        gte: new Date()
      }
    },
    include: {
      user: {
        include: {
          credit_transactions: {
            orderBy: { created_at: 'desc' },
            take: 1
          }
        }
      }
    }
  });
  
  // Allocate credits based on plan
  const planCredits = {
    starter: 100000,      // 100K tokens
    professional: 500000, // 500K tokens
    enterprise: 2000000   // 2M tokens
  };
  
  for (const subscription of subscriptions) {
    const credits = planCredits[subscription.plan_tier];
    const currentBalance = subscription.user.credit_transactions[0]?.balance_after || 0;
    
    await prisma.creditTransaction.create({
      data: {
        user_id: subscription.user_id,
        type: 'allocation',
        amount: credits,
        balance_after: currentBalance + credits,
        description: `Monthly allocation for ${subscription.plan_tier} plan`,
        related_payment_id: null
      }
    });
  }
  
  return { allocated: subscriptions.length };
});

// Daily cleanup of old completed jobs
creditQueue.add(
  'cleanup-old-jobs',
  {},
  {
    repeat: {
      cron: '0 2 * * *' // Daily at 2 AM
    },
    jobId: 'cleanup-old-jobs'
  }
);

creditQueue.process('cleanup-old-jobs', async () => {
  const thirtyDaysAgo = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000);
  
  const deleted = await prisma.translationJob.deleteMany({
    where: {
      status: 'completed',
      completed_at: {
        lt: thirtyDaysAgo
      }
    }
  });
  
  return { deleted: deleted.count };
});
```

### 6. Queue Monitoring and Metrics

```typescript
import { translationQueue, webhookQueue } from './queues';

// Admin endpoint for queue metrics
router.get('/admin/queues', requireAuth, async (req, res) => {
  const [translationCounts, webhookCounts] = await Promise.all([
    translationQueue.getJobCounts(),
    webhookQueue.getJobCounts()
  ]);
  
  // Get failed jobs
  const failedTranslations = await translationQueue.getFailed(0, 10);
  const failedWebhooks = await webhookQueue.getFailed(0, 10);
  
  res.json({
    translation_queue: {
      waiting: translationCounts.waiting,
      active: translationCounts.active,
      completed: translationCounts.completed,
      failed: translationCounts.failed,
      delayed: translationCounts.delayed,
      failed_jobs: failedTranslations.map(job => ({
        id: job.id,
        data: job.data,
        error: job.failedReason,
        attempts: job.attemptsMade
      }))
    },
    webhook_queue: {
      waiting: webhookCounts.waiting,
      active: webhookCounts.active,
      completed: webhookCounts.completed,
      failed: webhookCounts.failed,
      failed_jobs: failedWebhooks.map(job => ({
        id: job.id,
        data: job.data,
        error: job.failedReason,
        attempts: job.attemptsMade
      }))
    }
  });
});

// Retry failed job
router.post('/admin/queues/:queue/jobs/:jobId/retry', requireAuth, async (req, res) => {
  const { queue, jobId } = req.params;
  
  const targetQueue = queue === 'translation' ? translationQueue : webhookQueue;
  const job = await targetQueue.getJob(jobId);
  
  if (!job) {
    return res.status(404).json({ error: 'Job not found' });
  }
  
  await job.retry();
  
  res.json({ success: true });
});

// Queue event listeners for monitoring
translationQueue.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed in ${result.processingTimeMs}ms`);
});

translationQueue.on('failed', (job, error) => {
  console.error(`Job ${job.id} failed:`, error.message);
  // Send alert to admin
});

translationQueue.on('stalled', (job) => {
  console.warn(`Job ${job.id} stalled`);
});
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Processing jobs in HTTP request handlers | Use queue for async processing |
| Not handling worker crashes gracefully | Implement graceful shutdown and health checks |
| Infinite retries on failed jobs | Set `attempts` limit (3-5 attempts) |
| Not using exponential backoff | Configure `backoff: { type: 'exponential' }` |
| Storing large data in job payload | Store in database, pass ID only |
| Not monitoring queue health | Implement metrics endpoints and alerts |
| Retrying client errors (4xx) | Only retry 5xx and network errors |
| Not cleaning up completed jobs | Set `removeOnComplete` option |
| Blocking Redis connection | Use separate Redis connections for queues |
| Not validating job data | Use Zod schemas for type safety |

## Integration with Other Skills

**Often combined with:**
- `api-endpoint-creation` - Job submission endpoints
- `authentication-security` - Protecting admin queue endpoints
- `ml-service-integration` - Calling translation service
- `webhook-implementation` - Delivering callbacks
- `error-handling-logging` - Logging job failures

**Depends on:**
- `database-schema-design` - TranslationJob, WebhookDelivery tables

## Environment Variables Required

```bash
# Redis Configuration
REDIS_HOST="localhost"
REDIS_PORT="6379"
REDIS_PASSWORD=""  # Optional

# Google Gemini API (Translation Engine)
GEMINI_API_KEY="your-google-gemini-api-key"
GEMINI_MODEL="gemini-3-flash-preview"

# Queue Configuration
QUEUE_CONCURRENCY_TRANSLATION=5  # Concurrent translation jobs
QUEUE_CONCURRENCY_WEBHOOK=10     # Concurrent webhook deliveries
```

## Quick Reference

### Queue Configuration Options

```typescript
{
  attempts: 3,              // Retry count
  backoff: {
    type: 'exponential',    // or 'fixed'
    delay: 2000            // Initial delay in ms
  },
  timeout: 300000,          // 5 minutes
  removeOnComplete: 100,    // Keep last 100
  removeOnFail: false,      // Keep failed jobs
  priority: 1,              // Lower = higher priority
  jobId: 'unique-id'        // Custom ID (prevents duplicates)
}
```

### Retry Strategies by Job Type

| Job Type | Attempts | Backoff | Timeout |
|----------|----------|---------|---------|
| Translation (4b) | 3 | Exponential (2s) | 5 minutes |
| Translation (27b) | 3 | Exponential (2s) | 10 minutes |
| Webhook Delivery | 5 | Exponential (5s) | 30 seconds |
| Credit Allocation | 1 | None | 1 minute |
| Cleanup Tasks | 2 | Fixed (60s) | 5 minutes |

### Job Status Lifecycle

```
pending → processing → completed
                    ↘ failed (with retries)
                    ↘ cancelled (manual)
```

### Bull Events

- `completed` - Job finished successfully
- `failed` - Job failed (after all retries)
- `stalled` - Job hasn't reported progress
- `progress` - Job reported progress
- `active` - Job started processing
- `waiting` - Job added to queue
- `removed` - Job removed from queue

## Validation Checklist

Before completing queue implementation:

- [ ] Redis connection configured with retry logic
- [ ] All queues have `defaultJobOptions` set
- [ ] Worker concurrency tuned for system resources
- [ ] Exponential backoff configured for retries
- [ ] Job timeouts prevent hanging workers
- [ ] Failed jobs retained for debugging (`removeOnFail: false`)
- [ ] Completed jobs cleaned up (`removeOnComplete` set)
- [ ] Graceful shutdown implemented (SIGTERM handler)
- [ ] Queue metrics endpoint implemented
- [ ] Job data validated with Zod schemas
- [ ] Client errors (4xx) don't trigger retries
- [ ] Credits refunded on job failure
- [ ] Webhook signatures generated correctly
- [ ] Scheduled jobs use `jobId` to prevent duplicates
- [ ] Queue health monitored (stalled job alerts)

## Worker Deployment

### Development
```bash
# Start API server
npm run dev

# Start worker (separate process)
npm run worker
```

### Production (Docker)
```yaml
# docker-compose.yml
services:
  api:
    build: .
    command: npm start
    
  worker:
    build: .
    command: npm run worker
    replicas: 3  # Scale workers independently
    
  redis:
    image: redis:7-alpine
```

### Horizontal Scaling
- Multiple worker instances process from same queue
- Bull handles distributed locking automatically
- Scale workers independently from API servers
- Monitor Redis memory usage

## Common Issues & Solutions

| Issue | Cause | Solution |
|-------|-------|----------|
| Jobs stuck in "active" | Worker crash | Implement stalled job detection |
| Memory growth | Not removing completed jobs | Set `removeOnComplete` option |
| Slow processing | Too many concurrent jobs | Reduce `process()` concurrency |
| Lost jobs | Redis persistence disabled | Enable Redis AOF persistence |
| Duplicate jobs | No `jobId` specified | Use unique `jobId` for idempotency |
| High Redis memory | Large job payloads | Store data in Postgres, pass ID only |
# Skill: Webhook Implementation

## Identity
- **Skill ID**: `webhook-implementation`
- **Domain**: Webhook Sending/Receiving, HTTP Callbacks
- **Technologies**: Axios, HMAC Signatures, Retry Logic
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Sending webhooks to WordPress plugin
- Receiving webhooks from PayPal
- HMAC signature generation and verification
- Webhook retry logic with exponential backoff
- Webhook delivery tracking
- Webhook endpoint validation

**File patterns:**
- `api/src/services/webhook*.ts`
- `api/src/routes/webhooks/**/*.ts`
- `api/src/middleware/webhook-verify*.ts`

## Core Patterns

### 1. Outgoing Webhooks (to WordPress Plugin)

```typescript
import axios, { AxiosError } from 'axios';
import crypto from 'crypto';
import { prisma } from '../lib/prisma';

interface WebhookPayload {
  job_id: string;
  client_job_id: string | null;
  status: 'completed' | 'failed';
  translation?: string;
  tokens_used?: number;
  cost_usd?: number;
  error?: string;
}

// Generate HMAC-SHA256 signature
function generateSignature(payload: string, secret: string): string {
  return crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
}

// Send webhook with signature
async function sendWebhook(
  url: string,
  payload: WebhookPayload,
  secret?: string
): Promise<{ success: boolean; httpStatus?: number; error?: string }> {
  try {
    const timestamp = Date.now().toString();
    const payloadString = JSON.stringify(payload);
    
    // Generate signature if secret provided
    const signature = secret 
      ? generateSignature(payloadString, secret)
      : undefined;
    
    const response = await axios.post(url, payload, {
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': signature || '',
        'X-Webhook-Timestamp': timestamp,
        'User-Agent': 'TranslatePressZone/1.0'
      },
      timeout: 30000, // 30 second timeout
      validateStatus: (status) => status >= 200 && status < 300
    });
    
    return {
      success: true,
      httpStatus: response.status
    };
    
  } catch (error) {
    const axiosError = error as AxiosError;
    return {
      success: false,
      httpStatus: axiosError.response?.status,
      error: axiosError.message
    };
  }
}

// Webhook delivery with retry (use this from queue worker)
export async function deliverWebhook(
  jobId: string,
  url: string,
  payload: WebhookPayload,
  secret?: string,
  attemptNumber: number = 1
): Promise<void> {
  const result = await sendWebhook(url, payload, secret);
  
  // Log delivery attempt
  await prisma.webhookDelivery.create({
    data: {
      job_id: jobId,
      attempt_number: attemptNumber,
      success: result.success,
      http_status: result.httpStatus || null,
      error_message: result.error || null
    }
  });
  
  if (!result.success) {
    // Don't retry on client errors (4xx)
    if (result.httpStatus && result.httpStatus >= 400 && result.httpStatus < 500) {
      throw new Error(`Permanent failure: ${result.error}`);
    }
    
    // Retry on server errors (5xx) and network issues
    throw new Error(result.error);
  }
}
```

### 2. Incoming Webhooks (from PayPal)

```typescript
import { Request, Response, NextFunction } from 'express';
import crypto from 'crypto';

// Verify PayPal webhook signature
function verifyPayPalSignature(
  transmissionId: string,
  timestamp: string,
  webhookId: string,
  eventBody: string,
  certUrl: string,
  actualSignature: string,
  algorithm: string
): boolean {
  // In production, verify cert_url is from PayPal domain
  if (!certUrl.startsWith('https://api.paypal.com/')) {
    return false;
  }
  
  // Construct expected signature string
  const expectedSigString = `${transmissionId}|${timestamp}|${webhookId}|${crypto.createHash('sha256').update(eventBody).digest('base64')}`;
  
  // For simplicity, using HMAC with webhook secret
  // In production, use PayPal's SDK for proper verification
  const secret = process.env.PAYPAL_WEBHOOK_SECRET!;
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(expectedSigString)
    .digest('base64');
  
  return crypto.timingSafeEqual(
    Buffer.from(actualSignature),
    Buffer.from(expectedSignature)
  );
}

// Middleware to verify PayPal webhooks
export function verifyPayPalWebhook(req: Request, res: Response, next: NextFunction) {
  const transmissionId = req.headers['paypal-transmission-id'] as string;
  const timestamp = req.headers['paypal-transmission-time'] as string;
  const webhookId = process.env.PAYPAL_WEBHOOK_ID!;
  const certUrl = req.headers['paypal-cert-url'] as string;
  const actualSignature = req.headers['paypal-transmission-sig'] as string;
  const algorithm = req.headers['paypal-auth-algo'] as string;
  
  if (!transmissionId || !timestamp || !actualSignature) {
    return res.status(401).json({ error: 'Missing PayPal signature headers' });
  }
  
  // Verify timestamp (prevent replay attacks)
  const webhookTime = new Date(timestamp).getTime();
  const currentTime = Date.now();
  const timeDiff = Math.abs(currentTime - webhookTime);
  
  if (timeDiff > 5 * 60 * 1000) { // 5 minute window
    return res.status(401).json({ error: 'Webhook timestamp too old' });
  }
  
  // Verify signature
  const eventBody = JSON.stringify(req.body);
  const isValid = verifyPayPalSignature(
    transmissionId,
    timestamp,
    webhookId,
    eventBody,
    certUrl,
    actualSignature,
    algorithm
  );
  
  if (!isValid) {
    return res.status(401).json({ error: 'Invalid PayPal signature' });
  }
  
  next();
}

// PayPal webhook handler
router.post('/webhooks/paypal', verifyPayPalWebhook, async (req, res) => {
  const event = req.body;
  
  try {
    switch (event.event_type) {
      case 'BILLING.SUBSCRIPTION.CREATED':
        await handleSubscriptionCreated(event);
        break;
        
      case 'BILLING.SUBSCRIPTION.ACTIVATED':
        await handleSubscriptionActivated(event);
        break;
        
      case 'PAYMENT.SALE.COMPLETED':
        await handlePaymentCompleted(event);
        break;
        
      case 'BILLING.SUBSCRIPTION.CANCELLED':
        await handleSubscriptionCancelled(event);
        break;
        
      case 'BILLING.SUBSCRIPTION.SUSPENDED':
        await handleSubscriptionSuspended(event);
        break;
        
      default:
        console.log(`Unhandled PayPal event: ${event.event_type}`);
    }
    
    res.sendStatus(200);
  } catch (error) {
    console.error('PayPal webhook error:', error);
    res.sendStatus(500);
  }
});

// Event handlers
async function handleSubscriptionCreated(event: any) {
  const subscriptionId = event.resource.id;
  const customId = event.resource.custom_id; // User ID
  
  // Store subscription ID for later activation
  await prisma.subscription.update({
    where: { user_id: customId },
    data: {
      paypal_subscription_id: subscriptionId,
      status: 'pending'
    }
  });
}

async function handleSubscriptionActivated(event: any) {
  const subscriptionId = event.resource.id;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: {
      status: 'active',
      current_period_start: new Date(event.resource.billing_info.last_payment.time),
      current_period_end: new Date(event.resource.billing_info.next_billing_time)
    }
  });
  
  // Allocate initial credits (handled by queue in production)
}

async function handlePaymentCompleted(event: any) {
  const paymentId = event.resource.id;
  const amount = parseFloat(event.resource.amount.total);
  const subscriptionId = event.resource.billing_agreement_id;
  
  // Find user by subscription
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId }
  });
  
  if (!subscription) {
    console.error(`Subscription not found: ${subscriptionId}`);
    return;
  }
  
  // Record payment
  await prisma.payment.create({
    data: {
      user_id: subscription.user_id,
      paypal_payment_id: paymentId,
      amount,
      currency: 'USD',
      status: 'completed',
      type: 'subscription_payment',
      subscription_id: subscription.id
    }
  });
}

async function handleSubscriptionCancelled(event: any) {
  const subscriptionId = event.resource.id;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: {
      status: 'cancelled',
      cancel_at_period_end: true
    }
  });
}

async function handleSubscriptionSuspended(event: any) {
  const subscriptionId = event.resource.id;
  
  await prisma.subscription.update({
    where: { paypal_subscription_id: subscriptionId },
    data: { status: 'suspended' }
  });
  
  // Optionally suspend user account
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId }
  });
  
  if (subscription) {
    await prisma.user.update({
      where: { id: subscription.user_id },
      data: { status: 'suspended' }
    });
  }
}
```

### 3. Webhook Endpoint Validation

```typescript
import axios from 'axios';

// Validate callback URL before accepting job
export async function validateCallbackUrl(url: string): Promise<boolean> {
  try {
    // Parse URL
    const parsedUrl = new URL(url);
    
    // Only allow HTTPS in production
    if (process.env.NODE_ENV === 'production' && parsedUrl.protocol !== 'https:') {
      return false;
    }
    
    // Block localhost/internal IPs in production
    if (process.env.NODE_ENV === 'production') {
      const hostname = parsedUrl.hostname;
      if (
        hostname === 'localhost' ||
        hostname === '127.0.0.1' ||
        hostname.startsWith('192.168.') ||
        hostname.startsWith('10.') ||
        hostname.startsWith('172.')
      ) {
        return false;
      }
    }
    
    // Optional: Send test webhook to verify endpoint
    const testPayload = {
      test: true,
      timestamp: Date.now()
    };
    
    const response = await axios.post(url, testPayload, {
      timeout: 5000,
      validateStatus: (status) => status >= 200 && status < 500
    });
    
    return response.status >= 200 && response.status < 300;
    
  } catch (error) {
    return false;
  }
}

// Use in job submission endpoint
router.post('/jobs', requireApiKey, async (req, res) => {
  const { callback_url } = req.body;
  
  if (callback_url) {
    const isValid = await validateCallbackUrl(callback_url);
    
    if (!isValid) {
      return res.status(400).json({
        error: 'Invalid callback URL',
        details: 'URL must be HTTPS and publicly accessible'
      });
    }
  }
  
  // Continue with job creation...
});
```

### 4. Webhook Retry Logic

```typescript
// Exponential backoff calculation
function calculateBackoffDelay(attemptNumber: number): number {
  const baseDelay = 5000; // 5 seconds
  const maxDelay = 300000; // 5 minutes
  
  const delay = baseDelay * Math.pow(2, attemptNumber - 1);
  return Math.min(delay, maxDelay);
}

// Example retry schedule:
// Attempt 1: Immediate
// Attempt 2: 5 seconds
// Attempt 3: 10 seconds
// Attempt 4: 20 seconds
// Attempt 5: 40 seconds

// Webhook queue configuration (see queue-management skill)
import Queue from 'bull';

export const webhookQueue = new Queue('webhook-deliveries', {
  redis: { host: process.env.REDIS_HOST, port: 6379 },
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 5000
    }
  }
});
```

### 5. Webhook Status Endpoint

```typescript
// Check webhook delivery status
router.get('/jobs/:jobId/webhooks', requireApiKey, async (req: AuthRequest, res) => {
  const { jobId } = req.params;
  const userId = req.user!.userId;
  
  // Verify job ownership
  const job = await prisma.translationJob.findFirst({
    where: {
      id: jobId,
      user_id: userId
    },
    include: {
      webhook_deliveries: {
        orderBy: { attempted_at: 'desc' }
      }
    }
  });
  
  if (!job) {
    return res.status(404).json({ error: 'Job not found' });
  }
  
  const deliveries = job.webhook_deliveries.map(delivery => ({
    attempt: delivery.attempt_number,
    success: delivery.success,
    status: delivery.http_status,
    error: delivery.error_message,
    timestamp: delivery.attempted_at
  }));
  
  res.json({
    job_id: jobId,
    callback_url: job.callback_url,
    deliveries
  });
});
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Blocking HTTP request while sending webhook | Use queue for async delivery |
| No timeout on webhook requests | Set 30 second timeout |
| Infinite retries | Limit to 5 attempts max |
| Retrying 4xx client errors | Only retry 5xx and network errors |
| No signature verification | Use HMAC-SHA256 signatures |
| Trusting `X-Forwarded-For` header | Verify request origin properly |
| No timestamp validation | Check timestamp within 5 minute window |
| Storing webhook secrets in plain text | Use environment variables |
| No delivery tracking | Log all attempts to database |
| Allowing HTTP callbacks in production | Require HTTPS only |

## Integration with Other Skills

**Often combined with:**
- `queue-management` - Async webhook delivery with retries
- `authentication-security` - HMAC signature generation/verification
- `api-endpoint-creation` - Webhook receiver endpoints
- `error-handling-logging` - Logging failed deliveries

**Depends on:**
- `database-schema-design` - WebhookDelivery, TranslationJob tables

## Environment Variables Required

```bash
# PayPal Configuration
PAYPAL_CLIENT_ID="your-paypal-client-id"
PAYPAL_CLIENT_SECRET="your-paypal-secret"
PAYPAL_WEBHOOK_ID="your-webhook-id"
PAYPAL_WEBHOOK_SECRET="your-webhook-secret"
PAYPAL_MODE="sandbox"  # or "live"

# Webhook Configuration
WEBHOOK_TIMEOUT_MS=30000
WEBHOOK_MAX_RETRIES=5
```

## Quick Reference

### Webhook Headers (Outgoing)

```
Content-Type: application/json
X-Webhook-Signature: {hmac-sha256-hex}
X-Webhook-Timestamp: {unix-timestamp-ms}
User-Agent: TranslatePressZone/1.0
```

### Webhook Payload Format

```json
{
  "job_id": "uuid",
  "client_job_id": "wp_12345",
  "status": "completed",
  "translation": "<p>Translated content</p>",
  "tokens_used": 145,
  "cost_usd": 0.0000725
}
```

### PayPal Event Types

| Event | Description | Action |
|-------|-------------|--------|
| `BILLING.SUBSCRIPTION.CREATED` | Subscription created | Store subscription ID |
| `BILLING.SUBSCRIPTION.ACTIVATED` | Subscription activated | Activate account, allocate credits |
| `PAYMENT.SALE.COMPLETED` | Payment received | Record payment, extend period |
| `BILLING.SUBSCRIPTION.CANCELLED` | Subscription cancelled | Mark for cancellation at period end |
| `BILLING.SUBSCRIPTION.SUSPENDED` | Payment failed | Suspend account |

### HTTP Status Codes

| Code | Meaning | Retry? |
|------|---------|--------|
| 200-299 | Success | No |
| 400-499 | Client error | No (permanent failure) |
| 500-599 | Server error | Yes (temporary failure) |
| Network error | Connection failed | Yes |

## Validation Checklist

Before completing webhook implementation:

- [ ] HMAC-SHA256 signatures generated for outgoing webhooks
- [ ] Signature verification implemented for incoming webhooks
- [ ] Timestamp validation prevents replay attacks (5min window)
- [ ] Webhook timeout set to 30 seconds
- [ ] Exponential backoff configured (5 attempts max)
- [ ] 4xx errors don't trigger retries
- [ ] All delivery attempts logged to database
- [ ] HTTPS required for callback URLs in production
- [ ] Internal IPs blocked for callback URLs
- [ ] PayPal signature verification uses cert validation
- [ ] PayPal webhook secret stored securely
- [ ] Webhook delivery status endpoint implemented
- [ ] Graceful handling of missing callback_url
- [ ] Test mode webhook endpoint available
- [ ] Webhook documentation provided to WordPress plugin

## Security Considerations

### SSRF Prevention
```typescript
// Block internal IPs for callbacks
const blockedHosts = [
  'localhost',
  '127.0.0.1',
  '0.0.0.0',
  '169.254.169.254', // AWS metadata
  '::1',
  'metadata.google.internal' // GCP metadata
];

function isInternalIp(hostname: string): boolean {
  return blockedHosts.includes(hostname) ||
    hostname.startsWith('192.168.') ||
    hostname.startsWith('10.') ||
    hostname.match(/^172\.(1[6-9]|2\d|3[01])\./);
}
```

### Rate Limiting Webhook Endpoints
```typescript
import rateLimit from 'express-rate-limit';

const webhookLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 100, // 100 webhooks per minute
  message: 'Too many webhook requests'
});

router.post('/webhooks/paypal', webhookLimiter, verifyPayPalWebhook, handler);
```

## Testing Webhooks

### Development Webhook Testing
```bash
# Use ngrok for local testing
ngrok http 3000

# Test webhook delivery
curl -X POST http://localhost:3000/jobs \
  -H "Authorization: Bearer sk_test_..." \
  -H "Content-Type: application/json" \
  -d '{
    "source_lang": "en",
    "target_lang": "es",
    "content": "Hello",
    "model": "4b",
    "callback_url": "https://your-ngrok-url.ngrok.io/callback"
  }'
```

### Mock PayPal Webhook
```typescript
// Test route (development only)
if (process.env.NODE_ENV === 'development') {
  router.post('/test/paypal-webhook', async (req, res) => {
    const event = {
      event_type: 'PAYMENT.SALE.COMPLETED',
      resource: {
        id: 'test_payment_123',
        amount: { total: '29.00' },
        billing_agreement_id: 'test_subscription_123'
      }
    };
    
    await handlePaymentCompleted(event);
    res.json({ success: true });
  });
}
```

---

# Appendix A: Operations Runbook

## Overview

This appendix provides operational procedures for production deployment, database management, monitoring, and incident response. These procedures are essential for maintaining system reliability and data integrity.

## Database Backup & Restore

### Architecture

The Press Zone Backend uses PostgreSQL as the primary database and Redis for caching/session storage. Both require regular backups with different strategies:

- **PostgreSQL**: Full database dumps using `pg_dump` with custom format
- **Redis**: Snapshot-based backups using `SAVE` command
- **Retention**: 30-day rolling retention policy
- **Schedule**: Daily automated backups at 2:00 AM
- **Storage**: Local filesystem with compression

### Backup Script

**Location**: `backup/backup.sh`

#### Full Script

```bash
#!/bin/bash
# backup.sh - Backup PostgreSQL database and Redis data
# Run this as user 'press' (cron recommended)

set -e

# Configuration
BACKUP_DIR="$HOME/press-zone-backend/backup"
RETENTION_DAYS=30  # Keep backups for 30 days
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Database credentials (from .env)
DB_USER="translate_user"
DB_NAME="translate_db"

# Create backup directory
mkdir -p "$BACKUP_DIR"

echo "Starting backup at $(date)"

# Backup PostgreSQL
echo "Backing up PostgreSQL database..."
pg_dump -U "$DB_USER" -d "$DB_NAME" -F c -f "$BACKUP_DIR/postgres_${TIMESTAMP}.dump" || {
    echo "ERROR: PostgreSQL backup failed"
    exit 1
}

# Backup Redis
echo "Backing up Redis data..."
redis-cli SAVE
cp /var/lib/redis/dump.rdb "$BACKUP_DIR/redis_${TIMESTAMP}.rdb" 2>/dev/null || {
    echo "WARNING: Redis backup failed (may need sudo)"
}

# Compress backups
echo "Compressing backups..."
gzip "$BACKUP_DIR/postgres_${TIMESTAMP}.dump"
[ -f "$BACKUP_DIR/redis_${TIMESTAMP}.rdb" ] && gzip "$BACKUP_DIR/redis_${TIMESTAMP}.rdb"

# Calculate sizes
PG_SIZE=$(du -h "$BACKUP_DIR/postgres_${TIMESTAMP}.dump.gz" | cut -f1)
echo "PostgreSQL backup size: $PG_SIZE"

# Remove old backups
echo "Removing backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "postgres_*.dump.gz" -mtime +$RETENTION_DAYS -delete
find "$BACKUP_DIR" -name "redis_*.rdb.gz" -mtime +$RETENTION_DAYS -delete

# List recent backups
echo ""
echo "Recent backups:"
ls -lh "$BACKUP_DIR" | tail -10

echo "Backup completed successfully at $(date)"
```

#### Script Components Explained

**1. PostgreSQL Backup (`pg_dump`)**

```bash
pg_dump -U "$DB_USER" -d "$DB_NAME" -F c -f "$BACKUP_DIR/postgres_${TIMESTAMP}.dump"
```

- **`-U "$DB_USER"`**: Specifies database user (authenticate via `.pgpass` or environment)
- **`-d "$DB_NAME"`**: Target database name
- **`-F c`**: Custom format (compressed, allows selective restore, faster than SQL)
- **`-f`**: Output file path with timestamp

**Why Custom Format?**
- Smaller file size than plain SQL
- Allows selective table restore
- Built-in compression
- Faster restore with parallel processing
- Includes table structure + data + indexes

**2. Redis Backup**

```bash
redis-cli SAVE
cp /var/lib/redis/dump.rdb "$BACKUP_DIR/redis_${TIMESTAMP}.rdb"
```

- **`redis-cli SAVE`**: Forces synchronous save (blocks until complete)
- **Copy `dump.rdb`**: Redis's binary snapshot file
- **Error handling**: Graceful degradation if Redis backup fails (may need sudo for file access)

**Why SAVE instead of BGSAVE?**
- SAVE is synchronous and guarantees completion
- BGSAVE is async and may not complete before copy
- Trade-off: Brief Redis blocking acceptable during low-traffic backup window

**3. Compression Strategy**

```bash
gzip "$BACKUP_DIR/postgres_${TIMESTAMP}.dump"
gzip "$BACKUP_DIR/redis_${TIMESTAMP}.rdb"
```

- **gzip compression**: Reduces storage by 70-90%
- **PostgreSQL custom format**: Already compressed, but gzip adds 20-30% additional savings
- **Redis RDB**: Uncompressed binary, gzip provides 80-90% reduction

**4. Retention Policy**

```bash
find "$BACKUP_DIR" -name "postgres_*.dump.gz" -mtime +$RETENTION_DAYS -delete
```

- **30-day retention**: Balances storage costs with recovery flexibility
- **Automatic cleanup**: Runs after each backup
- **Pattern matching**: Only deletes backup files, preserves scripts

**5. Error Handling**

```bash
pg_dump ... || {
    echo "ERROR: PostgreSQL backup failed"
    exit 1
}
```

- **Critical failures**: PostgreSQL backup failure stops script immediately
- **Non-critical failures**: Redis backup failure logs warning but continues
- **Exit codes**: Non-zero exit triggers cron email notification

### Restore Script

**Location**: `backup/restore.sh`

#### Full Script

```bash
#!/bin/bash
# restore.sh - Restore PostgreSQL database from backup
# Run this as user 'press' with backup filename as argument

set -e

if [ $# -eq 0 ]; then
    echo "Usage: $0 <backup_file>"
    echo ""
    echo "Available backups:"
    ls -lh "$HOME/press-zone-backend/backup" | grep postgres
    exit 1
fi

BACKUP_FILE="$1"
DB_USER="translate_user"
DB_NAME="translate_db"

if [ ! -f "$BACKUP_FILE" ]; then
    echo "ERROR: Backup file not found: $BACKUP_FILE"
    exit 1
fi

echo "WARNING: This will restore database '$DB_NAME' from backup."
echo "Current data will be OVERWRITTEN!"
echo ""
read -p "Are you sure? (type 'yes' to continue): " confirm

if [ "$confirm" != "yes" ]; then
    echo "Restore cancelled."
    exit 0
fi

# Stop backend services
echo "Stopping backend services..."
systemctl --user stop presszone-backend.service || true

# Decompress if needed
if [[ "$BACKUP_FILE" == *.gz ]]; then
    echo "Decompressing backup..."
    TEMP_FILE="/tmp/restore_temp.dump"
    gunzip -c "$BACKUP_FILE" > "$TEMP_FILE"
    RESTORE_FILE="$TEMP_FILE"
else
    RESTORE_FILE="$BACKUP_FILE"
fi

# Drop and recreate database
echo "Dropping existing database..."
dropdb -U "$DB_USER" "$DB_NAME" --if-exists

echo "Creating new database..."
createdb -U "$DB_USER" "$DB_NAME"

# Restore backup
echo "Restoring database from backup..."
pg_restore -U "$DB_USER" -d "$DB_NAME" -F c "$RESTORE_FILE" || {
    echo "ERROR: Database restore failed"
    exit 1
}

# Clean up temp file
[ -f "$TEMP_FILE" ] && rm "$TEMP_FILE"

# Start backend services
echo "Starting backend services..."
systemctl --user start presszone-backend.service

echo "Database restored successfully!"
echo "Waiting for services to start..."
sleep 5

# Health check
if curl -f http://localhost:3000/health > /dev/null 2>&1; then
    echo "✓ Services are running correctly"
else
    echo "WARNING: Health check failed, please check service logs"
    systemctl --user status presszone-backend.service
fi
```

#### Restore Workflow

**1. Interactive Confirmation**

```bash
read -p "Are you sure? (type 'yes' to continue): " confirm
if [ "$confirm" != "yes" ]; then
    echo "Restore cancelled."
    exit 0
fi
```

- **Explicit confirmation**: User must type "yes" (not just "y")
- **Prevents accidents**: No automatic restore, requires human decision
- **Reversible**: Cancelled restore has zero impact

**2. Service Shutdown**

```bash
systemctl --user stop presszone-backend.service || true
```

- **Graceful shutdown**: Stops API server to prevent connection errors during restore
- **Non-blocking**: `|| true` ensures script continues even if service not running
- **User-level systemd**: Uses `--user` flag for non-root service management

**3. Decompression Handling**

```bash
if [[ "$BACKUP_FILE" == *.gz ]]; then
    gunzip -c "$BACKUP_FILE" > "$TEMP_FILE"
    RESTORE_FILE="$TEMP_FILE"
else
    RESTORE_FILE="$BACKUP_FILE"
fi
```

- **Automatic detection**: Checks file extension for `.gz`
- **Stream decompression**: `gunzip -c` streams to temp file (doesn't modify original)
- **Cleanup**: Temp file removed after restore completes

**4. Database Recreation**

```bash
dropdb -U "$DB_USER" "$DB_NAME" --if-exists
createdb -U "$DB_USER" "$DB_NAME"
```

- **Clean slate**: Drops existing database to avoid conflicts
- **Safe drop**: `--if-exists` prevents error if database doesn't exist
- **New database**: Fresh database with default encoding/collation

**5. Restore Execution**

```bash
pg_restore -U "$DB_USER" -d "$DB_NAME" -F c "$RESTORE_FILE"
```

- **`-F c`**: Specifies custom format (matches backup format)
- **Target database**: Restores into newly created database
- **Error handling**: Exits immediately if restore fails

**6. Health Check Validation**

```bash
if curl -f http://localhost:3000/health > /dev/null 2>&1; then
    echo "✓ Services are running correctly"
else
    echo "WARNING: Health check failed, please check service logs"
    systemctl --user status presszone-backend.service
fi
```

- **Automated verification**: Calls `/health` endpoint after 5-second startup delay
- **Silent check**: Output redirected to `/dev/null`
- **Failure response**: Shows service status for troubleshooting

### Cron Configuration

**Location**: `backup/crontab.example`

#### Crontab Setup

```bash
# Daily backup at 2:00 AM
0 2 * * * $HOME/press-zone-backend/backup/backup.sh >> $HOME/press-zone-backend/backup/backup.log 2>&1

# Weekly cleanup of old logs (every Sunday at 3:00 AM)
0 3 * * 0 find $HOME/press-zone-backend/api/logs -name "*.log" -mtime +30 -delete
```

#### Installation Steps

```bash
# Edit crontab as user 'press'
crontab -e

# Add the lines from crontab.example
# Save and exit

# Verify crontab installation
crontab -l

# Check if cron service is running
systemctl status cron
```

#### Cron Schedule Breakdown

**1. Daily Backup (2:00 AM)**

```
0 2 * * *
│ │ │ │ │
│ │ │ │ └─── Day of week (0-7, Sunday = 0 or 7)
│ │ │ └───── Month (1-12)
│ │ └─────── Day of month (1-31)
│ └───────── Hour (0-23)
└─────────── Minute (0-59)
```

- **Why 2:00 AM?**: Low-traffic period for most applications
- **Time zone**: Uses server's local time (verify with `date`)
- **Execution**: Runs every day at exactly 2:00 AM

**2. Output Redirection**

```bash
>> $HOME/press-zone-backend/backup/backup.log 2>&1
```

- **`>>`**: Append to log file (doesn't overwrite)
- **`2>&1`**: Redirect stderr to stdout (captures errors)
- **Log rotation**: Manual cleanup or logrotate configuration needed

**3. Weekly Log Cleanup (Sunday 3:00 AM)**

```bash
0 3 * * 0 find $HOME/press-zone-backend/api/logs -name "*.log" -mtime +30 -delete
```

- **Sunday 3:00 AM**: Runs after Saturday night backup completes
- **30-day retention**: Matches backup retention policy
- **Pattern**: Only deletes `.log` files, preserves other files

#### Email Notifications

Cron automatically sends email on script failures (non-zero exit codes).

**Setup Email Notifications**

```bash
# Install mail utilities
sudo apt install mailutils

# Configure email in crontab
MAILTO=admin@press.zone

# Cron will email script output on failure
0 2 * * * $HOME/press-zone-backend/backup/backup.sh
```

**Alternative: Healthchecks.io Integration**

```bash
# Add healthchecks.io ping to backup script
HEALTHCHECK_URL="https://hc-ping.com/your-uuid-here"

# Success ping
curl -fsS -m 10 --retry 3 "$HEALTHCHECK_URL" > /dev/null
```

#### Log Rotation Setup

**Create logrotate config**: `/etc/logrotate.d/presszone-backup`

```
/home/press/press-zone-backend/backup/backup.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0644 press press
}
```

### Disaster Recovery Procedures

#### Scenario 1: Database Corruption

**Symptoms:**
- API returns 500 errors
- PostgreSQL logs show corruption errors
- `systemctl --user status presszone-backend.service` shows database connection failures

**Recovery Steps:**

```bash
# 1. Identify latest backup
ls -lh ~/press-zone-backend/backup/postgres_*.dump.gz | tail -5

# 2. Stop services
systemctl --user stop presszone-backend.service

# 3. Verify backup integrity
gunzip -t ~/press-zone-backend/backup/postgres_20260127_020001.dump.gz
# (Should complete silently if backup is valid)

# 4. Restore from backup
cd ~/press-zone-backend/backup
./restore.sh postgres_20260127_020001.dump.gz

# 5. Verify health
curl http://localhost:3000/health

# 6. Check recent data
psql -U translate_user -d translate_db -c "SELECT COUNT(*) FROM jobs WHERE created_at > NOW() - INTERVAL '1 day';"
```

**Data Loss Window:**
- Maximum loss: Time since last backup (up to 24 hours if using daily backups)
- Mitigation: Increase backup frequency for critical systems (every 6 hours)

#### Scenario 2: Accidental Data Deletion

**Symptoms:**
- User reports missing translation jobs
- Admin accidentally deleted records
- Application bug deleted data

**Recovery Steps:**

```bash
# 1. IMMEDIATELY stop writes
systemctl --user stop presszone-backend.service

# 2. Find backup BEFORE deletion
# Check application logs for deletion timestamp
journalctl --user -u presszone-backend.service -S "2026-01-27 10:00" | grep DELETE

# 3. Identify backup before deletion
ls -lh ~/press-zone-backend/backup/postgres_*.dump.gz

# 4. Restore to temporary database for verification
createdb -U translate_user translate_db_recovery
gunzip -c ~/press-zone-backend/backup/postgres_20260127_020001.dump.gz | \
  pg_restore -U translate_user -d translate_db_recovery -F c

# 5. Extract specific data
psql -U translate_user -d translate_db_recovery -c \
  "COPY (SELECT * FROM jobs WHERE id = '123') TO '/tmp/recovered_data.csv' CSV HEADER;"

# 6. Restore specific records to production
psql -U translate_user -d translate_db -c \
  "\COPY jobs FROM '/tmp/recovered_data.csv' CSV HEADER;"

# 7. Drop recovery database
dropdb -U translate_user translate_db_recovery

# 8. Restart services
systemctl --user start presszone-backend.service
```

#### Scenario 3: Complete Server Failure

**Symptoms:**
- Server hardware failure
- Disk failure
- Catastrophic system crash

**Recovery Steps:**

**Prerequisites:**
- Offsite backup storage (rsync to remote server or S3)
- Documented server configuration (see Appendix B: Environment Variables)

**Step 1: Provision New Server**

```bash
# Provision new Ubuntu 22.04 LTS server
# Install required packages
sudo apt update && sudo apt install -y \
  postgresql-15 redis-server nodejs npm nginx curl

# Create 'press' user
sudo useradd -m -s /bin/bash press
sudo usermod -aG sudo press
```

**Step 2: Restore Application Code**

```bash
# Clone repository (or restore from backup)
su - press
git clone https://github.com/presszone/backend.git press-zone-backend
cd press-zone-backend/api
npm install
```

**Step 3: Restore Configuration**

```bash
# Restore .env file from secure backup
# (NEVER commit .env to git!)
cp /path/to/secure/backup/.env ~/press-zone-backend/api/.env

# Verify environment variables
cd ~/press-zone-backend/api
node -e "require('dotenv').config(); console.log('DB:', process.env.DATABASE_URL)"
```

**Step 4: Restore Database**

```bash
# Download latest backup from offsite storage
scp backup-server:/backups/postgres_latest.dump.gz ~/press-zone-backend/backup/

# Restore database
cd ~/press-zone-backend/backup
./restore.sh postgres_latest.dump.gz
```

**Step 5: Configure Services**

```bash
# Install systemd service
cp ~/press-zone-backend/systemd/presszone-backend.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable presszone-backend.service
systemctl --user start presszone-backend.service
```

**Step 6: Verify System**

```bash
# Health check
curl http://localhost:3000/health

# Test authentication
curl -X POST http://localhost:3000/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@press.zone","password":"..."}'

# Verify data integrity
psql -U translate_user -d translate_db -c "SELECT COUNT(*) FROM jobs;"
```

**Recovery Time Objective (RTO):**
- With documented procedures: 2-4 hours
- Without documentation: 8-24 hours

**Recovery Point Objective (RPO):**
- With daily backups: Up to 24 hours data loss
- With 6-hour backups: Up to 6 hours data loss

### Testing Disaster Recovery

**CRITICAL:** Test recovery procedures quarterly to ensure backups are valid.

#### Recovery Test Procedure

```bash
# 1. Create test environment
sudo -u postgres createdb translate_db_test

# 2. Restore latest backup to test database
gunzip -c ~/press-zone-backend/backup/postgres_20260127_020001.dump.gz | \
  pg_restore -U translate_user -d translate_db_test -F c

# 3. Verify table counts match production
psql -U translate_user -d translate_db -c "SELECT COUNT(*) FROM jobs;" > /tmp/prod_count.txt
psql -U translate_user -d translate_db_test -c "SELECT COUNT(*) FROM jobs;" > /tmp/test_count.txt
diff /tmp/prod_count.txt /tmp/test_count.txt

# 4. Verify critical data
psql -U translate_user -d translate_db_test -c "SELECT * FROM jobs ORDER BY created_at DESC LIMIT 10;"

# 5. Test application startup with test database
cd ~/press-zone-backend/api
DATABASE_URL="postgresql://translate_user@localhost/translate_db_test" npm start

# 6. Clean up
dropdb -U translate_user translate_db_test
```

#### Backup Validation Checklist

- [ ] Backup files are created daily
- [ ] Backup files are compressed and under expected size
- [ ] Retention policy deletes backups older than 30 days
- [ ] Cron email notifications are received (or healthchecks.io pings)
- [ ] `pg_restore` completes without errors on test database
- [ ] Table counts match between backup and production
- [ ] Application starts successfully with restored database
- [ ] Health check endpoint returns 200 OK
- [ ] Critical queries return expected data

### Rollback Procedures

#### If Restore Fails

**Scenario:** `pg_restore` fails midway through restore

```bash
# 1. Check PostgreSQL logs
sudo journalctl -u postgresql -n 100

# 2. Identify error (common issues)
# - Disk space exhausted
# - Permission denied
# - Incompatible PostgreSQL version
# - Corrupted backup file

# 3. Drop failed restore
dropdb -U translate_user translate_db --if-exists

# 4. Try older backup
ls -lh ~/press-zone-backend/backup/postgres_*.dump.gz | tail -10
./restore.sh postgres_20260126_020001.dump.gz

# 5. If all backups fail, check backup integrity
for backup in ~/press-zone-backend/backup/postgres_*.dump.gz; do
  echo "Testing $backup..."
  gunzip -t "$backup" && echo "✓ Valid" || echo "✗ Corrupted"
done
```

#### If Services Won't Start After Restore

```bash
# 1. Check service status
systemctl --user status presszone-backend.service

# 2. Check application logs
journalctl --user -u presszone-backend.service -n 100

# 3. Common issues:
# - Database connection failure (check DATABASE_URL in .env)
# - Missing dependencies (run npm install)
# - Port already in use (check for zombie processes)
# - File permissions (ensure 'press' user owns files)

# 4. Manual startup for debugging
cd ~/press-zone-backend/api
npm start
# (Check console output for errors)

# 5. If database schema issues
cd ~/press-zone-backend/api
npx prisma migrate deploy
# (Applies any missing migrations)
```

#### Emergency Contact Escalation

If disaster recovery fails after 2 hours:

1. **Level 1**: Contact DevOps team lead
2. **Level 2**: Contact database administrator
3. **Level 3**: Contact senior backend engineer
4. **Level 4**: Engage disaster recovery vendor (if contracted)

**Communication Template:**

```
Subject: [CRITICAL] Press Zone Backend Recovery Failure

Severity: P1 (Production Down)
Impact: Translation API unavailable, affecting X WordPress sites
Started: [Timestamp]
Failed Recovery Attempts:
  - Attempt 1: [Description + error]
  - Attempt 2: [Description + error]

Current Status:
  - Database: [Status]
  - API Service: [Status]
  - Last Known Good Backup: [Timestamp]

Requested Action: [Specific help needed]
```

### Offsite Backup Strategy

**CRITICAL:** Local backups are insufficient for disaster recovery. Implement offsite storage.

#### Option 1: Rsync to Remote Server

```bash
# Add to backup.sh after compression
REMOTE_SERVER="backup-server.press.zone"
REMOTE_PATH="/backups/presszone/"

rsync -avz --progress \
  ~/press-zone-backend/backup/postgres_${TIMESTAMP}.dump.gz \
  ${REMOTE_SERVER}:${REMOTE_PATH}
```

#### Option 2: AWS S3 Upload

```bash
# Install AWS CLI
sudo apt install awscli

# Configure credentials
aws configure

# Add to backup.sh
aws s3 cp \
  ~/press-zone-backend/backup/postgres_${TIMESTAMP}.dump.gz \
  s3://presszone-backups/postgres/ \
  --storage-class STANDARD_IA
```

#### Option 3: Encrypted Backup to Cloud Storage

```bash
# Encrypt before upload (GPG)
gpg --symmetric --cipher-algo AES256 \
  ~/press-zone-backend/backup/postgres_${TIMESTAMP}.dump.gz

# Upload encrypted file
rclone copy \
  ~/press-zone-backend/backup/postgres_${TIMESTAMP}.dump.gz.gpg \
  remote:presszone-backups/
```

### Backup Best Practices

1. **3-2-1 Rule**:
   - 3 copies of data
   - 2 different storage media
   - 1 offsite copy

2. **Test Restores Quarterly**:
   - Schedule recovery drills
   - Document recovery time
   - Identify bottlenecks

3. **Monitor Backup Success**:
   - Healthchecks.io or similar
   - Email notifications on failure
   - Dashboard for backup status

4. **Secure Backup Storage**:
   - Encrypt offsite backups
   - Restrict access (IAM policies)
   - Audit backup access logs

5. **Document Recovery Procedures**:
   - Runbook with step-by-step instructions
   - Contact escalation paths
   - Known issues and workarounds

6. **Automate Recovery Testing**:
   - Scripted restore to staging environment
   - Automated data validation
   - CI/CD integration for backup verification
# Skill 13: Token Estimation & Pricing

## Identity
- **Skill ID**: `token-estimation-ml`
- **Domain**: ML-Based Token Prediction, Pricing Estimation
- **Technologies**: TypeScript, Prisma, PostgreSQL, Statistical ML
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Token estimation algorithms
- Language-specific efficiency factors
- HTML complexity analysis
- ML-based accuracy improvement
- Pricing calculations for translations
- Confidence level determination
- Cost prediction for multi-language jobs

**File patterns:**
- `api/src/services/TokenEstimator.ts`
- `api/src/utils/tokenCalculation.ts`
- `api/src/routes/estimate.ts`
- `api/prisma/schema.prisma` (accuracy_stats, translation_records)

## Core Patterns

### 1. Five-Layer Estimation Architecture

The TokenEstimator uses a sophisticated 5-layer system to predict token consumption with increasing accuracy over time:

```
┌─────────────────────────────────────────────────────────────┐
│ Layer 1: Base Tokenization (char_count ÷ 4)                │
├─────────────────────────────────────────────────────────────┤
│ Layer 2: Language Factor (0.60x - 1.28x multiplier)        │
├─────────────────────────────────────────────────────────────┤
│ Layer 3: HTML Complexity (0-15% overhead)                   │
├─────────────────────────────────────────────────────────────┤
│ Layer 4: Safety Buffer (5-8%)                               │
├─────────────────────────────────────────────────────────────┤
│ Layer 5: ML Adjustment (learned from historical data)       │
└─────────────────────────────────────────────────────────────┘
```

### 2. Layer 1: Base Tokenization

**Rule of thumb**: 1 token ≈ 4 characters for English text.

```typescript
private estimateBaseTokens(content: string): number {
  return Math.ceil(content.length / 4);
}
```

**Example:**
- Input: "Hello, world!" (13 chars)
- Base tokens: ceil(13 / 4) = 4 tokens

### 3. Layer 2: Language Efficiency Factors

Each of the 131 supported languages has a unique efficiency multiplier based on linguistic density:

```typescript
private getLanguageFactor(targetLang: string): number {
  const factors: Record<string, number> = {
    // CJK languages (most efficient - fewer tokens per meaning)
    'zh': 0.60,      // Chinese (simplified/traditional)
    'ja': 0.65,      // Japanese
    'ko': 0.70,      // Korean

    // Compact languages
    'th': 0.75,      // Thai
    'vi': 0.80,      // Vietnamese
    'ar': 0.85,      // Arabic
    'he': 0.85,      // Hebrew

    // Average efficiency (Romance languages)
    'en': 1.00,      // English (baseline)
    'es': 1.05,      // Spanish
    'fr': 1.08,      // French
    'it': 1.06,      // Italian
    'pt': 1.05,      // Portuguese

    // Verbose languages (Germanic)
    'de': 1.20,      // German
    'nl': 1.15,      // Dutch
    'sv': 1.12,      // Swedish
    'no': 1.12,      // Norwegian

    // Most verbose (Agglutinative languages)
    'tr': 1.28,      // Turkish (highest multiplier)
    'fi': 1.25,      // Finnish
    'hu': 1.22,      // Hungarian

    // ... 131 languages total
  };

  return factors[targetLang.toLowerCase()] || 1.00;
}
```

**Language Categories:**

| Category | Factor Range | Example Languages | Reason |
|----------|-------------|-------------------|---------|
| CJK | 0.60-0.70 | Chinese, Japanese, Korean | Logographic/syllabic writing, high information density |
| Semitic | 0.85-0.90 | Arabic, Hebrew, Urdu | Consonant roots, compact morphology |
| Indic | 0.75-0.85 | Hindi, Tamil, Bengali | Abugida scripts, compact representation |
| Romance | 1.05-1.08 | Spanish, French, Italian | Moderate verbosity |
| Germanic | 1.12-1.20 | German, Dutch, Swedish | Compound words, longer constructions |
| Agglutinative | 1.20-1.28 | Turkish, Finnish, Hungarian | Multiple affixes per word |

**Example Calculation:**
```typescript
// English to German translation
const content = "Hello, world!"; // 13 chars
const baseTokens = 4;            // From Layer 1
const langFactor = 1.20;         // German is verbose
const adjusted = Math.ceil(4 * 1.20); // = 5 tokens
```

### 4. Layer 3: HTML Complexity Analysis

HTML tags and attributes increase token consumption due to preserved formatting:

```typescript
public analyzeHTMLComplexity(content: string): {
  level: 'none' | 'light' | 'medium' | 'heavy';
  overhead: number;
} {
  const htmlTagCount = (content.match(/<[^>]+>/g) || []).length;
  const totalLength = content.length;
  const htmlRatio = htmlTagCount > 0 ? (htmlTagCount * 10) / totalLength : 0;

  if (htmlRatio === 0) {
    return { level: 'none', overhead: 0 };
  } else if (htmlRatio < 0.05) {
    return { level: 'light', overhead: 0.05 };   // +5%
  } else if (htmlRatio < 0.15) {
    return { level: 'medium', overhead: 0.10 };  // +10%
  } else {
    return { level: 'heavy', overhead: 0.15 };   // +15%
  }
}
```

**Complexity Levels:**

| Level | HTML Ratio | Overhead | Example Content |
|-------|-----------|----------|-----------------|
| `none` | 0 | 0% | Plain text: "Hello, world!" |
| `light` | <5% | 5% | Simple markup: `<p>Hello</p>` |
| `medium` | 5-15% | 10% | Structured content: `<div><h1>Title</h1><p>Text</p></div>` |
| `heavy` | >15% | 15% | Rich formatting: tables, nested divs, inline styles |

**Example:**
```html
<!-- Heavy HTML complexity -->
<div class="post">
  <h1 class="title">Welcome</h1>
  <p style="color: blue;">This is <strong>important</strong> text.</p>
  <table>
    <tr><td>Cell 1</td><td>Cell 2</td></tr>
  </table>
</div>
```
- HTML tags: 15
- Total length: 200 chars
- HTML ratio: (15 * 10) / 200 = 0.75 (75%)
- Complexity: `heavy` (15% overhead)

### 5. Layer 4: Safety Buffer

Ensures estimates are never lower than actual usage:

```typescript
private calculateSafetyBuffer(
  baseTokens: number,
  complexity: ReturnType<typeof this.analyzeHTMLComplexity>
): number {
  const baseBuffer = 0.05; // 5% minimum
  const complexityBonus = complexity.level === 'heavy' ? 0.03 : 0.00; // +3% for heavy HTML
  return Math.ceil(baseTokens * (baseBuffer + complexityBonus));
}
```

**Buffer Levels:**
- **Normal content**: 5%
- **Heavy HTML**: 8% (5% base + 3% bonus)

**Why necessary?**
- Edge cases in tokenization
- Unexpected formatting preservation
- Model-specific variations
- Better to slightly overestimate than undercharge

### 6. Layer 5: ML Adjustment (Continuous Learning)

The system learns from actual token usage to improve accuracy over time:

```typescript
private async getMLAdjustment(targetLang: string): Promise<number> {
  try {
    const stats = await prisma.$queryRaw<Array<{ ml_adjustment_ratio: number }>>`
      SELECT ml_adjustment_ratio
      FROM accuracy_stats
      WHERE language = ${targetLang}
      AND sample_count >= 10
      LIMIT 1
    `;

    if (stats && stats.length > 0) {
      return stats[0].ml_adjustment_ratio || 1.00;
    }
  } catch (error) {
    logger.warn('Failed to get ML adjustment', { targetLang, error });
  }

  return 1.00; // Default: no adjustment
}
```

**How ML adjustment works:**

1. **Data Collection**: Every completed translation records estimated vs. actual tokens
   ```sql
   INSERT INTO translation_records (
     target_lang,
     estimated_tokens,
     actual_tokens,
     html_complexity
   ) VALUES ('es', 120, 115, 'light');
   ```

2. **Aggregation**: System calculates adjustment ratio per language
   ```sql
   -- Example: Spanish shows consistent over-estimation
   SELECT
     language,
     COUNT(*) as sample_count,
     AVG(actual_tokens / estimated_tokens) as ml_adjustment_ratio
   FROM translation_records
   WHERE language = 'es'
   GROUP BY language;

   -- Result: ml_adjustment_ratio = 0.96 (we over-estimate by 4%)
   ```

3. **Application**: Future estimates use learned ratio
   ```typescript
   const subtotal = 120; // From layers 1-4
   const mlAdjustment = 0.96; // From historical data
   const finalTokens = Math.ceil(120 * 0.96); // = 115 tokens
   ```

**Database Schema:**

```prisma
model TranslationRecord {
  id                BigInt          @id @default(autoincrement())
  target_lang       String          @db.VarChar(10)
  estimated_tokens  Int
  actual_tokens     Int
  html_complexity   HTMLComplexity  @default(none)
  created_at        DateTime        @default(now())

  @@index([target_lang])
  @@index([created_at])
}

model AccuracyStats {
  id                   Int      @id @default(autoincrement())
  language             String   @unique @db.VarChar(10)
  sample_count         Int      @default(0)
  avg_error_percent    Decimal  @default(0.00) @db.Decimal(5, 2)
  ml_adjustment_ratio  Decimal  @default(1.0000) @db.Decimal(5, 4)
  updated_at           DateTime @default(now()) @updatedAt
}
```

### 7. Complete Estimation Flow

```typescript
async estimateSingle(
  content: string,
  _sourceLang: string,
  targetLang: string
): Promise<TokenEstimate> {
  // Layer 1: Base tokens
  const baseTokens = this.estimateBaseTokens(content);

  // Layer 2: Language factor
  const langFactor = this.getLanguageFactor(targetLang);
  const languageAdjusted = Math.ceil(baseTokens * langFactor);

  // Layer 3: HTML complexity
  const complexity = this.analyzeHTMLComplexity(content);
  const complexityTokens = Math.ceil(languageAdjusted * complexity.overhead);

  // Layer 4: Safety buffer
  const safetyBuffer = this.calculateSafetyBuffer(languageAdjusted, complexity);

  // Layer 5: ML adjustment
  const mlAdjustment = await this.getMLAdjustment(targetLang);

  // Calculate final estimate
  const subtotal = languageAdjusted + complexityTokens + safetyBuffer;
  const finalTokens = Math.ceil(subtotal * mlAdjustment);

  // Determine confidence level
  const sampleCount = await this.getSampleCount(targetLang);
  let confidence: 'low' | 'medium' | 'high' = 'low';
  if (sampleCount >= 100) confidence = 'high';
  else if (sampleCount >= 10) confidence = 'medium';

  return {
    estimated_tokens: finalTokens,
    confidence,
    breakdown: {
      base: baseTokens,
      languageFactor: languageAdjusted - baseTokens,
      htmlComplexity: complexityTokens,
      safetyBuffer,
      mlAdjustment: finalTokens - subtotal,
    },
  };
}
```

**Example with breakdown:**

```typescript
// Input: Translate "Hello, <strong>world</strong>!" from English to German
const content = "Hello, <strong>world</strong>!"; // 34 chars
const sourceLang = "en";
const targetLang = "de";

// Layer 1: Base
const baseTokens = ceil(34 / 4) = 9 tokens

// Layer 2: Language (German is verbose)
const langFactor = 1.20;
const languageAdjusted = ceil(9 * 1.20) = 11 tokens

// Layer 3: HTML (light complexity)
const htmlRatio = 0.03; // 3% (1 tag / 34 chars)
const complexityTokens = ceil(11 * 0.05) = 1 token

// Layer 4: Safety buffer
const safetyBuffer = ceil(11 * 0.05) = 1 token

// Subtotal: 11 + 1 + 1 = 13 tokens

// Layer 5: ML adjustment (assume German learned ratio = 0.98)
const finalTokens = ceil(13 * 0.98) = 13 tokens

// Result:
{
  estimated_tokens: 13,
  confidence: 'high', // Assuming 100+ German samples
  breakdown: {
    base: 9,
    languageFactor: 2,
    htmlComplexity: 1,
    safetyBuffer: 1,
    mlAdjustment: 0
  }
}
```

### 8. Batch Estimation

For multi-language translation quotes:

```typescript
async estimateBatch(
  content: string,
  sourceLang: string,
  targetLangs: string[]
): Promise<BatchEstimateResult> {
  const estimates: Record<string, TokenEstimate> = {};
  let totalTokens = 0;

  for (const targetLang of targetLangs) {
    const estimate = await this.estimateSingle(content, sourceLang, targetLang);
    estimates[targetLang] = estimate;
    totalTokens += estimate.estimated_tokens;
  }

  return {
    estimates,
    total_tokens: totalTokens,
    cache_hit: false, // Caching handled by WordPress layer
  };
}
```

**Example request:**
```typescript
const result = await tokenEstimator.estimateBatch(
  "Hello, world!",
  "en",
  ["es", "fr", "de", "zh", "ja"]
);

// Response:
{
  estimates: {
    es: { estimated_tokens: 4, confidence: 'high', breakdown: {...} },
    fr: { estimated_tokens: 5, confidence: 'high', breakdown: {...} },
    de: { estimated_tokens: 5, confidence: 'high', breakdown: {...} },
    zh: { estimated_tokens: 3, confidence: 'medium', breakdown: {...} },
    ja: { estimated_tokens: 3, confidence: 'medium', breakdown: {...} }
  },
  total_tokens: 20,
  cache_hit: false
}
```

### 9. Confidence Levels

Confidence is determined by sample size in the ML training data:

```typescript
private async getSampleCount(language: string): Promise<number> {
  try {
    const result = await prisma.$queryRaw<Array<{ sample_count: number }>>`
      SELECT sample_count
      FROM accuracy_stats
      WHERE language = ${language}
      LIMIT 1
    `;

    return result && result.length > 0 ? result[0].sample_count : 0;
  } catch {
    return 0;
  }
}
```

**Confidence Thresholds:**

| Confidence | Sample Count | Meaning | Expected Accuracy |
|------------|--------------|---------|-------------------|
| `low` | 0-9 | New language, statistical factors only | ±15% error |
| `medium` | 10-99 | Learning phase, partial ML data | ±8% error |
| `high` | 100+ | Mature model, robust ML adjustment | ±3% error |

**Usage in UI:**
```typescript
if (estimate.confidence === 'low') {
  showWarning("Price estimate may vary. Based on statistical averages.");
} else if (estimate.confidence === 'medium') {
  showInfo("Price estimate is improving with usage data.");
} else {
  showSuccess("High-confidence estimate based on historical data.");
}
```

### 10. Integration with Pricing

Token estimates feed directly into cost calculation:

```typescript
// In routes/estimate.ts
router.post('/v1/estimate', authenticate, async (req, res) => {
  const { content, source_lang, target_langs } = req.body;

  const estimate = await tokenEstimator.estimateBatch(
    content,
    source_lang,
    target_langs
  );

  // Get pricing from settings
  const pricePerToken = await settingsService.getDecimal('price_per_token');

  // Calculate cost
  const costUSD = estimate.total_tokens * pricePerToken;

  res.json({
    ...estimate,
    cost_usd: costUSD.toFixed(4),
    price_per_token: pricePerToken
  });
});
```

**Example pricing:**
```json
{
  "estimates": {
    "es": { "estimated_tokens": 120, "confidence": "high" },
    "fr": { "estimated_tokens": 130, "confidence": "high" }
  },
  "total_tokens": 250,
  "cost_usd": "0.0025",
  "price_per_token": 0.00001
}
```

### 11. Error Handling & Graceful Fallbacks

```typescript
// Graceful degradation if database is unavailable
private async getMLAdjustment(targetLang: string): Promise<number> {
  try {
    const stats = await prisma.$queryRaw<Array<{ ml_adjustment_ratio: number }>>`
      SELECT ml_adjustment_ratio
      FROM accuracy_stats
      WHERE language = ${targetLang}
      AND sample_count >= 10
      LIMIT 1
    `;

    if (stats && stats.length > 0) {
      return stats[0].ml_adjustment_ratio || 1.00;
    }
  } catch (error) {
    logger.warn('Failed to get ML adjustment', {
      targetLang,
      error: error instanceof Error ? error.message : 'Unknown error'
    });
  }

  // Fallback: Use statistical factors only (Layers 1-4)
  return 1.00;
}

// Unknown language fallback
private getLanguageFactor(targetLang: string): number {
  const factors: Record<string, number> = {
    // ... 131 languages
  };

  // Default to English baseline if language not found
  return factors[targetLang.toLowerCase()] || 1.00;
}
```

### 12. TypeScript Interfaces

```typescript
export interface TokenEstimate {
  estimated_tokens: number;
  confidence: 'low' | 'medium' | 'high';
  breakdown?: {
    base: number;
    languageFactor: number;
    htmlComplexity: number;
    safetyBuffer: number;
    mlAdjustment: number;
  };
}

export interface BatchEstimateResult {
  estimates: Record<string, TokenEstimate>;
  total_tokens: number;
  cache_hit: boolean;
}

export interface HTMLComplexityResult {
  level: 'none' | 'light' | 'medium' | 'heavy';
  overhead: number;
}
```

### 13. Testing Token Estimation

```typescript
// __tests__/services/tokenEstimator.test.ts
import { TokenEstimator } from '../../services/TokenEstimator';

describe('TokenEstimator', () => {
  let estimator: TokenEstimator;

  beforeEach(() => {
    estimator = new TokenEstimator();
  });

  describe('Layer 1: Base Tokenization', () => {
    it('should calculate base tokens correctly', () => {
      const content = "Hello, world!"; // 13 chars
      const base = estimator['estimateBaseTokens'](content);
      expect(base).toBe(4); // ceil(13 / 4)
    });
  });

  describe('Layer 2: Language Factors', () => {
    it('should apply Chinese efficiency factor', () => {
      const factor = estimator['getLanguageFactor']('zh');
      expect(factor).toBe(0.60);
    });

    it('should apply German verbosity factor', () => {
      const factor = estimator['getLanguageFactor']('de');
      expect(factor).toBe(1.20);
    });

    it('should default to 1.00 for unknown language', () => {
      const factor = estimator['getLanguageFactor']('unknown');
      expect(factor).toBe(1.00);
    });
  });

  describe('Layer 3: HTML Complexity', () => {
    it('should detect no HTML', () => {
      const result = estimator.analyzeHTMLComplexity("Plain text");
      expect(result).toEqual({ level: 'none', overhead: 0 });
    });

    it('should detect light HTML', () => {
      const result = estimator.analyzeHTMLComplexity("<p>Hello</p>");
      expect(result.level).toBe('light');
      expect(result.overhead).toBe(0.05);
    });

    it('should detect heavy HTML', () => {
      const html = '<div><table><tr><td>Cell</td></tr></table></div>';
      const result = estimator.analyzeHTMLComplexity(html);
      expect(result.level).toBe('heavy');
      expect(result.overhead).toBe(0.15);
    });
  });

  describe('Complete Estimation', () => {
    it('should produce accurate estimate with breakdown', async () => {
      const estimate = await estimator.estimateSingle(
        "Hello, world!",
        "en",
        "es"
      );

      expect(estimate.estimated_tokens).toBeGreaterThan(0);
      expect(estimate.confidence).toMatch(/low|medium|high/);
      expect(estimate.breakdown).toBeDefined();
      expect(estimate.breakdown?.base).toBeGreaterThan(0);
    });
  });

  describe('Batch Estimation', () => {
    it('should estimate multiple languages', async () => {
      const result = await estimator.estimateBatch(
        "Hello",
        "en",
        ["es", "fr", "de"]
      );

      expect(result.estimates).toHaveProperty('es');
      expect(result.estimates).toHaveProperty('fr');
      expect(result.estimates).toHaveProperty('de');
      expect(result.total_tokens).toBeGreaterThan(0);
    });
  });
});
```

## Integration Points

### 1. Called by `/v1/estimate` Endpoint

```typescript
// api/src/routes/estimate.ts
import { tokenEstimator } from '../services/TokenEstimator';

router.post('/v1/estimate', authenticate, async (req, res) => {
  const { content, source_lang, target_langs } = req.body;

  const estimate = await tokenEstimator.estimateBatch(
    content,
    source_lang,
    target_langs
  );

  res.json(estimate);
});
```

### 2. Used by TranslationService

```typescript
// api/src/services/translationService.ts
import { tokenEstimator } from './TokenEstimator';

class TranslationService {
  async createJob(data: CreateJobData) {
    // Get estimate before creating job
    const estimate = await tokenEstimator.estimateSingle(
      data.content,
      data.source_lang,
      data.target_lang
    );

    // Check if user has enough credits
    const user = await this.getUser(data.user_id);
    if (user.credits < estimate.estimated_tokens) {
      throw new Error('Insufficient credits');
    }

    // Create job with estimate
    const job = await prisma.translationJob.create({
      data: {
        ...data,
        estimated_tokens: estimate.estimated_tokens
      }
    });

    return job;
  }
}
```

### 3. Accuracy Feedback Loop

```typescript
// After translation completes
async function recordAccuracy(
  jobId: string,
  estimatedTokens: number,
  actualTokens: number,
  targetLang: string,
  htmlComplexity: 'none' | 'light' | 'medium' | 'heavy'
) {
  // Record individual translation
  await prisma.translationRecord.create({
    data: {
      target_lang: targetLang,
      estimated_tokens: estimatedTokens,
      actual_tokens: actualTokens,
      html_complexity: htmlComplexity
    }
  });

  // Update aggregated stats (this would be done in a background job)
  await updateAccuracyStats(targetLang);
}

async function updateAccuracyStats(language: string) {
  const records = await prisma.translationRecord.findMany({
    where: { target_lang: language },
    select: { estimated_tokens: true, actual_tokens: true }
  });

  const sampleCount = records.length;
  const adjustmentRatio = records.reduce((sum, r) => {
    return sum + (r.actual_tokens / r.estimated_tokens);
  }, 0) / sampleCount;

  const avgErrorPercent = records.reduce((sum, r) => {
    const error = Math.abs(r.actual_tokens - r.estimated_tokens) / r.estimated_tokens;
    return sum + (error * 100);
  }, 0) / sampleCount;

  await prisma.accuracyStats.upsert({
    where: { language },
    create: {
      language,
      sample_count: sampleCount,
      ml_adjustment_ratio: adjustmentRatio,
      avg_error_percent: avgErrorPercent
    },
    update: {
      sample_count: sampleCount,
      ml_adjustment_ratio: adjustmentRatio,
      avg_error_percent: avgErrorPercent
    }
  });
}
```

## Performance Optimizations

### 1. Caching Language Factors

Language factors are static and can be cached:

```typescript
// Pre-compute all 131 factors at startup
const LANGUAGE_FACTORS = Object.freeze({
  'zh': 0.60,
  'ja': 0.65,
  // ... all 131 languages
});

private getLanguageFactor(targetLang: string): number {
  return LANGUAGE_FACTORS[targetLang.toLowerCase()] || 1.00;
}
```

### 2. Database Query Optimization

```sql
-- Create index for fast ML adjustment lookup
CREATE INDEX idx_accuracy_stats_language_sample
ON accuracy_stats(language, sample_count);

-- Create index for historical records
CREATE INDEX idx_translation_records_lang_date
ON translation_records(target_lang, created_at DESC);
```

### 3. Batch Database Queries

```typescript
// Instead of N queries for N languages, fetch all at once
async estimateBatch(
  content: string,
  sourceLang: string,
  targetLangs: string[]
): Promise<BatchEstimateResult> {
  // Fetch all ML adjustments in one query
  const stats = await prisma.$queryRaw<Array<{
    language: string;
    ml_adjustment_ratio: number;
  }>>`
    SELECT language, ml_adjustment_ratio
    FROM accuracy_stats
    WHERE language = ANY(${targetLangs})
    AND sample_count >= 10
  `;

  const mlMap = new Map(
    stats.map(s => [s.language, s.ml_adjustment_ratio])
  );

  // Use map for O(1) lookups
  const estimates: Record<string, TokenEstimate> = {};
  for (const lang of targetLangs) {
    const mlAdjustment = mlMap.get(lang) || 1.00;
    // ... rest of estimation logic
  }
}
```

## Verification Checklist

- [ ] Base tokenization divides by 4 and rounds up
- [ ] All 131 language factors defined (0.60-1.28 range)
- [ ] HTML complexity correctly categorized (none/light/medium/heavy)
- [ ] Safety buffer applied (5-8%)
- [ ] ML adjustment queried from accuracy_stats table
- [ ] Confidence level based on sample count (low <10, medium 10-99, high 100+)
- [ ] Breakdown object returned with all 5 layers
- [ ] Batch estimation handles multiple languages efficiently
- [ ] Unknown languages default to 1.00 factor
- [ ] Database errors gracefully fallback to statistical estimation
- [ ] HTML regex correctly counts tags
- [ ] Final tokens rounded up (Math.ceil)
- [ ] Translation records stored for ML learning
- [ ] Accuracy stats updated periodically (background job)
- [ ] Integration with /v1/estimate endpoint works
- [ ] Integration with TranslationService works
- [ ] Unit tests cover all 5 layers
- [ ] Performance optimized with caching and batch queries

## Common Pitfalls

### 1. Floating Point Precision
```typescript
// ❌ WRONG: Precision loss
const finalTokens = Math.ceil(subtotal * mlAdjustment);

// ✅ CORRECT: Use Decimal for ML adjustment
import { Decimal } from '@prisma/client/runtime';
const mlAdjustment = new Decimal(stats[0].ml_adjustment_ratio);
const finalTokens = Math.ceil(subtotal * mlAdjustment.toNumber());
```

### 2. HTML Regex Injection
```typescript
// ❌ WRONG: Can be fooled by malformed HTML
const htmlTagCount = (content.match(/<[^>]+>/g) || []).length;

// ✅ CORRECT: Sanitize first or use DOMParser (server-side)
import { JSDOM } from 'jsdom';
const dom = new JSDOM(content);
const htmlTagCount = dom.window.document.getElementsByTagName('*').length;
```

### 3. Missing Database Indexes
```typescript
// Ensure indexes exist for fast queries
@@index([target_lang])
@@index([created_at])
@@index([language, sample_count])
```

### 4. Race Condition in Stats Update
```typescript
// ❌ WRONG: Multiple processes updating same row
await prisma.accuracyStats.update({
  where: { language },
  data: { sample_count: sampleCount + 1 }
});

// ✅ CORRECT: Use atomic increment
await prisma.accuracyStats.update({
  where: { language },
  data: { sample_count: { increment: 1 } }
});
```

## Best Practices

1. **Always round up final estimates** - Never undercharge users
2. **Cache language factors** - They're static and frequently accessed
3. **Batch database queries** - Avoid N+1 queries for multi-language estimates
4. **Log estimation errors** - Track when ML adjustment fails to fallback gracefully
5. **Monitor accuracy drift** - Set alerts if avg_error_percent exceeds 10%
6. **Version language factors** - Allow A/B testing of new multipliers
7. **Sanitize HTML input** - Prevent injection attacks via malformed markup
8. **Use transactions for accuracy updates** - Ensure consistency between records and stats
9. **Provide detailed breakdowns** - Helps debugging and user transparency
10. **Test edge cases** - Empty content, pure HTML, very long strings

## Related Skills

- **Skill 7**: Translation Service Integration (Google Gemini API) - Actual token usage
- **Skill 8**: PayPal Payment Integration - Credit purchases
- **Skill 14**: Credit Management System - Deducting estimated credits
- **Skill 22**: ML Accuracy Feedback System - Updating accuracy_stats table

---

# Skill 12: Security Hardening & Best Practices

## Identity
- **Skill ID**: `security-hardening`
- **Domain**: Application Security, Authentication, Authorization
- **Technologies**: Express middleware, Helmet, JWT, bcrypt, HMAC-SHA256, Redis, Zod
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Security vulnerability assessment and remediation
- Authentication and authorization mechanisms
- Input validation and sanitization
- CSRF, XSS, SSRF prevention
- Rate limiting and abuse prevention
- Secure token/key generation
- Webhook signature verification
- Security headers configuration
- Multi-tenancy isolation
- Audit logging for security events

**File patterns:**
- `api/src/middleware/auth.ts`
- `api/src/middleware/rateLimiter.ts`
- `api/src/middleware/validator.ts`
- `api/src/middleware/errorHandler.ts`
- `api/src/auth/**/*`
- `api/src/utils/encryption.ts`
- `nginx/*.conf`

## Security Architecture Overview

The Press.Zone Backend implements defense-in-depth security with multiple layers:

1. **Network Layer**: Nginx reverse proxy with TLS 1.2+, rate limiting
2. **Transport Layer**: HTTPS-only in production, HSTS headers
3. **Application Layer**: Helmet middleware, CORS, body size limits
4. **Authentication Layer**: Triple-tier auth (API Key, JWT, Admin JWT)
5. **Authorization Layer**: User status checks, subscription validation
6. **Input Layer**: Zod schema validation, request sanitization
7. **Data Layer**: Multi-tenancy isolation, prepared statements (Prisma ORM)
8. **Output Layer**: Structured error responses, no stack traces in production

## Core Security Patterns

### 1. Authentication Layers

The backend supports **three independent authentication mechanisms**, each designed for different client types:

#### Layer 1: API Key Authentication (WordPress Plugin)

**Use Case**: Long-lived authentication for WordPress plugin integration

**Format**: `Authorization: Bearer sk_live_64_char_hex` or `sk_test_*` for sandbox

**Implementation**:
```typescript
// api/src/middleware/auth.ts

/**
 * SHA-256 hashing prevents rainbow table attacks
 * Only hash stored in database, never plaintext
 */
function hashApiKey(key: string): string {
  return crypto.createHash('sha256').update(key).digest('hex');
}

export async function authenticateApiKey(
  req: Request,
  res: Response,
  next: NextFunction
): Promise<void> {
  try {
    const authHeader = req.headers.authorization;

    // 1. Validate header format
    if (!authHeader || !authHeader.startsWith('Bearer ')) {
      return res.status(401).json(
        errorResponse('MISSING_API_KEY', 'API key is required. Use Authorization: Bearer <api_key>')
      );
    }

    const apiKey = authHeader.substring(7); // Remove "Bearer " prefix

    // 2. Validate key prefix (sk_live_ or sk_test_)
    if (!apiKey.startsWith('sk_live_') && !apiKey.startsWith('sk_test_')) {
      return res.status(401).json(
        errorResponse('INVALID_API_KEY_FORMAT', 'API key must start with sk_live_ or sk_test_')
      );
    }

    // 3. Hash and lookup in database
    const keyHash = hashApiKey(apiKey);
    const apiKeyData = await prisma.apiKey.findUnique({
      where: { key_hash: keyHash },
      include: {
        user: {
          include: {
            subscription: true,
          },
        },
      },
    });

    if (!apiKeyData) {
      return res.status(401).json(errorResponse('INVALID_API_KEY', 'Invalid API key'));
    }

    // 4. Check key is active
    if (!apiKeyData.is_active) {
      return res.status(401).json(errorResponse('API_KEY_INACTIVE', 'This API key has been deactivated'));
    }

    // 5. Check user account status
    if (apiKeyData.user.status !== 'active') {
      return res.status(403).json(errorResponse('ACCOUNT_SUSPENDED', 'Your account has been suspended'));
    }

    // 6. Check subscription status
    if (!apiKeyData.user.subscription || apiKeyData.user.subscription.status !== 'active') {
      return res.status(403).json(errorResponse('SUBSCRIPTION_REQUIRED', 'Active subscription required to use the API'));
    }

    // 7. Update last_used_at (non-blocking for performance)
    prisma.apiKey
      .update({
        where: { id: apiKeyData.id },
        data: { last_used_at: new Date() },
      })
      .catch((error) => {
        logger.error('Failed to update API key last used timestamp', { error, apiKeyId: apiKeyData.id });
      });

    // 8. Attach context to request
    req.apiKey = {
      id: apiKeyData.id,
      userId: apiKeyData.user_id,
      keyHash: apiKeyData.key_hash,
      prefix: apiKeyData.prefix,
      name: apiKeyData.name,
      isActive: apiKeyData.is_active,
      lastUsedAt: apiKeyData.last_used_at,
      createdAt: apiKeyData.created_at,
    };

    req.user = {
      userId: apiKeyData.user.id,
      email: apiKeyData.user.email,
      plan: apiKeyData.user.subscription.plan_tier as any,
      subscriptionStatus: apiKeyData.user.subscription.status as any,
    };

    next();
  } catch (error) {
    logger.error('Error in authenticateApiKey middleware', { error });
    res.status(500).json(errorResponse('AUTHENTICATION_ERROR', 'Internal authentication error'));
  }
}
```

**Security Properties**:
- API keys never stored in plaintext (SHA-256 hashed)
- Timing-safe comparison for hash lookups (Prisma handles this)
- Non-blocking audit logging preserves request latency
- Multi-check validation (format, activation status, account status, subscription)
- No information leakage in error messages (generic "Invalid API key")

**Key Generation**:
```typescript
// api/src/utils/encryption.ts

export function generateApiKey(prefix: string = 'sk_live'): { key: string; hash: string; prefix: string } {
  const randomBytes = crypto.randomBytes(32); // 32 bytes = 256 bits
  const keySecret = randomBytes.toString('hex'); // 64 hex chars
  const fullKey = `${prefix}_${keySecret}`;
  
  // Hash for storage
  const hash = crypto.createHash('sha256').update(fullKey).digest('hex');
  
  // Prefix for display (first 16 chars: "sk_live_abc12345")
  const displayPrefix = fullKey.substring(0, Math.min(16, fullKey.length));

  return {
    key: fullKey, // ONLY returned once during creation
    hash, // Stored in database
    prefix: displayPrefix, // Shown in UI for identification
  };
}
```

**Usage Example**:
```bash
# WordPress plugin stores API key securely in wp_options
curl -X POST https://api.press.zone/v1/translate \
  -H "Authorization: Bearer sk_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{
    "source_lang": "en",
    "target_lang": "es",
    "content": "Hello world"
  }'
```

---

#### Layer 2: JWT Authentication (User Dashboard)

**Use Case**: Short-lived tokens for web dashboard, mobile apps, third-party integrations

**Format**: `Authorization: Bearer <jwt_access_token>`

**Token Types**:
- **Access Token**: 15-minute expiry, contains user context
- **Refresh Token**: 7-day expiry, used to obtain new access tokens

**Implementation**:
```typescript
// api/src/middleware/auth.ts

export async function authenticateJWT(
  req: Request,
  res: Response,
  next: NextFunction
): Promise<void> {
  try {
    const authHeader = req.headers.authorization;

    if (!authHeader || !authHeader.startsWith('Bearer ')) {
      return res.status(401).json({
        error: true,
        code: 'MISSING_TOKEN',
        message: 'JWT token is required. Use Authorization: Bearer <token>',
        timestamp: new Date().toISOString(),
      });
    }

    const token = authHeader.substring(7);

    // 1. Verify JWT signature and expiration
    let payload: JWTPayload;
    try {
      payload = jwt.verify(token, config.jwtAccessSecret) as JWTPayload;
    } catch (error) {
      if (error instanceof jwt.TokenExpiredError) {
        return res.status(401).json({
          error: true,
          code: 'TOKEN_EXPIRED',
          message: 'JWT token has expired',
          timestamp: new Date().toISOString(),
        });
      } else if (error instanceof jwt.JsonWebTokenError) {
        return res.status(401).json({
          error: true,
          code: 'INVALID_TOKEN',
          message: 'Invalid JWT token',
          timestamp: new Date().toISOString(),
        });
      }
      throw error;
    }

    // 2. Verify user still exists and is active
    const user = await prisma.user.findUnique({
      where: { id: payload.userId },
      include: { subscription: true },
    });

    if (!user) {
      return res.status(401).json({
        error: true,
        code: 'USER_NOT_FOUND',
        message: 'User not found',
        timestamp: new Date().toISOString(),
      });
    }

    if (user.status !== 'active') {
      return res.status(403).json({
        error: true,
        code: 'ACCOUNT_SUSPENDED',
        message: 'Your account has been suspended',
        timestamp: new Date().toISOString(),
      });
    }

    // 3. Attach user context
    req.user = {
      userId: user.id,
      email: user.email,
      plan: (user.subscription?.plan_tier as any) || 'starter',
      subscriptionStatus: (user.subscription?.status as any) || 'active',
    };

    next();
  } catch (error) {
    logger.error('Error in authenticateJWT middleware', { error });
    res.status(500).json({
      error: true,
      code: 'AUTHENTICATION_ERROR',
      message: 'Internal authentication error',
      timestamp: new Date().toISOString(),
    });
  }
}
```

**Token Generation**:
```typescript
// api/src/auth/jwtService.ts

export interface JWTPayload {
  userId: string;
  email: string;
  plan: string;
  subscriptionStatus: string;
  type: 'access';
}

export function generateAccessToken(
  userId: string,
  email: string,
  plan: string,
  subscriptionStatus: string
): string {
  const payload: JWTPayload = {
    userId,
    email,
    plan,
    subscriptionStatus,
    type: 'access',
  };

  const options: SignOptions = {
    expiresIn: '15m', // config.jwtAccessExpiry
    issuer: 'translate.press.zone',
    audience: 'api',
  };

  return jwt.sign(payload, config.jwtAccessSecret, options);
}

export function generateRefreshToken(userId: string): string {
  const payload: JWTRefreshPayload = {
    userId,
    type: 'refresh',
  };

  const options: SignOptions = {
    expiresIn: '7d', // config.jwtRefreshExpiry
    issuer: 'translate.press.zone',
    audience: 'api',
  };

  return jwt.sign(payload, config.jwtRefreshSecret, options);
}
```

**Security Properties**:
- Separate secrets for access and refresh tokens
- Short-lived access tokens minimize exposure window
- Token type validation prevents token substitution attacks
- User revalidation on every request (catches banned users)
- Issuer/audience claims prevent token misuse across systems

**Token Refresh Flow**:
```typescript
// api/src/routes/auth.ts

router.post('/refresh', async (req: Request, res: Response) => {
  const { refreshToken } = req.body;

  try {
    // Verify refresh token
    const payload = verifyRefreshToken(refreshToken);
    
    // Get user
    const user = await prisma.user.findUnique({
      where: { id: payload.userId },
      include: { subscription: true },
    });

    if (!user || user.status !== 'active') {
      return res.status(401).json({ error: 'Invalid refresh token' });
    }

    // Issue new access token
    const newAccessToken = generateAccessToken(
      user.id,
      user.email,
      user.subscription?.plan_tier || 'starter',
      user.subscription?.status || 'active'
    );

    res.json({
      success: true,
      data: {
        accessToken: newAccessToken,
        expiresIn: 900, // 15 minutes in seconds
      },
    });
  } catch (error) {
    res.status(401).json({ error: 'Invalid refresh token' });
  }
});
```

---

#### Layer 3: Admin JWT Authentication (Admin Panel)

**Use Case**: Admin dashboard access with elevated privileges

**Format**: `Authorization: Bearer <admin_jwt_token>`

**Special Features**:
- **Development bypass**: `Bearer dev-admin-token` for local testing
- **Separate JWT secret**: Different from user JWT to prevent privilege escalation
- **Role-based access**: `admin` or `support` roles
- **Activity tracking**: `last_login_at` timestamp updated on each request

**Implementation**:
```typescript
// api/src/middleware/auth.ts

export async function authenticateAdmin(
  req: Request,
  res: Response,
  next: NextFunction
): Promise<void> {
  try {
    // Development mode bypass
    if (config.nodeEnv === 'development') {
      const authHeader = req.headers.authorization;
      if (authHeader === 'Bearer dev-admin-token') {
        req.admin = {
          id: 'dev-admin',
          email: 'admin@dev.local',
          role: 'admin' as AdminRole,
        };
        next();
        return;
      }
    }

    const authHeader = req.headers.authorization;

    if (!authHeader || !authHeader.startsWith('Bearer ')) {
      return res.status(401).json({
        error: true,
        code: 'MISSING_TOKEN',
        message: 'Admin JWT token is required',
        timestamp: new Date().toISOString(),
      });
    }

    const token = authHeader.substring(7);

    // Verify JWT using admin-specific secret
    let payload: { adminId: string; email: string; role: AdminRole };
    try {
      payload = jwt.verify(token, config.jwtAdminSecret) as { adminId: string; email: string; role: AdminRole };
    } catch (error) {
      if (error instanceof jwt.TokenExpiredError) {
        return res.status(401).json({
          error: true,
          code: 'TOKEN_EXPIRED',
          message: 'Admin JWT token has expired',
          timestamp: new Date().toISOString(),
        });
      } else if (error instanceof jwt.JsonWebTokenError) {
        return res.status(401).json({
          error: true,
          code: 'INVALID_TOKEN',
          message: 'Invalid admin JWT token',
          timestamp: new Date().toISOString(),
        });
      }
      throw error;
    }

    // Verify admin exists and is active
    const admin = await prisma.adminUser.findUnique({
      where: { id: payload.adminId },
    });

    if (!admin) {
      return res.status(401).json({
        error: true,
        code: 'ADMIN_NOT_FOUND',
        message: 'Admin user not found',
        timestamp: new Date().toISOString(),
      });
    }

    if (!(admin as any).is_active) {
      return res.status(403).json({
        error: true,
        code: 'ADMIN_INACTIVE',
        message: 'Admin account has been deactivated',
        timestamp: new Date().toISOString(),
      });
    }

    // Update last login timestamp (non-blocking)
    prisma.adminUser
      .update({
        where: { id: admin.id },
        data: { last_login_at: new Date() },
      })
      .catch((error) => {
        logger.error('Failed to update admin last login timestamp', { error, adminId: admin.id });
      });

    // Attach admin context
    req.admin = {
      id: admin.id,
      email: admin.email,
      role: admin.role as AdminRole,
    };

    next();
  } catch (error) {
    logger.error('Error in authenticateAdmin middleware', { error });
    res.status(500).json({
      error: true,
      code: 'AUTHENTICATION_ERROR',
      message: 'Internal authentication error',
      timestamp: new Date().toISOString(),
    });
  }
}
```

**Security Properties**:
- Separate database table (`AdminUser`) prevents user privilege escalation
- Separate JWT secret isolates admin tokens from user tokens
- Development bypass only active when `NODE_ENV=development`
- Non-blocking audit logging for activity tracking
- Role-based access control ready for future RBAC expansion

---

### 2. Rate Limiting (Redis-Backed, Tier-Based)

**Architecture**: Distributed rate limiting using Redis with tier-based quotas

**Implementation**:
```typescript
// api/src/middleware/rateLimiter.ts

import rateLimit, { RateLimitRequestHandler } from 'express-rate-limit';
import RedisStore from 'rate-limit-redis';
import Redis from 'ioredis';

// Create Redis client
const redis = new Redis({
  host: config.redisHost,
  port: config.redisPort,
  password: config.redisPassword,
  db: config.redisDb,
  retryStrategy: (times) => {
    const delay = Math.min(times * 50, 2000);
    return delay;
  },
});

/**
 * Create a rate limiter with specified window and max requests
 */
export function createRateLimiter(windowMs: number, max: number): RateLimitRequestHandler {
  return rateLimit({
    windowMs,
    max,
    standardHeaders: true, // Return rate limit info in `RateLimit-*` headers
    legacyHeaders: false, // Disable `X-RateLimit-*` headers
    store: new RedisStore({
      sendCommand: (...args: string[]) => redis.call(...args),
      prefix: 'rl:',
    }),
    keyGenerator: (req: Request) => {
      // Tier 1: User ID (authenticated requests)
      if (req.user) {
        return `user:${req.user.userId}`;
      } 
      // Tier 2: API Key ID (WordPress plugin)
      else if (req.apiKey) {
        return `apikey:${req.apiKey.id}`;
      } 
      // Tier 3: IP address (unauthenticated requests)
      else {
        return req.ip || req.socket.remoteAddress || 'unknown';
      }
    },
    handler: (req: Request, res: Response) => {
      logger.warn('Rate limit exceeded', {
        userId: req.user?.userId,
        apiKeyId: req.apiKey?.id,
        ip: req.ip,
        path: req.path,
      });

      res.status(429).json(errorResponse('RATE_LIMIT_EXCEEDED', 'Too many requests. Please slow down and try again later.'));
    },
    skip: (req: Request) => {
      // Enterprise tier gets unlimited requests
      if (req.user?.plan === SubscriptionPlan.ENTERPRISE) {
        return true;
      }
      return false;
    },
  });
}
```

**Tier-Based Quotas**:

| Plan | Rate Limit | Window | Notes |
|------|-----------|--------|-------|
| **Starter** | 60 requests | 1 minute | Free tier |
| **Professional** | 120 requests | 1 minute | Paid tier |
| **Enterprise** | Unlimited | N/A | Skip rate limiting entirely |

**Dynamic Rate Limiter** (adjusts based on user plan):
```typescript
export function apiRateLimiter(req: Request, res: Response, next: NextFunction): void {
  // Determine user's subscription tier
  let plan: SubscriptionPlan = SubscriptionPlan.STARTER;
  if (req.user) {
    plan = req.user.plan as SubscriptionPlan;
  }

  // Get rate limit for tier
  const limit = getRateLimit(plan);

  // Enterprise tier has unlimited requests
  if (limit === 0 || plan === SubscriptionPlan.ENTERPRISE) {
    next();
    return;
  }

  // Create rate limiter for this tier (1 minute window)
  const limiter = createRateLimiter(60 * 1000, limit);

  // Apply rate limiter
  limiter(req, res, next);
}
```

**Pre-Configured Limiters**:
```typescript
// Public endpoints (registration, login)
export const publicRateLimiter = createRateLimiter(
  15 * 60 * 1000, // 15 minutes
  100 // 100 requests per 15 minutes
);

// Sensitive endpoints (password reset, email verification)
export const strictRateLimiter = createRateLimiter(
  60 * 60 * 1000, // 1 hour
  5 // 5 requests per hour
);

// Admin endpoints
export const adminRateLimiter = createRateLimiter(
  60 * 1000, // 1 minute
  300 // 300 requests per minute
);
```

**Usage in Routes**:
```typescript
// api/src/routes/auth.ts

import { publicRateLimiter, strictRateLimiter } from '../middleware/rateLimiter';

router.post('/register', publicRateLimiter, validate(registerSchema), async (req, res) => {
  // Registration logic
});

router.post('/forgot-password', strictRateLimiter, validate(forgotPasswordSchema), async (req, res) => {
  // Password reset logic
});
```

**Security Properties**:
- Distributed across instances via Redis (horizontal scaling)
- Per-user/per-key isolation prevents noisy neighbor attacks
- Tier-based quotas incentivize paid subscriptions
- Standard headers (`RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`)
- Graceful degradation if Redis unavailable (falls back to in-memory)
- Exponential backoff in Redis connection retries

---

### 3. Input Validation (Zod Schemas)

**Architecture**: Type-safe request validation with automatic TypeScript inference

**Implementation**:
```typescript
// api/src/middleware/validator.ts

import { ZodSchema, ZodError } from 'zod';
import { ValidationError, ValidationErrorResponse } from '../types';

/**
 * Format Zod error into ValidationError array
 */
function formatZodError(error: ZodError): ValidationError[] {
  return error.errors.map((err) => ({
    field: err.path.join('.'),
    message: err.message,
    rule: err.code,
    expected: 'expected' in err ? String((err as any).expected) : undefined,
    received: 'received' in err ? String((err as any).received) : undefined,
  }));
}

/**
 * Validate request body against Zod schema
 */
export function validate(schema: ZodSchema) {
  return (req: Request, res: Response, next: NextFunction): void => {
    try {
      // Validate and parse request body
      const validatedData = schema.parse(req.body);

      // Replace request body with validated data (type safety!)
      req.body = validatedData;

      next();
    } catch (error) {
      if (error instanceof ZodError) {
        const validationErrors = formatZodError(error);

        logger.warn('Validation error', {
          path: req.path,
          method: req.method,
          errors: validationErrors,
          requestId: req.requestId,
        });

        const errorResponse: ValidationErrorResponse = {
          error: true,
          code: 'VALIDATION_ERROR',
          message: 'Request validation failed',
          errors: validationErrors,
          timestamp: new Date().toISOString(),
        };

        if (req.requestId) {
          errorResponse.requestId = req.requestId;
        }

        res.status(400).json(errorResponse);
      } else {
        next(error);
      }
    }
  };
}

/**
 * Validate query parameters
 */
export function validateQuery(schema: ZodSchema) {
  return (req: Request, res: Response, next: NextFunction): void => {
    try {
      const validatedData = schema.parse(req.query);
      req.query = validatedData;
      next();
    } catch (error) {
      if (error instanceof ZodError) {
        const validationErrors = formatZodError(error);
        res.status(400).json({
          error: true,
          code: 'VALIDATION_ERROR',
          message: 'Query parameter validation failed',
          errors: validationErrors,
          timestamp: new Date().toISOString(),
        });
      } else {
        next(error);
      }
    }
  };
}

/**
 * Validate URL parameters
 */
export function validateParams(schema: ZodSchema) {
  return (req: Request, res: Response, next: NextFunction): void => {
    try {
      const validatedData = schema.parse(req.params);
      req.params = validatedData;
      next();
    } catch (error) {
      if (error instanceof ZodError) {
        const validationErrors = formatZodError(error);
        res.status(400).json({
          error: true,
          code: 'VALIDATION_ERROR',
          message: 'URL parameter validation failed',
          errors: validationErrors,
          timestamp: new Date().toISOString(),
        });
      } else {
        next(error);
      }
    }
  };
}
```

**Example Schema**:
```typescript
// api/src/routes/translate.ts

import { z } from 'zod';

const translateSchema = z.object({
  source_lang: z.string().min(2).max(10),
  target_lang: z.string().min(2).max(10),
  content: z.string().min(1).max(50000),
  model: z.enum(['gemini-2.0-flash-exp', 'gemini-1.5-flash', 'gemini-1.5-pro']).optional(),
  tone: z.enum(['formal', 'casual', 'technical', 'creative']).optional(),
  callback_url: z.string().url().optional(),
  callback_secret: z.string().min(32).optional(),
  client_job_id: z.string().max(255).optional(),
});

router.post('/translate', authenticateApiKey, validate(translateSchema), async (req, res) => {
  // req.body is now typed as z.infer<typeof translateSchema>
  const { source_lang, target_lang, content, model, tone } = req.body;
  
  // Translation logic
});
```

**Security Properties**:
- Type coercion prevents injection attacks (e.g., `"5"` → `5`)
- URL validation prevents SSRF in callback URLs
- String length limits prevent DoS attacks
- Enum validation prevents invalid values
- Automatic sanitization of special characters
- Clear error messages aid debugging without leaking system info

**Custom Validators**:
```typescript
/**
 * Custom validation middleware factory
 */
export function customValidate(
  validationFn: (req: Request) => ValidationError[] | null
) {
  return (req: Request, res: Response, next: NextFunction): void => {
    try {
      const errors = validationFn(req);

      if (errors && errors.length > 0) {
        logger.warn('Custom validation error', {
          path: req.path,
          method: req.method,
          errors,
          requestId: req.requestId,
        });

        res.status(400).json({
          error: true,
          code: 'VALIDATION_ERROR',
          message: 'Request validation failed',
          errors,
          timestamp: new Date().toISOString(),
        });
      } else {
        next();
      }
    } catch (error) {
      next(error);
    }
  };
}

// Example: Validate callback URL not on private network
const validateCallbackUrl = customValidate((req) => {
  if (!req.body.callback_url) return null;

  const url = new URL(req.body.callback_url);
  
  // Block private IPs
  if (isPrivateIp(url.hostname)) {
    return [{
      field: 'callback_url',
      message: 'Callback URL must not point to private network',
      rule: 'ssrf_prevention',
    }];
  }

  return null;
});
```

---

### 4. Error Handling Hierarchy

**Architecture**: 10 custom error classes with structured responses

**Base Error Class**:
```typescript
// api/src/middleware/errorHandler.ts

export class ApiError extends Error {
  public statusCode: number;
  public code: string;
  public details?: Record<string, unknown>;
  public isOperational: boolean;

  constructor(
    statusCode: number,
    code: string,
    message: string,
    details?: Record<string, unknown>,
    isOperational = true
  ) {
    super(message);
    this.statusCode = statusCode;
    this.code = code;
    this.details = details;
    this.isOperational = isOperational;

    // Maintains proper stack trace
    Error.captureStackTrace(this, this.constructor);

    // Set prototype explicitly (TypeScript requirement)
    Object.setPrototypeOf(this, ApiError.prototype);
  }
}
```

**Error Class Hierarchy**:

| Error Class | HTTP Status | Code | Use Case |
|-------------|-------------|------|----------|
| `ValidationError` | 400 | `VALIDATION_ERROR` | Zod schema failures, malformed input |
| `AuthenticationError` | 401 | `AUTHENTICATION_FAILED` | Invalid API key, expired JWT |
| `AuthorizationError` | 403 | `ACCESS_DENIED` | Insufficient permissions |
| `InsufficientCreditsError` | 402 | `INSUFFICIENT_CREDITS` | Not enough credits for translation |
| `NotFoundError` | 404 | `NOT_FOUND` | Resource doesn't exist |
| `ConflictError` | 409 | `CONFLICT` | Duplicate resource, race condition |
| `RateLimitError` | 429 | `RATE_LIMIT_EXCEEDED` | Too many requests |
| `InternalServerError` | 500 | `INTERNAL_ERROR` | Unhandled exceptions |
| `ServiceUnavailableError` | 503 | `SERVICE_UNAVAILABLE` | Redis down, Gemini API timeout |

**Example Error Classes**:
```typescript
export class InsufficientCreditsError extends ApiError {
  constructor(
    required: number,
    available: number,
    message = 'Insufficient credits to complete this request'
  ) {
    super(402, 'INSUFFICIENT_CREDITS', message, {
      required,
      available,
      deficit: required - available,
    });
    Object.setPrototypeOf(this, InsufficientCreditsError.prototype);
  }
}

export class NotFoundError extends ApiError {
  constructor(resource = 'Resource', message?: string) {
    super(404, 'NOT_FOUND', message || `${resource} not found`);
    Object.setPrototypeOf(this, NotFoundError.prototype);
  }
}
```

**Global Error Handler**:
```typescript
export function errorHandler(
  error: Error | ApiError,
  req: Request,
  res: Response,
  _next: NextFunction
): void {
  // Log error
  if (error instanceof ApiError) {
    if (error.statusCode >= 500) {
      logError(error, {
        requestId: req.requestId,
        userId: req.user?.userId,
        apiKeyId: req.apiKey?.id,
        path: req.path,
        method: req.method,
      });
    } else {
      logger.warn('API error', {
        code: error.code,
        message: error.message,
        statusCode: error.statusCode,
        requestId: req.requestId,
      });
    }
  } else {
    // Unhandled error
    logError(error, {
      requestId: req.requestId,
      userId: req.user?.userId,
      path: req.path,
      method: req.method,
    });
  }

  // Track error in metrics
  if (error instanceof ApiError) {
    trackError(error.constructor.name, error.code);
  } else {
    trackError('UnhandledError', 'UNHANDLED_ERROR');
  }

  // Send error response
  if (error instanceof ApiError) {
    const errorResponse = formatErrorResponse(error, req.requestId);
    res.status(error.statusCode).json(errorResponse);
  } else {
    // Generic 500 error
    const errorResponse: ErrorResponse = {
      error: true,
      code: 'INTERNAL_ERROR',
      message: isProduction() ? 'An unexpected error occurred' : error.message,
      timestamp: new Date().toISOString(),
    };

    if (req.requestId) {
      errorResponse.requestId = req.requestId;
    }

    // Include stack trace in development
    if (!isProduction() && error.stack) {
      errorResponse.details = {
        stack: error.stack.split('\n'),
      };
    }

    res.status(500).json(errorResponse);
  }
}
```

**Usage in Routes**:
```typescript
import { asyncHandler } from '../middleware/errorHandler';
import { NotFoundError, InsufficientCreditsError } from '../middleware/errorHandler';

router.get('/jobs/:id', authenticateApiKey, asyncHandler(async (req, res) => {
  const job = await prisma.translationJob.findUnique({
    where: { id: req.params.id },
  });

  if (!job) {
    throw new NotFoundError('Translation Job');
  }

  // Authorization check
  if (job.user_id !== req.user!.userId) {
    throw new AuthorizationError('You do not have access to this job');
  }

  res.json({ success: true, data: job });
}));
```

**Security Properties**:
- No stack traces in production (prevents information disclosure)
- Structured error codes for client error handling
- Request ID tracking for debugging
- Operational vs. non-operational errors differentiated
- Error metrics for monitoring

---

### 5. CSRF Protection

**Architecture**: Token-based CSRF for state-changing operations

**Implementation Strategy**:

1. **API Key Auth**: CSRF not applicable (Bearer tokens don't use cookies)
2. **JWT Auth**: CSRF not applicable (tokens sent in Authorization header)
3. **Admin Panel**: Implements CSRF tokens for form submissions

**Admin Panel CSRF**:
```typescript
// api/src/routes/admin/auth.ts

import csrf from 'csurf';

const csrfProtection = csrf({ cookie: true });

router.post('/login', csrfProtection, async (req, res) => {
  // CSRF token automatically validated
  const { email, password } = req.body;
  
  // Login logic
});

router.get('/csrf-token', (req, res) => {
  res.json({ csrfToken: req.csrfToken() });
});
```

**React Admin Panel**:
```typescript
// admin-panel/src/services/api.ts

// Fetch CSRF token on app load
const getCsrfToken = async () => {
  const response = await fetch('/v1/admin/csrf-token');
  const { csrfToken } = await response.json();
  return csrfToken;
};

// Include in form submissions
const loginUser = async (email: string, password: string) => {
  const csrfToken = await getCsrfToken();
  
  await fetch('/v1/admin/auth/login', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-CSRF-Token': csrfToken,
    },
    body: JSON.stringify({ email, password }),
    credentials: 'include',
  });
};
```

**Security Properties**:
- Double-submit cookie pattern
- Synchronizer token pattern for forms
- SameSite=Strict cookie attribute
- Token rotation after authentication

---

### 6. XSS Prevention

**Architecture**: Output encoding, CSP headers, sanitized error messages

**Helmet Configuration**:
```typescript
// api/src/server.ts

import helmet from 'helmet';

app.use(helmet({
  contentSecurityPolicy: {
    directives: {
      defaultSrc: ["'self'"],
      scriptSrc: ["'self'", "'unsafe-inline'"], // Only for admin panel
      styleSrc: ["'self'", "'unsafe-inline'"],
      imgSrc: ["'self'", 'data:', 'https:'],
      connectSrc: ["'self'", 'https://api.press.zone'],
      fontSrc: ["'self'"],
      objectSrc: ["'none'"],
      mediaSrc: ["'self'"],
      frameSrc: ["'none'"],
    },
  },
  xssFilter: true,
  noSniff: true,
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' },
}));
```

**Output Encoding**:
```typescript
// All error messages are JSON-encoded by Express
// No user input reflected in HTML responses
// API is JSON-only, no HTML templates

// Example: Safe error response
res.status(400).json({
  error: true,
  code: 'INVALID_INPUT',
  message: 'Invalid language code', // Static message
  timestamp: new Date().toISOString(),
});

// Unsafe: NEVER do this
res.status(400).send(`<html><body>Error: ${req.body.input}</body></html>`);
```

**Input Sanitization**:
```typescript
// Zod automatically sanitizes strings
const schema = z.object({
  content: z.string().max(50000), // Prevents excessively long strings
});

// For HTML content (e.g., webhook payloads), use DOMPurify or similar
import DOMPurify from 'isomorphic-dompurify';

const sanitizedHtml = DOMPurify.sanitize(userInput);
```

**Security Properties**:
- No user input reflected in responses without sanitization
- CSP headers prevent inline scripts
- X-XSS-Protection header enables browser XSS filter
- JSON-only API eliminates HTML injection vectors

---

### 7. SSRF Prevention (Webhook URLs)

**Architecture**: URL validation, private IP blocking, HTTPS enforcement

**Implementation**:
```typescript
// api/src/utils/validation.ts

const BLOCKED_HOSTS = [
  'localhost',
  '127.0.0.1',
  '0.0.0.0',
  '::1',
  '169.254.169.254', // AWS metadata
  'metadata.google.internal', // GCP metadata
];

const PRIVATE_IP_PATTERNS = [
  /^10\./, // 10.0.0.0/8
  /^172\.(1[6-9]|2\d|3[01])\./, // 172.16.0.0/12
  /^192\.168\./, // 192.168.0.0/16
  /^fc00:/, // IPv6 private
  /^fe80:/, // IPv6 link-local
];

export function isPrivateIp(hostname: string): boolean {
  // Check exact matches
  if (BLOCKED_HOSTS.includes(hostname.toLowerCase())) {
    return true;
  }

  // Check private IP ranges
  return PRIVATE_IP_PATTERNS.some(pattern => pattern.test(hostname));
}

export function validateCallbackUrl(url: string, requireHttps: boolean = true): void {
  let parsedUrl: URL;
  
  try {
    parsedUrl = new URL(url);
  } catch (error) {
    throw new ValidationError([{
      field: 'callback_url',
      message: 'Invalid URL format',
      rule: 'url_format',
    }]);
  }

  // Enforce HTTPS in production
  if (requireHttps && parsedUrl.protocol !== 'https:') {
    throw new ValidationError([{
      field: 'callback_url',
      message: 'Callback URL must use HTTPS',
      rule: 'https_required',
    }]);
  }

  // Block private IPs
  if (isPrivateIp(parsedUrl.hostname)) {
    throw new ValidationError([{
      field: 'callback_url',
      message: 'Callback URL must not point to private network',
      rule: 'ssrf_prevention',
    }]);
  }
}
```

**Usage in Routes**:
```typescript
// api/src/routes/jobs.ts

const translateSchema = z.object({
  callback_url: z.string().url().optional(),
  callback_secret: z.string().min(32).optional(),
  // ... other fields
});

router.post('/translate', 
  authenticateApiKey, 
  validate(translateSchema),
  customValidate((req) => {
    if (req.body.callback_url) {
      try {
        validateCallbackUrl(req.body.callback_url, config.nodeEnv === 'production');
      } catch (error) {
        if (error instanceof ValidationError) {
          return error.errors;
        }
      }
    }
    return null;
  }),
  async (req, res) => {
    // Translation logic
  }
);
```

**Security Properties**:
- Private IP ranges blocked (RFC 1918)
- Cloud metadata endpoints blocked (AWS, GCP)
- Localhost blocked
- HTTPS enforced in production
- DNS rebinding protection (validate hostname, not resolved IP)

---

### 8. Multi-Tenancy Isolation

**Architecture**: User ID checks in all queries, row-level security

**Pattern**:
```typescript
// ✅ CORRECT: Always filter by user_id
const job = await prisma.translationJob.findFirst({
  where: {
    id: jobId,
    user_id: req.user!.userId, // Critical: ensures user owns this job
  },
});

if (!job) {
  throw new NotFoundError('Translation Job');
}

// ❌ INCORRECT: Missing user_id check (IDOR vulnerability)
const job = await prisma.translationJob.findUnique({
  where: { id: jobId },
});
```

**Prisma Middleware** (automatic user_id injection):
```typescript
// api/src/middleware/prisma.ts

import { PrismaClient } from '@prisma/client';

export function createPrismaClientWithMiddleware(userId: string): PrismaClient {
  const prisma = new PrismaClient();

  prisma.$use(async (params, next) => {
    // Auto-inject user_id for User-scoped models
    if (['TranslationJob', 'ApiKey', 'WordPressSite'].includes(params.model || '')) {
      if (params.action === 'findMany' || params.action === 'findFirst') {
        params.args.where = {
          ...params.args.where,
          user_id: userId,
        };
      }
    }

    return next(params);
  });

  return prisma;
}
```

**Security Properties**:
- All queries scoped to authenticated user
- No cross-tenant data leakage
- Prisma middleware provides defense-in-depth
- Admin queries use separate code paths (no middleware)

---

### 9. Security Headers (Nginx + Helmet)

**Nginx Configuration**:
```nginx
# nginx/api.press.zone.conf

# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

# CORS headers (API-specific)
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, PATCH, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,X-API-Key' always;
add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range' always;

# SSL configuration (Mozilla Modern)
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
```

**Helmet Middleware**:
```typescript
// api/src/server.ts

import helmet from 'helmet';

app.use(helmet());
```

**Header Breakdown**:

| Header | Value | Purpose |
|--------|-------|---------|
| `Strict-Transport-Security` | `max-age=63072000; includeSubDomains; preload` | Force HTTPS for 2 years |
| `X-Frame-Options` | `DENY` | Prevent clickjacking |
| `X-Content-Type-Options` | `nosniff` | Prevent MIME sniffing |
| `X-XSS-Protection` | `1; mode=block` | Enable browser XSS filter |
| `Referrer-Policy` | `strict-origin-when-cross-origin` | Limit referrer leakage |
| `Content-Security-Policy` | (See CSP section) | Prevent XSS, injection |

---

### 10. Webhook Signature Verification

**Architecture**: HMAC-SHA256 signatures, timestamp validation, replay attack prevention

**Outgoing Webhooks** (Backend → WordPress):
```typescript
// api/src/services/webhookService.ts

import crypto from 'crypto';

export async function deliverWebhook(
  jobId: string,
  payload: WebhookPayload,
  callbackUrl: string,
  callbackSecret: string
): Promise<WebhookDelivery> {
  // Prepare payload body
  const body = JSON.stringify(payload);

  // Add timestamp for replay attack prevention
  const timestamp = Date.now().toString();

  // Generate HMAC signature
  const signature = crypto
    .createHmac('sha256', callbackSecret)
    .update(body)
    .digest('hex');

  // Prepare headers
  const headers = {
    'Content-Type': 'application/json',
    'X-TPZ-Signature': signature, // WordPress verifies this
    'X-TPZ-Timestamp': timestamp,
    'User-Agent': 'TranslatePressZone-Webhook/1.0',
  };

  // Send webhook
  const response = await axios.post(callbackUrl, payload, {
    headers,
    timeout: 10000, // 10 second timeout
  });

  // Log delivery attempt
  await prisma.webhookDelivery.create({
    data: {
      job_id: jobId,
      attempt_number: 1,
      success: true,
      http_status: response.status,
    },
  });
}
```

**WordPress Verification** (WordPress → Backend):
```php
// WordPress plugin verifies signature

$payload = file_get_contents('php://input');
$signature = $_SERVER['HTTP_X_TPZ_SIGNATURE'];
$timestamp = $_SERVER['HTTP_X_TPZ_TIMESTAMP'];

// Check timestamp (5 minute window)
if (abs(time() - (int)$timestamp / 1000) > 300) {
    wp_send_json_error('Webhook timestamp expired', 401);
}

// Verify signature
$expected_signature = hash_hmac('sha256', $payload, $callback_secret);

if (!hash_equals($expected_signature, $signature)) {
    wp_send_json_error('Invalid webhook signature', 401);
}

// Process webhook
$data = json_decode($payload, true);
```

**Incoming Webhooks** (PayPal → Backend):
```typescript
// api/src/routes/webhooks.ts

import { verifyPayPalWebhook } from '../services/paypalService';

router.post('/paypal', async (req, res) => {
  // Verify PayPal signature using certificate validation
  const isValid = await verifyPayPalWebhook(req.headers, req.body);

  if (!isValid) {
    logger.warn('Invalid PayPal webhook signature', {
      headers: req.headers,
      ip: req.ip,
    });
    return res.status(401).json({ error: 'Invalid signature' });
  }

  // Process webhook
  await handlePayPalEvent(req.body);
  res.json({ success: true });
});
```

**Security Properties**:
- HMAC-SHA256 prevents tampering
- Timestamp validation prevents replay attacks (5-minute window)
- Timing-safe comparison (`hash_equals`, `crypto.timingSafeEqual`)
- PayPal webhooks use certificate validation
- All delivery attempts logged for audit trail

---

## Security Checklist

### Deployment Checklist

Before deploying to production, verify:

- [ ] All secrets in environment variables (not hardcoded)
- [ ] `NODE_ENV=production` set
- [ ] HTTPS enforced (Nginx redirects HTTP → HTTPS)
- [ ] HSTS header enabled (`max-age=63072000`)
- [ ] Admin dev bypass disabled (`NODE_ENV !== 'development'`)
- [ ] Database credentials rotated
- [ ] Redis password set
- [ ] Gemini API key secured
- [ ] PayPal credentials (client ID, secret) secured
- [ ] SendGrid API key secured
- [ ] JWT secrets unique and strong (256-bit)
- [ ] API key prefix configured (`sk_live_`)
- [ ] CORS origins whitelisted (not `*` in production)
- [ ] Rate limiting enabled
- [ ] Error stack traces disabled
- [ ] Request logging excludes sensitive data
- [ ] File upload size limits configured
- [ ] Database backups automated (daily)
- [ ] Monitoring alerts configured
- [ ] Security headers validated (SecurityHeaders.com)

### Code Review Checklist

When reviewing security-related code:

- [ ] Authentication middleware applied to protected routes
- [ ] Authorization checks verify user owns resource
- [ ] Input validation with Zod schemas
- [ ] SQL injection impossible (Prisma ORM used)
- [ ] No user input reflected without sanitization
- [ ] Error messages don't leak system info
- [ ] Webhook URLs validated (SSRF prevention)
- [ ] Rate limiting appropriate for endpoint sensitivity
- [ ] Passwords hashed with bcrypt (12 rounds)
- [ ] API keys hashed with SHA-256
- [ ] JWT tokens include expiration
- [ ] Refresh tokens stored securely (httpOnly cookies)
- [ ] CSRF protection on state-changing operations
- [ ] Security headers configured (Helmet)
- [ ] Audit logging for sensitive operations
- [ ] No secrets in error messages or logs
- [ ] Timing-safe comparison for secrets
- [ ] Multi-tenancy isolation (user_id checks)

### Incident Response

If a security incident occurs:

1. **Immediate Actions**:
   - Rotate compromised secrets (API keys, JWT secrets, database passwords)
   - Revoke affected API keys via admin panel
   - Block malicious IPs in Nginx
   - Enable maintenance mode if needed

2. **Investigation**:
   - Review logs for attack patterns (`/var/log/nginx/api.press.zone.access.log`)
   - Check Redis for rate limit bypass attempts
   - Audit database for unauthorized access
   - Check Prometheus metrics for anomalies

3. **Remediation**:
   - Patch vulnerabilities
   - Deploy hotfix
   - Notify affected users
   - Update security documentation

4. **Post-Mortem**:
   - Document incident timeline
   - Identify root cause
   - Implement preventive measures
   - Update security checklist

---

## Testing Security

### Unit Tests

```typescript
// api/src/__tests__/unit/auth/jwtService.test.ts

describe('JWT Service', () => {
  it('should reject expired tokens', () => {
    const token = generateAccessToken('user123', 'test@example.com', 'starter', 'active');
    
    // Fast-forward time
    jest.advanceTimersByTime(16 * 60 * 1000); // 16 minutes
    
    expect(() => verifyAccessToken(token)).toThrow('Access token expired');
  });

  it('should reject tokens with invalid signature', () => {
    const token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.invalid.signature';
    
    expect(() => verifyAccessToken(token)).toThrow('Invalid access token');
  });
});
```

### Integration Tests

```typescript
// api/src/__tests__/integration/security/ssrf.test.ts

describe('SSRF Prevention', () => {
  it('should block private IP addresses in callback URLs', async () => {
    const response = await request(app)
      .post('/v1/translate')
      .set('Authorization', `Bearer ${validApiKey}`)
      .send({
        source_lang: 'en',
        target_lang: 'es',
        content: 'Hello',
        callback_url: 'http://192.168.1.1/callback',
      });

    expect(response.status).toBe(400);
    expect(response.body.code).toBe('VALIDATION_ERROR');
    expect(response.body.errors[0].message).toContain('private network');
  });

  it('should block AWS metadata endpoint', async () => {
    const response = await request(app)
      .post('/v1/translate')
      .set('Authorization', `Bearer ${validApiKey}`)
      .send({
        source_lang: 'en',
        target_lang: 'es',
        content: 'Hello',
        callback_url: 'http://169.254.169.254/latest/meta-data/',
      });

    expect(response.status).toBe(400);
  });
});
```

---

## Security Monitoring

### Metrics to Track

```typescript
// api/src/utils/metrics.ts

import { Counter, Histogram } from 'prom-client';

// Authentication failures
export const authFailuresCounter = new Counter({
  name: 'auth_failures_total',
  help: 'Total number of authentication failures',
  labelNames: ['auth_type', 'reason'],
});

// Rate limit hits
export const rateLimitHitsCounter = new Counter({
  name: 'rate_limit_hits_total',
  help: 'Total number of rate limit hits',
  labelNames: ['endpoint', 'user_tier'],
});

// Webhook signature failures
export const webhookSignatureFailuresCounter = new Counter({
  name: 'webhook_signature_failures_total',
  help: 'Total number of webhook signature verification failures',
  labelNames: ['webhook_type'],
});

// Track errors
export function trackError(errorType: string, errorCode: string) {
  authFailuresCounter.labels(errorType, errorCode).inc();
}
```

### Alerting Rules (Prometheus)

```yaml
# prometheus/alerts.yml

groups:
  - name: security
    interval: 30s
    rules:
      - alert: HighAuthFailureRate
        expr: rate(auth_failures_total[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High authentication failure rate"
          description: "{{ $value }} auth failures per second"

      - alert: RateLimitAbuse
        expr: rate(rate_limit_hits_total[5m]) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Possible rate limit abuse"

      - alert: WebhookSignatureFailures
        expr: rate(webhook_signature_failures_total[5m]) > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Webhook signature verification failures"
```

---

## Common Security Pitfalls to Avoid

### 1. Don't Trust User Input
```typescript
// ❌ BAD
const job = await prisma.translationJob.findUnique({
  where: { id: req.params.id }, // No user_id check!
});

// ✅ GOOD
const job = await prisma.translationJob.findFirst({
  where: {
    id: req.params.id,
    user_id: req.user!.userId, // Ensure user owns resource
  },
});
```

### 2. Don't Log Sensitive Data
```typescript
// ❌ BAD
logger.info('User login', { email, password });

// ✅ GOOD
logger.info('User login', { email, success: true });
```

### 3. Don't Use Weak Secrets
```typescript
// ❌ BAD
const JWT_SECRET = 'secret123';

// ✅ GOOD
const JWT_SECRET = process.env.JWT_ACCESS_SECRET; // 256-bit random
```

### 4. Don't Expose Stack Traces
```typescript
// ❌ BAD
res.status(500).json({ error: error.stack });

// ✅ GOOD
if (isProduction()) {
  res.status(500).json({ error: 'Internal server error' });
} else {
  res.status(500).json({ error: error.message, stack: error.stack });
}
```

### 5. Don't Skip Input Validation
```typescript
// ❌ BAD
router.post('/translate', async (req, res) => {
  const { content } = req.body; // No validation!
  // ...
});

// ✅ GOOD
router.post('/translate', validate(translateSchema), async (req, res) => {
  const { content } = req.body; // Type-safe, validated
  // ...
});
```

---

## Further Reading

- [OWASP Top 10](https://owasp.org/www-project-top-ten/)
- [Node.js Security Best Practices](https://nodejs.org/en/docs/guides/security/)
- [Express Security Best Practices](https://expressjs.com/en/advanced/best-practice-security.html)
- [JWT Security Best Practices](https://tools.ietf.org/html/rfc8725)
- [SSRF Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html)

# Skill: API Endpoint Reference

## Identity
- **Skill ID**: `api-endpoint-reference`
- **Domain**: REST API Documentation, OpenAPI Specification
- **Technologies**: Express.js, Zod Validation, JWT/API Key Auth, Rate Limiting
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when:
- Implementing new API endpoints
- Modifying existing endpoint behavior
- Debugging API request/response issues
- Integrating WordPress plugin with backend
- Implementing API client libraries
- Understanding authentication flows
- Configuring rate limiting strategies
- Documenting API for external developers

**File patterns:**
- `api/src/routes/**/*.ts`
- `api/src/middleware/validator.ts`
- `api/src/middleware/auth.ts`
- `api/src/middleware/rateLimiter.ts`

---

## Complete Endpoint Catalog (57 Endpoints)

### Authentication Methods

| Method | Header Format | Use Case |
|--------|---------------|----------|
| **JWT** | `Authorization: Bearer <jwt_token>` | User dashboard access, subscription management |
| **API Key** | `Authorization: Bearer sk_live_...` or `sk_test_...` | WordPress plugin, translation requests |
| **Admin JWT** | `Authorization: Bearer <admin_jwt_token>` | Admin panel operations |
| **PayPal Signature** | PayPal webhook headers | Webhook event verification |

---

## 1. Health Check Endpoints (3)

### 1.1 Basic Health Check
```http
GET /health
```

**Auth**: None
**Rate Limit**: None

**Response** (200):
```json
{
  "status": "healthy",
  "service": "translate-api",
  "version": "1.0.0",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

### 1.2 Readiness Check
```http
GET /health/ready
```

**Auth**: None
**Rate Limit**: None

**Response** (200):
```json
{
  "status": "ready",
  "checks": {
    "database": { "status": "healthy" },
    "redis": { "status": "healthy" }
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Response** (503 - Not Ready):
```json
{
  "status": "not ready",
  "checks": {
    "database": { "status": "unhealthy", "error": "Connection refused" },
    "redis": { "status": "healthy" }
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

### 1.3 Liveness Check
```http
GET /health/live
```

**Auth**: None
**Rate Limit**: None

**Response** (200):
```json
{
  "status": "alive",
  "uptime": 86400.5,
  "memory": {
    "rss": 52428800,
    "heapTotal": 18874368,
    "heapUsed": 12345678,
    "external": 1234567
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

## 2. Authentication Endpoints (6)

### 2.1 User Registration
```http
POST /v1/auth/register
```

**Auth**: None
**Rate Limit**: 100 requests / 15 minutes (per IP)

**Request Schema** (Zod):
```typescript
z.object({
  email: z.string().email(),
  password: z.string().min(8)
})
```

**Request**:
```json
{
  "email": "user@example.com",
  "password": "SecurePass123"
}
```

**Response** (201):
```json
{
  "message": "User registered successfully. Please check your email to verify your account.",
  "userId": "550e8400-e29b-41d4-a716-446655440000"
}
```

**Errors**:
- `400 USER_EXISTS` - Email already registered
- `400 VALIDATION_ERROR` - Invalid email or password

---

### 2.2 User Login
```http
POST /v1/auth/login
```

**Auth**: None
**Rate Limit**: 100 requests / 15 minutes (per IP)

**Request Schema**:
```typescript
z.object({
  email: z.string().email(),
  password: z.string()
})
```

**Request**:
```json
{
  "email": "user@example.com",
  "password": "SecurePass123"
}
```

**Response** (200):
```json
{
  "accessToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refreshToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "user": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "email": "user@example.com",
    "email_verified": true,
    "plan": "professional",
    "subscriptionStatus": "active"
  }
}
```

**Errors**:
- `401 INVALID_CREDENTIALS` - Wrong email or password
- `403 ACCOUNT_SUSPENDED` - Account is suspended

---

### 2.3 Refresh Access Token
```http
POST /v1/auth/refresh
```

**Auth**: None (requires refresh token in body)
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  refreshToken: z.string()
})
```

**Request**:
```json
{
  "refreshToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
```

**Response** (200):
```json
{
  "accessToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
```

**Errors**:
- `401 TOKEN_EXPIRED` - Refresh token expired
- `401 INVALID_TOKEN` - Invalid refresh token

---

### 2.4 Verify Email
```http
POST /v1/auth/verify-email
```

**Auth**: None
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  token: z.string()
})
```

**Request**:
```json
{
  "token": "a1b2c3d4e5f6..."
}
```

**Response** (200):
```json
{
  "message": "Email verified successfully"
}
```

**Errors**:
- `400 INVALID_TOKEN` - Invalid or expired token

---

### 2.5 Forgot Password
```http
POST /v1/auth/forgot-password
```

**Auth**: None
**Rate Limit**: 5 requests / 1 hour (per IP)

**Request Schema**:
```typescript
z.object({
  email: z.string().email()
})
```

**Request**:
```json
{
  "email": "user@example.com"
}
```

**Response** (200):
```json
{
  "message": "If a user with that email exists, a password reset link has been sent."
}
```

---

### 2.6 Reset Password
```http
POST /v1/auth/reset-password
```

**Auth**: None
**Rate Limit**: 100 requests / 15 minutes (per IP)

**Request Schema**:
```typescript
z.object({
  token: z.string(),
  password: z.string().min(8)
})
```

**Request**:
```json
{
  "token": "a1b2c3d4e5f6...",
  "password": "NewSecurePass456"
}
```

**Response** (200):
```json
{
  "message": "Password reset successfully"
}
```

**Errors**:
- `400 INVALID_TOKEN` - Invalid or expired reset token

---

## 3. User Account Endpoints (11)

### 3.1 Validate API Key
```http
GET /v1/account/validate
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "valid": true,
  "user": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "email": "user@example.com"
  }
}
```

---

### 3.2 Get Credit Balance
```http
GET /v1/account/credits
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "credits_balance": 450000,
  "subscription": {
    "tier": "professional",
    "status": "active",
    "current_period_end": "2026-02-27T12:00:00.000Z"
  }
}
```

---

### 3.3 Get Account Details
```http
GET /v1/account
```

**Auth**: JWT
**Rate Limit**: None

**Response** (200):
```json
{
  "user": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "email": "user@example.com",
    "emailVerified": true,
    "status": "active",
    "createdAt": "2025-12-15T10:00:00.000Z"
  },
  "subscription": {
    "planTier": "professional",
    "billingCycle": "monthly",
    "status": "active",
    "currentPeriodStart": "2026-01-27T12:00:00.000Z",
    "currentPeriodEnd": "2026-02-27T12:00:00.000Z",
    "cancelAtPeriodEnd": false
  },
  "credits": {
    "balance": 450000,
    "allocation": 500000
  }
}
```

---

### 3.4 Create API Key
```http
POST /v1/account/api-keys
```

**Auth**: JWT
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  name: z.string().min(1).max(100)
})
```

**Request**:
```json
{
  "name": "Production WordPress Site"
}
```

**Response** (201):
```json
{
  "id": "660f9500-f39c-52e5-b827-557766551111",
  "key": "sk_live_abc123def456ghi789jkl012mno345pqr678stu901",
  "prefix": "sk_live_",
  "name": "Production WordPress Site",
  "isActive": true,
  "lastUsedAt": null,
  "createdAt": "2026-01-27T12:00:00.000Z"
}
```

**Note**: The full `key` is only returned on creation. Store it securely!

---

### 3.5 List API Keys
```http
GET /v1/account/api-keys
```

**Auth**: JWT
**Rate Limit**: None

**Response** (200):
```json
{
  "apiKeys": [
    {
      "id": "660f9500-f39c-52e5-b827-557766551111",
      "name": "Production WordPress Site",
      "prefix": "sk_live_",
      "isActive": true,
      "lastUsedAt": "2026-01-27T11:30:00.000Z",
      "createdAt": "2025-12-20T14:00:00.000Z"
    }
  ]
}
```

---

### 3.6 Revoke API Key
```http
DELETE /v1/account/api-keys/:keyId
```

**Auth**: JWT
**Rate Limit**: None

**Response** (200):
```json
{
  "message": "API key revoked successfully"
}
```

---

### 3.7 Get Translation Jobs (Account)
```http
GET /v1/account/jobs?status=completed&limit=50
```

**Auth**: API Key
**Rate Limit**: None

**Query Parameters**:
- `status` (optional): Filter by status (`pending`, `processing`, `completed`, `failed`)
- `limit` (optional): Max 100, default 50

**Response** (200):
```json
{
  "jobs": [
    {
      "id": "770fa611-g40d-63f6-c938-668877662222",
      "status": "completed",
      "source_lang": "en",
      "target_lang": "es",
      "model": "4b",
      "tokens_used": 1250,
      "cost": "0.0025",
      "error_message": null,
      "created_at": "2026-01-27T10:00:00.000Z",
      "completed_at": "2026-01-27T10:00:05.123Z"
    }
  ]
}
```

---

### 3.8 Get Usage History
```http
GET /v1/account/usage?page=1&limit=50
```

**Auth**: JWT
**Rate Limit**: None

**Query Parameters**:
- `page` (optional): Page number (1-indexed)
- `limit` (optional): Items per page

**Response** (200):
```json
{
  "transactions": [
    {
      "id": "880fb722-h51e-74g7-d049-779988773333",
      "type": "deduction",
      "amount": -1250,
      "balance_after": 448750,
      "description": "Translation job 770fa611",
      "created_at": "2026-01-27T10:00:05.123Z"
    }
  ],
  "jobs": [],
  "pagination": {
    "page": 1,
    "limit": 50,
    "total": 125,
    "totalPages": 3
  }
}
```

---

### 3.9 List Subscription Plans
```http
GET /v1/subscriptions/plans
```

**Auth**: None (Public)
**Rate Limit**: None

**Response** (200):
```json
{
  "plans": [
    {
      "tier": "starter",
      "name": "Starter",
      "monthlyPrice": 29,
      "annualPrice": 290,
      "credits": 100000,
      "models": ["4b"],
      "features": ["100K tokens/month", "4B model", "Email support"]
    },
    {
      "tier": "professional",
      "name": "Professional",
      "monthlyPrice": 99,
      "annualPrice": 990,
      "credits": 500000,
      "models": ["4b", "27b"],
      "features": ["500K tokens/month", "All models", "Priority support", "Higher rate limits"]
    },
    {
      "tier": "enterprise",
      "name": "Enterprise",
      "monthlyPrice": 299,
      "annualPrice": 2990,
      "credits": 2000000,
      "models": ["4b", "27b"],
      "features": ["2M tokens/month", "All models", "SLA", "Unlimited rate limits", "Dedicated support"]
    }
  ]
}
```

---

### 3.10 Create Checkout Session
```http
POST /v1/subscriptions/checkout
```

**Auth**: JWT
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  planTier: z.enum(['starter', 'professional', 'enterprise']),
  billingCycle: z.enum(['monthly', 'annual'])
})
```

**Request**:
```json
{
  "planTier": "professional",
  "billingCycle": "monthly"
}
```

**Response** (200):
```json
{
  "approval_url": "https://www.paypal.com/checkoutnow?token=EC-abc123",
  "subscription_id": "I-abc123def456"
}
```

---

### 3.11 Get Subscription Details
```http
GET /v1/account/subscription
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "tier": "professional",
  "status": "active",
  "billing_cycle": "monthly",
  "current_period_start": "2026-01-27",
  "current_period_end": "2026-02-27",
  "next_billing_date": "2026-02-27",
  "auto_renew": true,
  "paypal_subscription_id": "I-abc123def456",
  "created_at": "2025-12-20T14:00:00.000Z",
  "updated_at": "2026-01-27T12:00:00.000Z"
}
```

---

## 4. Translation Endpoints (2)

### 4.1 Synchronous Translation
```http
POST /v1/translate
```

**Auth**: API Key
**Rate Limit**: Tier-based (Starter: 10/min, Pro: 30/min, Enterprise: Unlimited)

**Request Schema**:
```typescript
z.object({
  sourceLang: z.string().regex(/^[a-z]{2}(-[a-z]{2})?$/),
  targetLang: z.string().regex(/^[a-z]{2}(-[a-z]{2})?$/),
  content: z.string().min(1).max(5000),
  model: z.enum(['4b', '27b']),
  tone: z.enum(['neutral', 'formal', 'casual']).optional(),
  clientJobId: z.string().optional()
})
```

**Request**:
```json
{
  "sourceLang": "en",
  "targetLang": "es",
  "content": "Hello, how are you?",
  "model": "4b",
  "tone": "neutral",
  "clientJobId": "wp_post_123"
}
```

**Response** (200):
```json
{
  "success": true,
  "translation": "Hola, ¿cómo estás?",
  "tokens_used": 15,
  "cost_usd": 0.00003,
  "job_id": "770fa611-g40d-63f6-c938-668877662222",
  "status": "completed",
  "processing_time_ms": 1234,
  "data": {
    "jobId": "770fa611-g40d-63f6-c938-668877662222",
    "status": "completed",
    "translation": "Hola, ¿cómo estás?",
    "tokensUsed": 15,
    "cost": 0.00003,
    "processingTimeMs": 1234,
    "creditBalance": 448735
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Errors**:
- `402 INSUFFICIENT_CREDITS` - Not enough credits
- `413 CONTENT_TOO_LONG` - Content exceeds 5000 characters
- `500 TRANSLATION_FAILED` - Translation service error

**WordPress Payload Transform**: The middleware automatically transforms WordPress camelCase to snake_case:
```json
// WordPress sends this:
{
  "source_lang": "en",
  "target_lang": "es",
  "content": "Hello",
  "model": "4b"
}

// Middleware transforms to:
{
  "sourceLang": "en",
  "targetLang": "es",
  "content": "Hello",
  "model": "4b"
}
```

---

### 4.2 Token Estimation
```http
POST /v1/estimate
```

**Auth**: API Key
**Rate Limit**: Tier-based

**Request Schema**:
```typescript
z.object({
  content: z.string().min(1).max(50000),
  source_lang: z.string().regex(/^[a-z]{2}(-[a-z]{2})?$/),
  target_langs: z.array(z.string()).min(1).max(20)
})
```

**Request**:
```json
{
  "content": "This is a sample text for estimation.",
  "source_lang": "en",
  "target_langs": ["es", "fr", "de"]
}
```

**Response** (200):
```json
{
  "success": true,
  "data": {
    "total_estimated_tokens": 135,
    "total_estimated_cost_usd": 0.00027,
    "per_language": [
      {
        "language": "es",
        "estimated_tokens": 45,
        "estimated_cost_usd": 0.00009,
        "confidence": "high"
      },
      {
        "language": "fr",
        "estimated_tokens": 45,
        "estimated_cost_usd": 0.00009,
        "confidence": "high"
      },
      {
        "language": "de",
        "estimated_tokens": 45,
        "estimated_cost_usd": 0.00009,
        "confidence": "high"
      }
    ]
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

## 5. Translation Jobs Endpoints (6)

### 5.1 Submit Async Job
```http
POST /v1/jobs
```

**Auth**: API Key
**Rate Limit**: Tier-based

**Request Schema**:
```typescript
z.object({
  sourceLang: z.string().regex(/^[a-z]{2}(-[a-z]{2})?$/),
  targetLang: z.string().regex(/^[a-z]{2}(-[a-z]{2})?$/),
  content: z.string().min(1).max(50000),
  model: z.enum(['4b', '27b']),
  tone: z.enum(['neutral', 'formal', 'casual']).optional(),
  callbackUrl: z.string().url().optional(),
  callbackSecret: z.string().optional(),
  clientJobId: z.string().optional()
})
```

**Request**:
```json
{
  "sourceLang": "en",
  "targetLang": "zh",
  "content": "Very long article content...",
  "model": "27b",
  "tone": "formal",
  "callbackUrl": "https://example.com/webhook/translation-complete",
  "callbackSecret": "your-hmac-secret",
  "clientJobId": "wp_post_456"
}
```

**Response** (202):
```json
{
  "success": true,
  "data": {
    "jobId": "880fb722-h51e-74g7-d049-779988773333",
    "status": "pending",
    "clientJobId": "wp_post_456",
    "estimatedCompletionTime": "2026-01-27T12:05:00.000Z"
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

### 5.2 Get Job Status
```http
GET /v1/jobs/:jobId
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200 - Pending):
```json
{
  "success": true,
  "data": {
    "jobId": "880fb722-h51e-74g7-d049-779988773333",
    "clientJobId": "wp_post_456",
    "status": "processing",
    "sourceLang": "en",
    "targetLang": "zh",
    "model": "27b",
    "tone": "formal",
    "createdAt": "2026-01-27T12:00:00.000Z",
    "updatedAt": "2026-01-27T12:00:15.000Z"
  },
  "timestamp": "2026-01-27T12:00:30.000Z"
}
```

**Response** (200 - Completed):
```json
{
  "success": true,
  "data": {
    "jobId": "880fb722-h51e-74g7-d049-779988773333",
    "clientJobId": "wp_post_456",
    "status": "completed",
    "sourceLang": "en",
    "targetLang": "zh",
    "model": "27b",
    "tone": "formal",
    "translation": "很长的文章内容...",
    "tokensUsed": 5678,
    "cost": 0.01136,
    "processingTimeMs": 12345,
    "createdAt": "2026-01-27T12:00:00.000Z",
    "updatedAt": "2026-01-27T12:02:15.000Z",
    "completedAt": "2026-01-27T12:02:15.000Z"
  },
  "timestamp": "2026-01-27T12:03:00.000Z"
}
```

**Errors**:
- `404 JOB_NOT_FOUND` - Job doesn't exist
- `403 ACCESS_DENIED` - Job belongs to different user

---

### 5.3 Cancel Job
```http
POST /v1/jobs/:jobId/cancel
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "success": true,
  "data": {
    "jobId": "880fb722-h51e-74g7-d049-779988773333",
    "status": "cancelled"
  },
  "message": "Job cancelled successfully",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Errors**:
- `400 CANNOT_CANCEL` - Job already completed or failed

---

### 5.4 List Jobs
```http
GET /v1/jobs?status=completed&limit=20&offset=0&sortBy=created_at&sortOrder=desc
```

**Auth**: API Key
**Rate Limit**: None

**Query Parameters**:
- `status` (optional): Comma-separated statuses
- `limit` (optional): Max 100, default 50
- `offset` (optional): Pagination offset
- `sortBy` (optional): `created_at`, `updated_at`
- `sortOrder` (optional): `asc`, `desc`

**Response** (200):
```json
{
  "success": true,
  "data": [
    {
      "jobId": "880fb722-h51e-74g7-d049-779988773333",
      "status": "completed",
      "sourceLang": "en",
      "targetLang": "zh",
      "model": "27b",
      "tokensUsed": 5678,
      "cost": 0.01136,
      "createdAt": "2026-01-27T12:00:00.000Z",
      "completedAt": "2026-01-27T12:02:15.000Z"
    }
  ],
  "pagination": {
    "total": 125,
    "limit": 20,
    "offset": 0,
    "hasMore": true
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

### 5.5 Retry Failed Job
```http
POST /v1/jobs/:jobId/retry
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "success": true,
  "data": {
    "jobId": "880fb722-h51e-74g7-d049-779988773333",
    "status": "pending"
  },
  "message": "Job queued for retry",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Errors**:
- `400 CANNOT_RETRY` - Job is not in failed state

---

### 5.6 Stream Job Status (SSE)
```http
GET /v1/jobs/:jobId/status/stream
```

**Auth**: API Key
**Rate Limit**: None
**Response Type**: Server-Sent Events (SSE)

**Headers**:
```
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
```

**Event Stream**:
```
data: {"jobId":"880fb722-h51e-74g7-d049-779988773333","status":"pending","createdAt":"2026-01-27T12:00:00.000Z"}

data: {"jobId":"880fb722-h51e-74g7-d049-779988773333","status":"processing","updatedAt":"2026-01-27T12:00:15.000Z"}

data: {"jobId":"880fb722-h51e-74g7-d049-779988773333","status":"completed","translation":"...","tokensUsed":5678,"cost":0.01136,"completedAt":"2026-01-27T12:02:15.000Z"}
```

**Usage (JavaScript)**:
```javascript
const eventSource = new EventSource(
  'https://api.press.zone/v1/jobs/880fb722-h51e-74g7-d049-779988773333/status/stream',
  {
    headers: {
      'Authorization': 'Bearer sk_live_...'
    }
  }
);

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Job status:', data.status);

  if (['completed', 'failed', 'cancelled'].includes(data.status)) {
    eventSource.close();
  }
};
```

---

## 6. Site Registration Endpoints (4)

### 6.1 Register WordPress Site
```http
POST /v1/sites/register
```

**Auth**: API Key
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  siteUrl: z.string().url(),
  siteName: z.string().min(1).max(255),
  wpVersion: z.string().min(1).max(50),
  pluginVersion: z.string().min(1).max(50),
  phpVersion: z.string().optional(),
  activeTheme: z.string().optional(),
  locale: z.string().optional(),
  timezone: z.string().optional(),
  installedPlugins: z.array(z.object({
    name: z.string(),
    version: z.string(),
    active: z.boolean()
  })).optional()
})
```

**Request**:
```json
{
  "siteUrl": "https://example.com",
  "siteName": "My WordPress Site",
  "wpVersion": "6.4.2",
  "pluginVersion": "1.2.3",
  "phpVersion": "8.2.0",
  "activeTheme": "twentytwentyfour",
  "locale": "en_US",
  "timezone": "America/New_York",
  "installedPlugins": [
    {
      "name": "WooCommerce",
      "version": "8.5.0",
      "active": true
    }
  ]
}
```

**Response** (201 - New Site):
```json
{
  "success": true,
  "siteId": "990gc833-i62f-85h8-e150-880099884444",
  "message": "Site registered successfully",
  "site": {
    "id": "990gc833-i62f-85h8-e150-880099884444",
    "siteUrl": "https://example.com",
    "siteName": "My WordPress Site",
    "wpVersion": "6.4.2",
    "pluginVersion": "1.2.3",
    "isActive": true,
    "createdAt": "2026-01-27T12:00:00.000Z",
    "lastSeenAt": "2026-01-27T12:00:00.000Z"
  }
}
```

**Response** (200 - Updated Existing):
```json
{
  "success": true,
  "siteId": "990gc833-i62f-85h8-e150-880099884444",
  "message": "Site updated successfully",
  "site": { /* ... */ }
}
```

---

### 6.2 List Registered Sites
```http
GET /v1/sites
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "success": true,
  "sites": [
    {
      "id": "990gc833-i62f-85h8-e150-880099884444",
      "siteUrl": "https://example.com",
      "siteName": "My WordPress Site",
      "wpVersion": "6.4.2",
      "pluginVersion": "1.2.3",
      "phpVersion": "8.2.0",
      "activeTheme": "twentytwentyfour",
      "locale": "en_US",
      "timezone": "America/New_York",
      "isActive": true,
      "createdAt": "2026-01-27T12:00:00.000Z",
      "lastSeenAt": "2026-01-27T12:00:00.000Z"
    }
  ]
}
```

---

### 6.3 Get Site Details
```http
GET /v1/sites/:siteId
```

**Auth**: API Key
**Rate Limit**: None

**Response** (200):
```json
{
  "success": true,
  "site": {
    "id": "990gc833-i62f-85h8-e150-880099884444",
    "siteUrl": "https://example.com",
    "siteName": "My WordPress Site",
    "wpVersion": "6.4.2",
    "pluginVersion": "1.2.3",
    "phpVersion": "8.2.0",
    "activeTheme": "twentytwentyfour",
    "locale": "en_US",
    "timezone": "America/New_York",
    "installedPlugins": [
      {
        "name": "WooCommerce",
        "version": "8.5.0",
        "active": true
      }
    ],
    "isActive": true,
    "createdAt": "2026-01-27T12:00:00.000Z",
    "lastSeenAt": "2026-01-27T12:00:00.000Z"
  }
}
```

**Errors**:
- `404 SITE_NOT_FOUND` - Site doesn't exist or doesn't belong to user

---

### 6.4 Update Site Details
```http
PATCH /v1/sites/:siteId
```

**Auth**: API Key
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  siteName: z.string().min(1).max(255).optional(),
  wpVersion: z.string().min(1).max(50).optional(),
  pluginVersion: z.string().min(1).max(50).optional(),
  phpVersion: z.string().optional(),
  activeTheme: z.string().optional(),
  locale: z.string().optional(),
  timezone: z.string().optional(),
  installedPlugins: z.array(/* ... */).optional(),
  isActive: z.boolean().optional()
})
```

**Request**:
```json
{
  "wpVersion": "6.4.3",
  "pluginVersion": "1.3.0"
}
```

**Response** (200):
```json
{
  "success": true,
  "message": "Site updated successfully",
  "site": {
    "id": "990gc833-i62f-85h8-e150-880099884444",
    "siteUrl": "https://example.com",
    "siteName": "My WordPress Site",
    "wpVersion": "6.4.3",
    "pluginVersion": "1.3.0",
    "isActive": true,
    "lastSeenAt": "2026-01-27T12:05:00.000Z"
  }
}
```

---

## 7. Pricing & Stats Endpoints (2)

### 7.1 Get Current Pricing
```http
GET /v1/pricing
```

**Auth**: None (Public)
**Rate Limit**: None

**Response** (200):
```json
{
  "pricing": {
    "per_1k_tokens": 0.002
  },
  "currency": "USD",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

### 7.2 Get Accuracy Statistics
```http
GET /v1/stats/accuracy?lang=es
```

**Auth**: API Key
**Rate Limit**: Tier-based

**Query Parameters**:
- `lang` (optional): Filter by target language

**Response** (200):
```json
{
  "success": true,
  "stats": [
    {
      "target_language": "es",
      "sample_count": 1250,
      "avg_confidence": 0.87,
      "avg_estimated_tokens": 450,
      "avg_actual_tokens": 465,
      "estimation_accuracy": 0.97,
      "last_updated": "2026-01-27T12:00:00.000Z"
    }
  ],
  "total_languages": 1,
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

## 8. Webhook Endpoint (1)

### 8.1 PayPal Webhook Handler
```http
POST /v1/webhooks/paypal
```

**Auth**: PayPal Signature Verification
**Rate Limit**: 100 requests / 1 minute

**Headers Required**:
- `paypal-transmission-id`
- `paypal-transmission-time`
- `paypal-transmission-sig`
- `paypal-cert-url`
- `paypal-auth-algo`

**Request** (Example - Subscription Activated):
```json
{
  "id": "WH-abc123",
  "event_type": "BILLING.SUBSCRIPTION.ACTIVATED",
  "event_version": "1.0",
  "create_time": "2026-01-27T12:00:00Z",
  "resource_type": "subscription",
  "resource": {
    "id": "I-abc123def456",
    "plan_id": "P-starter-monthly",
    "subscriber": {
      "email_address": "user@example.com"
    },
    "status": "ACTIVE"
  }
}
```

**Response** (200):
```json
{
  "received": true
}
```

**Errors**:
- `401 INVALID_SIGNATURE` - PayPal signature verification failed

**Supported Event Types**:
- `BILLING.SUBSCRIPTION.ACTIVATED` - Credit allocation
- `BILLING.SUBSCRIPTION.CANCELLED` - Mark subscription as cancelled
- `BILLING.SUBSCRIPTION.SUSPENDED` - Suspend subscription
- `BILLING.SUBSCRIPTION.UPDATED` - Update plan tier
- `PAYMENT.SALE.COMPLETED` - Record payment

---

## 9. Admin Authentication Endpoints (2)

### 9.1 Admin Login
```http
POST /v1/admin/auth/login
```

**Auth**: None
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  email: z.string().email(),
  password: z.string()
})
```

**Request**:
```json
{
  "email": "admin@press.zone",
  "password": "AdminSecurePass123"
}
```

**Response** (200):
```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "user": {
    "id": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
    "email": "admin@press.zone",
    "role": "admin",
    "created_at": "2025-01-01T00:00:00.000Z"
  }
}
```

**Development Mode Bypass**:
```
Authorization: Bearer dev-admin-token
```

---

### 9.2 Admin Refresh Token
```http
POST /v1/admin/auth/refresh
```

**Auth**: None (requires refresh token)
**Rate Limit**: None

**Request Schema**:
```typescript
z.object({
  refreshToken: z.string()
})
```

**Response** (200):
```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
```

---

## 10. Admin Analytics Endpoints (4)

### 10.1 Dashboard Metrics
```http
GET /v1/admin/analytics/dashboard
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "users": {
    "total": 1250,
    "active": 980,
    "new30Days": 125
  },
  "subscriptions": {
    "active": 750,
    "starter": 400,
    "professional": 300,
    "enterprise": 50
  },
  "translations": {
    "total": 125000,
    "today": 450,
    "thisMonth": 12500,
    "completed": 123000,
    "failed": 500,
    "avgProcessingTimeMs": 1234
  },
  "revenue": {
    "total": "125000.00",
    "thisMonth": "9850.00",
    "lastMonth": "9200.00"
  },
  "system": {
    "errorRate": 0.4,
    "avgTokensPerJob": 1250
  }
}
```

---

### 10.2 Revenue Breakdown
```http
GET /v1/admin/analytics/revenue?days=30&groupBy=day
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Query Parameters**:
- `days` (optional): Max 365, default 30
- `groupBy` (optional): `day` (only option currently)

**Response** (200):
```json
[
  {
    "date": "2026-01-27",
    "amount": "350.00",
    "subscriptions": 5,
    "transactions": 8
  },
  {
    "date": "2026-01-26",
    "amount": "420.00",
    "subscriptions": 6,
    "transactions": 10
  }
]
```

---

### 10.3 Usage Statistics
```http
GET /v1/admin/analytics/usage?days=30
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
[
  {
    "date": "2026-01-27",
    "translations": 450,
    "tokensUsed": 562500,
    "uniqueUsers": 125
  }
]
```

---

### 10.4 Model Usage
```http
GET /v1/admin/analytics/models?days=30
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
[
  {
    "model": "4b",
    "count": 95000,
    "tokensUsed": 118750000,
    "avgProcessingTimeMs": 980,
    "successRate": 99.5
  },
  {
    "model": "27b",
    "count": 30000,
    "tokensUsed": 37500000,
    "avgProcessingTimeMs": 1850,
    "successRate": 99.2
  }
]
```

---

## 11. Admin User Management Endpoints (6)

### 11.1 List Users
```http
GET /v1/admin/users?page=1&limit=20&search=user@example.com&status=active&plan=professional&sortBy=created_at&sortOrder=desc
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Query Parameters**:
- `page` (optional): Page number (1-indexed)
- `limit` (optional): Max 100, default 20
- `search` (optional): Email search (case-insensitive)
- `status` (optional): Filter by status
- `plan` (optional): Filter by plan tier
- `sortBy` (optional): `created_at`, `updated_at`, `email`, `status`
- `sortOrder` (optional): `asc`, `desc`

**Response** (200):
```json
{
  "data": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "email": "user@example.com",
      "emailVerified": true,
      "status": "active",
      "subscription": {
        "plan": "professional",
        "billingCycle": "monthly",
        "status": "active",
        "currentPeriodEnd": "2026-02-27T12:00:00.000Z",
        "cancelAtPeriodEnd": false
      },
      "stats": {
        "apiKeys": 2,
        "translationJobs": 350,
        "payments": 12
      },
      "createdAt": "2025-12-15T10:00:00.000Z",
      "updatedAt": "2026-01-27T12:00:00.000Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 1250,
    "totalPages": 63,
    "hasNext": true,
    "hasPrev": false
  }
}
```

---

### 11.2 Get User Details
```http
GET /v1/admin/users/:id
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "email": "user@example.com",
  "emailVerified": true,
  "status": "active",
  "subscription": {
    "id": "660f9500-f39c-52e5-b827-557766551111",
    "plan": "professional",
    "billingCycle": "monthly",
    "status": "active",
    "paypalSubscriptionId": "I-abc123def456",
    "currentPeriodStart": "2026-01-27T12:00:00.000Z",
    "currentPeriodEnd": "2026-02-27T12:00:00.000Z",
    "cancelAtPeriodEnd": false,
    "createdAt": "2025-12-20T14:00:00.000Z",
    "updatedAt": "2026-01-27T12:00:00.000Z"
  },
  "creditBalance": 450000,
  "apiKeys": [
    {
      "id": "770fa611-g40d-63f6-c938-668877662222",
      "name": "Production Site",
      "prefix": "sk_live_",
      "isActive": true,
      "lastUsedAt": "2026-01-27T11:30:00.000Z",
      "createdAt": "2025-12-20T14:30:00.000Z"
    }
  ],
  "recentJobs": [ /* last 10 jobs */ ],
  "recentPayments": [ /* last 10 payments */ ],
  "recentCreditTransactions": [ /* last 10 transactions */ ],
  "createdAt": "2025-12-15T10:00:00.000Z",
  "updatedAt": "2026-01-27T12:00:00.000Z"
}
```

---

### 11.3 Update User
```http
PATCH /v1/admin/users/:id
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Request**:
```json
{
  "email": "newemail@example.com",
  "emailVerified": true,
  "status": "active"
}
```

**Response** (200):
```json
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "email": "newemail@example.com",
  "emailVerified": true,
  "status": "active",
  "subscription": {
    "plan": "professional",
    "status": "active"
  },
  "updatedAt": "2026-01-27T12:05:00.000Z"
}
```

**Errors**:
- `400 EMAIL_TAKEN` - Email already in use by another user
- `400 INVALID_STATUS` - Invalid status value

---

### 11.4 Suspend User
```http
POST /v1/admin/users/:id/suspend
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Request**:
```json
{
  "reason": "Terms of service violation"
}
```

**Response** (200):
```json
{
  "success": true,
  "message": "User suspended successfully",
  "userId": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Errors**:
- `400 ALREADY_SUSPENDED` - User already suspended

---

### 11.5 Activate User
```http
POST /v1/admin/users/:id/activate
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "success": true,
  "message": "User activated successfully",
  "userId": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Errors**:
- `400 ALREADY_ACTIVE` - User already active
- `400 CANNOT_ACTIVATE_DELETED` - Cannot activate deleted users

---

### 11.6 Revoke All API Keys
```http
POST /v1/admin/users/:id/api-keys/revoke
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "success": true,
  "message": "Revoked 3 API key(s)",
  "keysRevoked": 3,
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

---

## 12. Admin Jobs Endpoints (2)

### 12.1 List All Jobs
```http
GET /v1/admin/jobs?page=1&limit=50&status=completed&model=4b&userId=550e8400-e29b-41d4-a716-446655440000&startDate=2026-01-01&endDate=2026-01-31&sortBy=created_at&sortOrder=desc
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Query Parameters**:
- `page`, `limit`, `sortBy`, `sortOrder` (same as user list)
- `status` (optional): Comma-separated statuses
- `model` (optional): Comma-separated models
- `userId` (optional): Filter by user
- `sourceLang` (optional): Filter by source language
- `targetLang` (optional): Filter by target language
- `startDate`, `endDate` (optional): Date range

**Response** (200):
```json
{
  "data": [
    {
      "id": "770fa611-g40d-63f6-c938-668877662222",
      "userId": "550e8400-e29b-41d4-a716-446655440000",
      "userEmail": "user@example.com",
      "clientJobId": "wp_post_123",
      "status": "completed",
      "sourceLang": "en",
      "targetLang": "es",
      "model": "4b",
      "tokensUsed": 1250,
      "cost": "0.0025",
      "callbackUrl": "https://example.com/webhook",
      "webhookDeliveries": 1,
      "lastWebhookStatus": true,
      "createdAt": "2026-01-27T10:00:00.000Z",
      "completedAt": "2026-01-27T10:00:05.123Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 50,
    "total": 125000,
    "totalPages": 2500,
    "hasNext": true,
    "hasPrev": false
  }
}
```

---

### 12.2 Get Job Details
```http
GET /v1/admin/jobs/:id
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "jobId": "770fa611-g40d-63f6-c938-668877662222",
  "userId": "550e8400-e29b-41d4-a716-446655440000",
  "userEmail": "user@example.com",
  "clientJobId": "wp_post_123",
  "status": "completed",
  "sourceLang": "en",
  "targetLang": "es",
  "model": "4b",
  "tone": "neutral",
  "contentPreview": "Hello, how are you? This is a sample text for...",
  "translation": "Hola, ¿cómo estás? Este es un texto de muestra para...",
  "translationPreview": "Hola, ¿cómo estás? Este es un texto de muestra...",
  "tokensUsed": 1250,
  "cost": 0.0025,
  "processingTimeMs": 1234,
  "callbackUrl": "https://example.com/webhook",
  "webhookDeliveries": [
    {
      "id": "880fb722-h51e-74g7-d049-779988773333",
      "attemptNumber": 1,
      "success": true,
      "httpStatus": 200,
      "responseBody": "{\"received\":true}",
      "errorMessage": null,
      "attemptedAt": "2026-01-27T10:00:06.000Z"
    }
  ],
  "createdAt": "2026-01-27T10:00:00.000Z",
  "updatedAt": "2026-01-27T10:00:05.123Z",
  "completedAt": "2026-01-27T10:00:05.123Z"
}
```

---

## 13. Admin Transactions Endpoints (3)

### 13.1 List Credit Transactions
```http
GET /v1/admin/transactions?page=1&limit=20&userId=550e8400-e29b-41d4-a716-446655440000&type=deduction&startDate=2026-01-01&endDate=2026-01-31&sortBy=created_at&sortOrder=desc&format=json
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Query Parameters**:
- Standard pagination + filtering
- `format` (optional): `json` (default) or `csv`

**Response** (200 - JSON):
```json
{
  "data": [
    {
      "id": "990gc833-i62f-85h8-e150-880099884444",
      "userId": "550e8400-e29b-41d4-a716-446655440000",
      "user": {
        "email": "user@example.com",
        "status": "active"
      },
      "type": "deduction",
      "amount": -1250,
      "balanceAfter": 448750,
      "description": "Translation job 770fa611",
      "relatedJob": {
        "id": "770fa611-g40d-63f6-c938-668877662222",
        "status": "completed",
        "sourceLang": "en",
        "targetLang": "es",
        "model": "4b"
      },
      "relatedPayment": null,
      "createdAt": "2026-01-27T10:00:05.123Z"
    }
  ],
  "pagination": { /* ... */ }
}
```

**Response** (200 - CSV):
```csv
Transaction ID,User Email,Type,Amount,Balance After,Description,Related Job ID,Related Payment ID,Created At
990gc833-i62f-85h8-e150-880099884444,user@example.com,deduction,-1250,448750,"Translation job 770fa611",770fa611-g40d-63f6-c938-668877662222,,2026-01-27T10:00:05.123Z
```

---

### 13.2 Get Transaction Details
```http
GET /v1/admin/transactions/:id
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "id": "990gc833-i62f-85h8-e150-880099884444",
  "userId": "550e8400-e29b-41d4-a716-446655440000",
  "user": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "email": "user@example.com",
    "status": "active",
    "subscription": {
      "plan": "professional",
      "status": "active"
    }
  },
  "type": "deduction",
  "amount": -1250,
  "balanceAfter": 448750,
  "description": "Translation job 770fa611",
  "relatedJob": {
    "id": "770fa611-g40d-63f6-c938-668877662222",
    "status": "completed",
    "sourceLang": "en",
    "targetLang": "es",
    "model": "4b",
    "tone": "neutral",
    "tokensUsed": 1250,
    "cost": "0.0025",
    "createdAt": "2026-01-27T10:00:00.000Z",
    "completedAt": "2026-01-27T10:00:05.123Z"
  },
  "relatedPayment": null,
  "createdAt": "2026-01-27T10:00:05.123Z"
}
```

---

### 13.3 List Payments
```http
GET /v1/admin/transactions/payments?page=1&limit=20&userId=550e8400-e29b-41d4-a716-446655440000&status=completed&type=subscription_payment&startDate=2026-01-01&format=json
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "data": [
    {
      "id": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
      "userId": "550e8400-e29b-41d4-a716-446655440000",
      "user": {
        "email": "user@example.com",
        "status": "active"
      },
      "paypalPaymentId": "PAYID-abc123",
      "amount": "99.00",
      "currency": "USD",
      "status": "completed",
      "type": "subscription_payment",
      "subscription": {
        "id": "660f9500-f39c-52e5-b827-557766551111",
        "plan": "professional",
        "status": "active"
      },
      "createdAt": "2026-01-27T12:00:00.000Z"
    }
  ],
  "pagination": { /* ... */ }
}
```

---

## 14. Admin Settings Endpoints (4)

### 14.1 List All Settings
```http
GET /v1/admin/settings
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
[
  {
    "id": "bb2cc3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6",
    "key": "pricing_per_1k_tokens",
    "value": 0.002,
    "description": "Price per 1000 tokens for translation (USD)",
    "updatedAt": "2026-01-15T10:00:00.000Z",
    "updated_by": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
    "updated_byEmail": "admin@press.zone"
  },
  {
    "id": "cc3dd4e5-f6g7-h8i9-j0k1-l2m3n4o5p6q7",
    "key": "credits_professional",
    "value": 500000,
    "description": "Credit allocation for professional tier",
    "updatedAt": "2026-01-15T10:00:00.000Z",
    "updated_by": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
    "updated_byEmail": "admin@press.zone"
  }
]
```

---

### 14.2 Get Single Setting
```http
GET /v1/admin/settings/:key
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Response** (200):
```json
{
  "id": "bb2cc3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6",
  "key": "pricing_per_1k_tokens",
  "value": 0.002,
  "description": "Price per 1000 tokens for translation (USD)",
  "updatedAt": "2026-01-15T10:00:00.000Z",
  "updated_by": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5",
  "updated_byEmail": "admin@press.zone"
}
```

**Errors**:
- `404 SETTING_NOT_FOUND` - Setting key doesn't exist

---

### 14.3 Batch Update Settings
```http
PATCH /v1/admin/settings
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Request Schema**:
```typescript
z.object({
  settings: z.array(z.object({
    key: z.string().min(1),
    value: z.unknown()
  }))
})
```

**Request**:
```json
{
  "settings": [
    {
      "key": "pricing_per_1k_tokens",
      "value": 0.0025
    },
    {
      "key": "credits_professional",
      "value": 600000
    }
  ]
}
```

**Response** (200):
```json
{
  "success": true,
  "message": "2 setting(s) updated successfully",
  "settings": [
    {
      "id": "bb2cc3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6",
      "key": "pricing_per_1k_tokens",
      "value": 0.0025,
      "description": "Price per 1000 tokens for translation (USD)",
      "updatedAt": "2026-01-27T12:00:00.000Z",
      "updated_by": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5"
    }
  ],
  "configRefreshed": ["pricing", "credits"]
}
```

**Errors**:
- `400 VALIDATION_ERROR` - Invalid setting key or value
- `400 UNKNOWN_SETTING` - Setting key not recognized

**Supported Settings**:
- `pricing_per_1k_tokens` (number)
- `credits_starter`, `credits_professional`, `credits_enterprise` (number)
- `rate_limit_starter`, `rate_limit_professional`, `rate_limit_enterprise` (number)
- `paypal_client_id`, `paypal_client_secret`, `paypal_webhook_id` (string)
- `paypal_mode` (enum: "sandbox" | "live")

---

### 14.4 Update Single Setting
```http
PUT /v1/admin/settings/:key
```

**Auth**: Admin JWT
**Rate Limit**: 300 requests / 1 minute

**Request**:
```json
{
  "value": 0.003
}
```

**Response** (200):
```json
{
  "id": "bb2cc3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6",
  "key": "pricing_per_1k_tokens",
  "value": 0.003,
  "description": "Price per 1000 tokens for translation (USD)",
  "updatedAt": "2026-01-27T12:00:00.000Z",
  "updated_by": "aa1bb2c3-d4e5-f6g7-h8i9-j0k1l2m3n4o5"
}
```

---

## 15. Metrics Endpoint (1)

### 15.1 Prometheus Metrics
```http
GET /metrics
```

**Auth**: None (should be secured at infrastructure level)
**Rate Limit**: None

**Response** (200 - Prometheus format):
```
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/v1/translate",status="200"} 12500

# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1",method="POST",route="/v1/translate"} 9500
http_request_duration_seconds_bucket{le="0.5",method="POST",route="/v1/translate"} 11800
http_request_duration_seconds_sum{method="POST",route="/v1/translate"} 1234.56
http_request_duration_seconds_count{method="POST",route="/v1/translate"} 12000

# HELP translation_jobs_total Total number of translation jobs by status
# TYPE translation_jobs_total counter
translation_jobs_total{status="completed"} 123000
translation_jobs_total{status="failed"} 500

# HELP active_subscriptions Current number of active subscriptions by tier
# TYPE active_subscriptions gauge
active_subscriptions{tier="starter"} 400
active_subscriptions{tier="professional"} 300
active_subscriptions{tier="enterprise"} 50
```

---

## Rate Limiting Strategy

### Tier-Based Limits (API Endpoints)

| Tier | Requests/Minute | Applies To |
|------|-----------------|------------|
| **Starter** | 10 | Translation, Jobs, Estimation |
| **Professional** | 30 | Translation, Jobs, Estimation |
| **Enterprise** | Unlimited | All endpoints |

### Public Endpoints

| Endpoint | Limit | Window |
|----------|-------|--------|
| `/v1/auth/register` | 100 | 15 minutes |
| `/v1/auth/login` | 100 | 15 minutes |
| `/v1/auth/forgot-password` | 5 | 1 hour |
| `/v1/auth/reset-password` | 100 | 15 minutes |

### Admin Endpoints

| Endpoint | Limit | Window |
|----------|-------|--------|
| All admin routes | 300 | 1 minute |

### Webhook Endpoints

| Endpoint | Limit | Window |
|----------|-------|--------|
| `/v1/webhooks/paypal` | 100 | 1 minute |

---

## Error Response Format

All error responses follow this structure:

```json
{
  "error": true,
  "code": "ERROR_CODE",
  "message": "Human-readable error message",
  "details": { /* optional additional context */ },
  "requestId": "req_1706356800123_abc123def",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

### Validation Errors

```json
{
  "error": true,
  "code": "VALIDATION_ERROR",
  "message": "Request validation failed",
  "errors": [
    {
      "field": "email",
      "message": "Invalid email",
      "rule": "string",
      "expected": "valid email address",
      "received": "invalid-email"
    }
  ],
  "requestId": "req_1706356800123_abc123def",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

### Common Error Codes

| Code | HTTP Status | Description |
|------|-------------|-------------|
| `VALIDATION_ERROR` | 400 | Request validation failed |
| `MISSING_API_KEY` | 401 | API key not provided |
| `INVALID_API_KEY` | 401 | API key is invalid |
| `INVALID_CREDENTIALS` | 401 | Wrong email/password |
| `TOKEN_EXPIRED` | 401 | JWT token expired |
| `UNAUTHORIZED` | 401 | Authentication required |
| `INSUFFICIENT_CREDITS` | 402 | Not enough credits |
| `ACCESS_DENIED` | 403 | Access forbidden |
| `ACCOUNT_SUSPENDED` | 403 | Account suspended |
| `NOT_FOUND` | 404 | Resource not found |
| `CONTENT_TOO_LONG` | 413 | Content exceeds limit |
| `RATE_LIMIT_EXCEEDED` | 429 | Too many requests |
| `INTERNAL_ERROR` | 500 | Server error |
| `TRANSLATION_FAILED` | 500 | Translation service error |

---

## Pagination Pattern

All paginated endpoints use this consistent structure:

**Request**:
```http
GET /v1/admin/users?page=2&limit=50&sortBy=created_at&sortOrder=desc
```

**Response**:
```json
{
  "data": [ /* array of items */ ],
  "pagination": {
    "page": 2,
    "limit": 50,
    "total": 1250,
    "totalPages": 25,
    "hasNext": true,
    "hasPrev": true
  }
}
```

---

## Webhook Delivery Format

When a translation job completes, a webhook is sent to the `callbackUrl`:

**Headers**:
```
Content-Type: application/json
X-TPZ-Signature: sha256=abc123def456...
```

**Payload**:
```json
{
  "event": "translation.completed",
  "jobId": "770fa611-g40d-63f6-c938-668877662222",
  "clientJobId": "wp_post_123",
  "status": "completed",
  "translation": "Translated text here...",
  "tokensUsed": 1250,
  "cost": 0.0025,
  "processingTimeMs": 1234,
  "timestamp": "2026-01-27T10:00:05.123Z"
}
```

**Signature Verification** (HMAC-SHA256):
```typescript
import crypto from 'crypto';

function verifyWebhookSignature(
  payload: string,
  signature: string,
  secret: string
): boolean {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  return signature === `sha256=${expectedSignature}`;
}

// Usage
const rawBody = JSON.stringify(req.body);
const signature = req.headers['x-tpz-signature'];
const isValid = verifyWebhookSignature(rawBody, signature, callbackSecret);
```

---

## Implementation Checklist

- [ ] All 57 endpoints documented with complete schemas
- [ ] Authentication methods clearly specified per endpoint
- [ ] Rate limiting strategies documented and implemented
- [ ] Error response formats standardized across all endpoints
- [ ] Pagination pattern consistent across list endpoints
- [ ] Webhook payload format and signature verification documented
- [ ] Admin endpoints secured with Admin JWT
- [ ] PayPal webhook signature verification implemented
- [ ] CSV export format documented for admin endpoints
- [ ] SSE (Server-Sent Events) streaming endpoint documented
- [ ] Zod validation schemas extracted and reusable
- [ ] OpenAPI/Swagger spec generation ready (future task)
- [ ] Postman collection available for testing (future task)
- [ ] WordPress plugin integration examples provided (Appendix C)

---

## Next Steps

1. Generate OpenAPI 3.0 specification from route definitions
2. Create Postman collection for API testing
3. Build SDK/client libraries (JavaScript, PHP)
4. Add GraphQL layer (optional future enhancement)
5. Implement webhook retry dashboard in admin panel
6. Add endpoint deprecation strategy for future changes


---

# Skill: Credit Management System

## Identity
- **Skill ID**: `credit-management`
- **Domain**: User Credits, Transaction Ledger, Balance Tracking
- **Technologies**: Prisma, PostgreSQL, TypeScript
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Credit allocation (subscriptions, manual grants, promotions)
- Credit deduction for translation jobs
- Credit refunds (failed jobs, cancellations)
- Balance queries and credit history
- Token cost calculation
- Credit transaction ledger
- Integration with payment system
- Integration with translation service

**File patterns:**
- `api/src/services/creditService.ts`
- `api/prisma/schema.prisma` (CreditTransaction model)
- `api/src/routes/account.ts` (balance endpoints)
- `api/src/routes/admin/transactions.ts`

## Core Patterns

### 1. Credit as Currency Model

**Key Principle: 1 Token = 1 Credit (1:1 mapping)**

Credits are the billing currency for the translation service. Every Google Gemini API token consumed equals one credit deducted from the user's balance.

```typescript
// Simple token-to-credit conversion
export function calculateCreditCost(tokensUsed: number): number {
  return tokensUsed; // 1:1 mapping
}
```

### 2. Immutable Ledger Pattern

**CRITICAL: Credits use an append-only transaction ledger, NOT a denormalized balance column.**

```typescript
// ❌ WRONG - Denormalized balance (race conditions)
model User {
  id              String @id
  credit_balance  Int    @default(0)  // Don't do this!
}

// ✅ CORRECT - Immutable transaction ledger
model CreditTransaction {
  id                 String   @id @default(uuid())
  user_id            String
  type               CreditTransactionType  // allocation, deduction, refund
  amount             Int                    // Positive for allocation/refund, negative for deduction
  balance_after      Int                    // Computed balance after this transaction
  description        String
  related_job_id     String?
  related_payment_id String?
  created_at         DateTime @default(now())
}
```

### 3. Atomic Balance Computation

```typescript
/**
 * Get user's current credit balance
 *
 * Retrieves the most recent balance_after value from credit transactions
 * If no transactions exist, returns 0
 */
export async function getCurrentBalance(userId: string): Promise<number> {
  try {
    const latestTransaction = await prisma.creditTransaction.findFirst({
      where: { user_id: userId },
      orderBy: { created_at: 'desc' },
      select: { balance_after: true },
    });

    return latestTransaction?.balance_after ?? 0;
  } catch (error) {
    logger.error('Failed to get current balance', { userId, error });
    throw error;
  }
}
```

### 4. Credit Allocation

**Used when:**
- Subscription is activated or renewed
- Manual credit grants by admin
- Promotional credits

```typescript
/**
 * Allocate credits to a user account
 *
 * @param userId - User ID to receive credits
 * @param amount - Number of credits to allocate (must be positive)
 * @param description - Human-readable description
 * @param relatedPaymentId - Optional payment ID for tracking
 * @returns Created credit transaction record
 * @throws Error if amount is not positive
 */
export async function allocateCredits(
  userId: string,
  amount: number,
  description: string,
  relatedPaymentId?: string
): Promise<CreditTransaction> {
  // Validation
  if (amount <= 0) {
    throw new Error('Allocation amount must be positive');
  }

  try {
    // Use Prisma transaction for atomicity
    const result = await prisma.$transaction(async (tx) => {
      // Get current balance
      const currentBalance = await getCurrentBalance(userId);
      const newBalance = currentBalance + amount;

      // Create credit transaction record
      const transaction = await tx.creditTransaction.create({
        data: {
          user_id: userId,
          type: CreditTransactionType.allocation,
          amount,
          balance_after: newBalance,
          description,
          related_payment_id: relatedPaymentId,
        },
      });

      logger.info('Credits allocated', {
        userId,
        amount,
        balanceAfter: newBalance,
        description,
        transactionId: transaction.id,
      });

      // Track metrics
      creditsAllocatedTotal.inc({ tier: 'unknown' }, amount);

      return transaction;
    });

    return result as unknown as CreditTransaction;
  } catch (error) {
    logger.error('Failed to allocate credits', {
      userId,
      amount,
      description,
      error,
    });
    throw error;
  }
}
```

**Example Usage:**

```typescript
// After subscription activation
await allocateCredits(
  userId,
  50000, // 50k credits for starter plan
  'Monthly subscription allocation - Starter Plan',
  paymentId
);

// Manual grant by admin
await allocateCredits(
  userId,
  10000,
  'Promotional credits - New user bonus'
);
```

### 5. Credit Deduction

**Used when:**
- Translation job is completed
- Any credit-consuming operation

```typescript
/**
 * Deduct credits from a user account
 *
 * @param userId - User ID to deduct from
 * @param amount - Number of credits to deduct (must be positive)
 * @param description - Human-readable description
 * @param relatedJobId - Optional job ID for tracking
 * @returns Created credit transaction record
 * @throws Error if insufficient credits or amount is not positive
 */
export async function deductCredits(
  userId: string,
  amount: number,
  description: string,
  relatedJobId?: string
): Promise<CreditTransaction> {
  // Validation
  if (amount <= 0) {
    throw new Error('Deduction amount must be positive');
  }

  try {
    // Use Prisma transaction for atomicity
    const result = await prisma.$transaction(async (tx) => {
      // Get current balance
      const currentBalance = await getCurrentBalance(userId);

      // Check sufficient credits
      if (currentBalance < amount) {
        throw new Error(
          `Insufficient credits. Required: ${amount}, Available: ${currentBalance}`
        );
      }

      const newBalance = currentBalance - amount;

      // Create credit transaction record (negative amount for deduction)
      const transaction = await tx.creditTransaction.create({
        data: {
          user_id: userId,
          type: CreditTransactionType.deduction,
          amount: -amount, // Negative amount for deductions
          balance_after: newBalance,
          description,
          related_job_id: relatedJobId,
        },
      });

      logger.info('Credits deducted', {
        userId,
        amount,
        balanceAfter: newBalance,
        description,
        transactionId: transaction.id,
      });

      // Track metrics
      creditsDeductedTotal.inc({ tier: 'unknown', model: 'unknown' }, amount);

      return transaction;
    });

    return result as unknown as CreditTransaction;
  } catch (error) {
    logger.error('Failed to deduct credits', {
      userId,
      amount,
      description,
      error,
    });
    throw error;
  }
}
```

**Example Usage:**

```typescript
// After translation job completion
const tokensUsed = 1245;
const creditCost = calculateCreditCost(tokensUsed);

await deductCredits(
  userId,
  creditCost,
  `Translation job ${jobId} - ${tokensUsed} tokens used`,
  jobId
);
```

### 6. Credit Refunds

**Used when:**
- Translation job fails
- Job is cancelled
- Manual refund by admin

```typescript
/**
 * Refund credits to a user account
 *
 * @param userId - User ID to refund
 * @param amount - Number of credits to refund (must be positive)
 * @param description - Human-readable description
 * @param relatedJobId - Optional job ID for tracking
 * @returns Created credit transaction record
 * @throws Error if amount is not positive
 */
export async function refundCredits(
  userId: string,
  amount: number,
  description: string,
  relatedJobId?: string
): Promise<CreditTransaction> {
  // Validation
  if (amount <= 0) {
    throw new Error('Refund amount must be positive');
  }

  try {
    // Use Prisma transaction for atomicity
    const result = await prisma.$transaction(async (tx) => {
      // Get current balance
      const currentBalance = await getCurrentBalance(userId);
      const newBalance = currentBalance + amount;

      // Create credit transaction record
      const transaction = await tx.creditTransaction.create({
        data: {
          user_id: userId,
          type: CreditTransactionType.refund,
          amount,
          balance_after: newBalance,
          description,
          related_job_id: relatedJobId,
        },
      });

      logger.info('Credits refunded', {
        userId,
        amount,
        balanceAfter: newBalance,
        description,
        transactionId: transaction.id,
      });

      return transaction;
    });

    return result as unknown as CreditTransaction;
  } catch (error) {
    logger.error('Failed to refund credits', {
      userId,
      amount,
      description,
      error,
    });
    throw error;
  }
}
```

**Example Usage:**

```typescript
// Refund after job failure
await refundCredits(
  userId,
  creditCost,
  `Refund for failed translation job ${jobId}`,
  jobId
);

// Manual refund by admin
await refundCredits(
  userId,
  5000,
  'Manual refund - Customer support request #12345'
);
```

### 7. Credit Sufficiency Check

```typescript
/**
 * Check if user has sufficient credits for an operation
 *
 * @param userId - User ID to check
 * @param requiredAmount - Number of credits required
 * @returns True if user has enough credits, false otherwise
 */
export async function hasSufficientCredits(
  userId: string,
  requiredAmount: number
): Promise<boolean> {
  try {
    const currentBalance = await getCurrentBalance(userId);
    return currentBalance >= requiredAmount;
  } catch (error) {
    logger.error('Failed to check credit sufficiency', {
      userId,
      requiredAmount,
      error,
    });
    throw error;
  }
}
```

**Example Usage (Pre-flight Check):**

```typescript
// Before submitting translation job
router.post('/jobs', requireApiKey, async (req: AuthRequest, res) => {
  const userId = req.user!.userId;
  const { content, model } = req.body;

  // Estimate tokens
  const estimatedTokens = estimateTokens(content);
  const estimatedCost = calculateCreditCost(estimatedTokens);

  // Check if user has sufficient credits
  const hasSufficientCredits = await creditService.hasSufficientCredits(
    userId,
    estimatedCost
  );

  if (!hasSufficientCredits) {
    const currentBalance = await creditService.getCurrentBalance(userId);
    return res.status(402).json({
      error: 'Insufficient credits',
      details: {
        required: estimatedCost,
        available: currentBalance,
        deficit: estimatedCost - currentBalance,
      },
    });
  }

  // Continue with job creation...
});
```

### 8. Credit History (Paginated)

```typescript
/**
 * Get credit transaction history for a user
 *
 * @param userId - User ID to query
 * @param limit - Number of records to return (default: 50)
 * @param offset - Number of records to skip (default: 0)
 * @returns Array of credit transactions ordered by created_at DESC
 */
export async function getCreditHistory(
  userId: string,
  limit: number = 50,
  offset: number = 0
): Promise<CreditTransaction[]> {
  try {
    const transactions = await prisma.creditTransaction.findMany({
      where: { user_id: userId },
      orderBy: { created_at: 'desc' },
      take: limit,
      skip: offset,
    });

    return transactions as unknown as CreditTransaction[];
  } catch (error) {
    logger.error('Failed to get credit history', { userId, error });
    throw error;
  }
}
```

**Example Usage (Account Page):**

```typescript
// GET /account/credits/history
router.get('/account/credits/history', requireApiKey, async (req: AuthRequest, res) => {
  const userId = req.user!.userId;
  const limit = parseInt(req.query.limit as string) || 50;
  const offset = parseInt(req.query.offset as string) || 0;

  const history = await creditService.getCreditHistory(userId, limit, offset);
  const currentBalance = await creditService.getCurrentBalance(userId);

  res.json({
    current_balance: currentBalance,
    transactions: history,
    pagination: {
      limit,
      offset,
      has_more: history.length === limit,
    },
  });
});
```

## Database Model

### CreditTransaction Table

```prisma
model CreditTransaction {
  id                 String                 @id @default(uuid()) @db.Uuid
  user_id            String                 @db.Uuid
  type               CreditTransactionType  // allocation, deduction, refund
  amount             Int                    // Can be negative for deductions
  balance_after      Int                    // Computed balance after this transaction
  description        String
  related_job_id     String?                @db.Uuid
  related_payment_id String?                @db.Uuid
  created_at         DateTime               @default(now())

  // Relations
  user    User            @relation(fields: [user_id], references: [id], onDelete: Cascade)
  job     TranslationJob? @relation(fields: [related_job_id], references: [id], onDelete: SetNull)
  payment Payment?        @relation(fields: [related_payment_id], references: [id], onDelete: SetNull)

  @@index([user_id])
  @@index([created_at])
  @@index([related_job_id])
  @@index([related_payment_id])
  @@map("credit_transactions")
}

enum CreditTransactionType {
  allocation
  deduction
  refund
}
```

### Key Fields

| Field | Type | Description |
|-------|------|-------------|
| `id` | UUID | Primary key |
| `user_id` | UUID | User who owns the transaction |
| `type` | Enum | `allocation`, `deduction`, or `refund` |
| `amount` | Int | Positive for allocation/refund, negative for deduction |
| `balance_after` | Int | User's balance after this transaction |
| `description` | String | Human-readable description |
| `related_job_id` | UUID? | Link to translation job (deductions/refunds) |
| `related_payment_id` | UUID? | Link to payment (allocations) |
| `created_at` | DateTime | Transaction timestamp |

### Indexes

```sql
-- User transactions query
CREATE INDEX idx_user_id ON credit_transactions(user_id);

-- Latest balance query
CREATE INDEX idx_created_at ON credit_transactions(created_at);

-- Job-related transactions
CREATE INDEX idx_related_job_id ON credit_transactions(related_job_id);

-- Payment-related transactions
CREATE INDEX idx_related_payment_id ON credit_transactions(related_payment_id);
```

## Integration Points

### 1. Translation Service Integration

```typescript
// api/src/services/translationService.ts
export async function processTranslationJob(jobId: string): Promise<void> {
  const job = await prisma.translationJob.findUnique({ where: { id: jobId } });

  if (!job) throw new Error('Job not found');

  try {
    // Call Gemini API
    const result = await geminiClient.translate({
      content: job.content,
      sourceLang: job.source_lang,
      targetLang: job.target_lang,
      model: job.model,
      tone: job.tone,
    });

    // Deduct credits based on actual token usage
    await deductCredits(
      job.user_id,
      result.tokensUsed,
      `Translation job ${jobId} - ${result.tokensUsed} tokens`,
      jobId
    );

    // Update job status
    await prisma.translationJob.update({
      where: { id: jobId },
      data: {
        status: 'completed',
        translation: result.translation,
        tokens_used: result.tokensUsed,
        cost: calculateCreditCost(result.tokensUsed),
      },
    });

  } catch (error) {
    logger.error('Translation job failed', { jobId, error });

    // Update job status to failed
    await prisma.translationJob.update({
      where: { id: jobId },
      data: {
        status: 'failed',
        error_message: error.message,
      },
    });

    // Credits are NOT deducted on failure (no need to refund)
  }
}
```

### 2. PayPal Payment Integration

```typescript
// api/src/services/paypalService.ts
export async function handleSubscriptionActivated(event: any): Promise<void> {
  const subscriptionId = event.resource.id;
  const planId = event.resource.plan_id;

  // Find user by subscription
  const subscription = await prisma.subscription.findUnique({
    where: { paypal_subscription_id: subscriptionId },
  });

  if (!subscription) {
    logger.error('Subscription not found', { subscriptionId });
    return;
  }

  // Determine credit allocation based on plan
  const creditAllocation = getCreditAllocationForPlan(subscription.plan_tier);

  // Allocate credits to user
  await allocateCredits(
    subscription.user_id,
    creditAllocation,
    `Monthly subscription allocation - ${subscription.plan_tier} plan`,
    undefined // No payment ID yet (handled by PAYMENT.SALE.COMPLETED)
  );

  logger.info('Subscription activated, credits allocated', {
    userId: subscription.user_id,
    subscriptionId,
    creditsAllocated: creditAllocation,
  });
}

function getCreditAllocationForPlan(tier: PlanTier): number {
  const allocations = {
    starter: 50000,       // 50k credits/month
    professional: 150000, // 150k credits/month
    enterprise: 500000,   // 500k credits/month
  };

  return allocations[tier];
}
```

### 3. Account Balance Endpoint

```typescript
// api/src/routes/account.ts
import { Router } from 'express';
import { requireApiKey } from '../middleware/auth';
import * as creditService from '../services/creditService';
import type { AuthRequest } from '../types';

const router = Router();

/**
 * GET /account/balance
 *
 * Returns current credit balance for authenticated user
 */
router.get('/account/balance', requireApiKey, async (req: AuthRequest, res) => {
  try {
    const userId = req.user!.userId;
    const balance = await creditService.getCurrentBalance(userId);

    res.json({
      balance,
      currency: 'credits',
      timestamp: new Date().toISOString(),
    });
  } catch (error) {
    res.status(500).json({ error: 'Failed to retrieve balance' });
  }
});

/**
 * GET /account/credits/history
 *
 * Returns paginated credit transaction history
 */
router.get('/account/credits/history', requireApiKey, async (req: AuthRequest, res) => {
  try {
    const userId = req.user!.userId;
    const limit = Math.min(parseInt(req.query.limit as string) || 50, 100);
    const offset = parseInt(req.query.offset as string) || 0;

    const history = await creditService.getCreditHistory(userId, limit, offset);
    const balance = await creditService.getCurrentBalance(userId);

    res.json({
      current_balance: balance,
      transactions: history.map(tx => ({
        id: tx.id,
        type: tx.type,
        amount: tx.amount,
        balance_after: tx.balance_after,
        description: tx.description,
        created_at: tx.created_at,
        related_job_id: tx.related_job_id,
        related_payment_id: tx.related_payment_id,
      })),
      pagination: {
        limit,
        offset,
        has_more: history.length === limit,
      },
    });
  } catch (error) {
    res.status(500).json({ error: 'Failed to retrieve credit history' });
  }
});

export default router;
```

## Error Handling

### 1. Insufficient Credits

```typescript
// Clear error message with actionable details
if (currentBalance < amount) {
  throw new Error(
    `Insufficient credits. Required: ${amount}, Available: ${currentBalance}`
  );
}
```

**HTTP Response (402 Payment Required):**

```json
{
  "error": "Insufficient credits",
  "details": {
    "required": 1500,
    "available": 1200,
    "deficit": 300
  },
  "links": {
    "purchase_credits": "https://api.press.zone/pricing",
    "account_balance": "https://api.press.zone/account/balance"
  }
}
```

### 2. Invalid Amount Validation

```typescript
// All credit operations validate positive amounts
if (amount <= 0) {
  throw new Error('Amount must be positive');
}
```

### 3. Race Condition Prevention

```typescript
// Use Prisma $transaction for atomicity
const result = await prisma.$transaction(async (tx) => {
  // 1. Get current balance
  const currentBalance = await getCurrentBalance(userId);

  // 2. Check sufficiency (for deductions)
  if (currentBalance < amount) {
    throw new Error('Insufficient credits');
  }

  // 3. Create transaction record
  const transaction = await tx.creditTransaction.create({...});

  return transaction;
});
```

**CRITICAL: Do NOT use READ COMMITTED isolation level for credit operations. Use default SERIALIZABLE.**

### 4. Negative Balance Prevention

```typescript
// Deduction operation ALWAYS checks balance first
if (currentBalance < amount) {
  throw new Error(`Insufficient credits. Required: ${amount}, Available: ${currentBalance}`);
}
```

**The database does NOT enforce balance >= 0. Application logic must prevent negative balances.**

## Edge Cases

### 1. Concurrent Deductions

**Problem:** Two translation jobs submit simultaneously, both checking balance before deduction.

**Solution:** Use Prisma transactions for atomicity.

```typescript
// Each deduction is wrapped in a transaction
const result = await prisma.$transaction(async (tx) => {
  const currentBalance = await getCurrentBalance(userId);

  if (currentBalance < amount) {
    throw new Error('Insufficient credits');
  }

  // Create transaction record (atomic with balance check)
  return await tx.creditTransaction.create({...});
});
```

### 2. Zero Balance User

**Scenario:** User has 0 credits and attempts translation.

**Handling:**

```typescript
// Pre-flight check in job submission endpoint
const hasSufficientCredits = await creditService.hasSufficientCredits(userId, estimatedCost);

if (!hasSufficientCredits) {
  return res.status(402).json({
    error: 'Insufficient credits',
    details: {
      required: estimatedCost,
      available: 0,
      deficit: estimatedCost,
    },
    message: 'Please purchase credits or upgrade your subscription',
  });
}
```

### 3. First Transaction (No History)

**Scenario:** New user has no credit transactions yet.

**Handling:**

```typescript
export async function getCurrentBalance(userId: string): Promise<number> {
  const latestTransaction = await prisma.creditTransaction.findFirst({
    where: { user_id: userId },
    orderBy: { created_at: 'desc' },
    select: { balance_after: true },
  });

  // Return 0 if no transactions exist (null coalescing)
  return latestTransaction?.balance_after ?? 0;
}
```

### 4. Refund After Partial Consumption

**Scenario:** Job fails midway after consuming some tokens.

**Handling:**

```typescript
// In translationService error handler
catch (error) {
  // Check if any tokens were consumed before failure
  const tokensConsumed = error.tokensConsumed || 0;

  if (tokensConsumed > 0) {
    // Deduct only consumed tokens
    await deductCredits(
      userId,
      tokensConsumed,
      `Partial translation job ${jobId} - ${tokensConsumed} tokens (failed)`,
      jobId
    );
  }

  // No refund needed if no deduction occurred
}
```

### 5. Credit Type Tracking

**Transaction Types:**

| Type | Amount Sign | Use Case |
|------|-------------|----------|
| `allocation` | Positive | Subscription renewal, manual grant, promo |
| `deduction` | Negative | Translation job completion |
| `refund` | Positive | Job failure, cancellation, manual refund |

**Query by Type:**

```typescript
// Get all refunds for a user
const refunds = await prisma.creditTransaction.findMany({
  where: {
    user_id: userId,
    type: CreditTransactionType.refund,
  },
  orderBy: { created_at: 'desc' },
});
```

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Denormalized `credit_balance` column | Use immutable ledger with computed balance |
| Direct balance UPDATE queries | Use transaction creation with `balance_after` |
| Skipping Prisma transactions | Always wrap operations in `prisma.$transaction()` |
| Allowing negative amounts | Validate `amount > 0` in all functions |
| No sufficiency check before deduction | Use `hasSufficientCredits()` pre-flight check |
| Hardcoded credit allocations | Use `getCreditAllocationForPlan()` lookup |
| No transaction logging | Log all operations with `logger.info()` |
| Missing Prometheus metrics | Track allocations/deductions with `creditsAllocatedTotal.inc()` |
| No related entity tracking | Always link to `related_job_id` or `related_payment_id` |
| Using `User.credit_balance` | Compute balance from `CreditTransaction` table |

## Integration with Other Skills

**Often combined with:**
- `translation-service-integration` - Deduct credits after translation
- `paypal-payment-integration` - Allocate credits after payment
- `api-endpoint-creation` - Account balance/history endpoints
- `queue-management` - Async credit allocation via Bull queue

**Depends on:**
- `database-schema-design` - CreditTransaction table schema
- `authentication-security` - User identification for transactions
- `error-handling-logging` - Logging credit operations

## Testing Strategy

### 1. Unit Tests

```typescript
// __tests__/unit/services/creditService.test.ts
describe('creditService', () => {
  describe('allocateCredits', () => {
    it('should increase balance correctly', async () => {
      const userId = 'test-user-123';

      await allocateCredits(userId, 1000, 'Test allocation');
      const balance = await getCurrentBalance(userId);

      expect(balance).toBe(1000);
    });

    it('should reject negative amounts', async () => {
      await expect(
        allocateCredits('user-123', -100, 'Invalid')
      ).rejects.toThrow('Allocation amount must be positive');
    });
  });

  describe('deductCredits', () => {
    it('should reject insufficient credits', async () => {
      const userId = 'test-user-456';

      await allocateCredits(userId, 100, 'Setup');

      await expect(
        deductCredits(userId, 200, 'Over budget')
      ).rejects.toThrow('Insufficient credits');
    });
  });

  describe('concurrent deductions', () => {
    it('should handle race conditions', async () => {
      const userId = 'test-user-789';

      await allocateCredits(userId, 1000, 'Setup');

      // Simulate two simultaneous deductions
      await Promise.all([
        deductCredits(userId, 600, 'Job 1'),
        deductCredits(userId, 600, 'Job 2'),
      ]).catch(err => {
        expect(err.message).toContain('Insufficient credits');
      });

      const balance = await getCurrentBalance(userId);
      expect(balance).toBeGreaterThanOrEqual(0);
    });
  });
});
```

### 2. Integration Tests

```typescript
// __tests__/integration/routes/account.test.ts
describe('GET /account/balance', () => {
  it('should return current balance', async () => {
    const apiKey = 'sk_test_123';

    const response = await request(app)
      .get('/account/balance')
      .set('Authorization', `Bearer ${apiKey}`);

    expect(response.status).toBe(200);
    expect(response.body).toMatchObject({
      balance: expect.any(Number),
      currency: 'credits',
      timestamp: expect.any(String),
    });
  });
});
```

## Quick Reference

### Function Signatures

```typescript
// Balance queries
getCurrentBalance(userId: string): Promise<number>
hasSufficientCredits(userId: string, requiredAmount: number): Promise<boolean>

// Credit operations
allocateCredits(userId: string, amount: number, description: string, relatedPaymentId?: string): Promise<CreditTransaction>
deductCredits(userId: string, amount: number, description: string, relatedJobId?: string): Promise<CreditTransaction>
refundCredits(userId: string, amount: number, description: string, relatedJobId?: string): Promise<CreditTransaction>

// History
getCreditHistory(userId: string, limit: number, offset: number): Promise<CreditTransaction[]>

// Cost calculation
calculateCreditCost(tokensUsed: number): number
```

### Typical Credit Flow

```
1. User subscribes → PayPal webhook → allocateCredits()
2. User submits job → hasSufficientCredits() check
3. Job processing → Gemini API consumes tokens
4. Job completes → deductCredits() with actual token count
5. Job fails → No deduction (or partial deduction if tokens were used)
```

### Common Queries

```typescript
// Get user's current balance
const balance = await getCurrentBalance(userId);

// Check if user can afford a translation
const canAfford = await hasSufficientCredits(userId, estimatedCost);

// Get recent transactions
const recent = await getCreditHistory(userId, 10, 0);

// Calculate cost for job
const cost = calculateCreditCost(tokensUsed);
```

## Validation Checklist

Before completing credit management implementation:

- [ ] `getCurrentBalance()` returns 0 for new users
- [ ] `allocateCredits()` validates positive amounts
- [ ] `deductCredits()` checks sufficient balance before deduction
- [ ] `refundCredits()` validates positive amounts
- [ ] All operations wrapped in Prisma transactions
- [ ] Concurrent deductions cannot result in negative balance
- [ ] Transaction type correctly set (allocation, deduction, refund)
- [ ] `balance_after` computed correctly for each transaction
- [ ] Related entity IDs linked (`related_job_id`, `related_payment_id`)
- [ ] Descriptive messages for all transactions
- [ ] Prometheus metrics tracked (`creditsAllocatedTotal`, `creditsDeductedTotal`)
- [ ] Structured logging for all operations
- [ ] 402 Payment Required returned for insufficient credits
- [ ] Credit history pagination implemented
- [ ] Account balance endpoint secured with API key
- [ ] Unit tests cover edge cases (zero balance, concurrent ops)
- [ ] Integration tests verify end-to-end flows

---

# Skill 18: Translation Service Orchestration

## Overview

**`translationService`** is the core orchestrator that coordinates the entire translation workflow in the Press.Zone Backend. It acts as the primary interface between client applications and the translation infrastructure, managing:

- **Credit validation and deduction** via `creditService`
- **Token estimation** via `TokenEstimator`
- **Translation execution** via `geminiClient` (Google Gemini API)
- **Async job queueing** via Bull queue (`translationQueue`)
- **Webhook callbacks** via `webhookService`
- **Job lifecycle management** (pending → processing → completed/failed)
- **Deduplication** via content hashing
- **ML feedback** via `accuracyTracker`

The service provides both **synchronous** and **asynchronous** translation modes, ensuring optimal performance for different content sizes and use cases.

**Source Files:**
- `/api/src/services/translationService.ts` - Main orchestrator class
- `/api/src/worker.ts` - Async job processor (Bull queue worker)

---

## Architecture

### Translation Modes

| Mode | Content Size | Processing | Response Time | Use Case |
|------|--------------|------------|---------------|----------|
| **Synchronous** | ≤ 5,000 chars | Immediate | ~2-5 seconds | Short texts, real-time UI |
| **Asynchronous** | ≤ 50,000 chars | Queued (Bull) | Minutes | Long articles, batch jobs |

### Data Flow

#### Synchronous Translation Flow

```
┌──────────────┐
│ Client (WP)  │
└──────┬───────┘
       │ POST /v1/translate (sync)
       │
┌──────▼────────────────────────────────────────────────────────────────┐
│ translationService.translateSync()                                     │
│                                                                        │
│  1. Validate content length (≤ 5,000 chars)                           │
│  2. Estimate tokens & cost (TokenEstimator)                           │
│  3. Check credits (creditService.hasSufficientCredits)                │
│  4. Check for duplicate (content_hash + 24h cache)                    │
│  5. Create TranslationJob record (status: processing)                 │
│  6. Call Gemini API (geminiClient.translate)                          │
│  7. Deduct credits (creditService.deductCredits)                      │
│  8. Update job (status: completed, translation, tokens_used, cost)    │
│  9. Track metrics (Prometheus)                                        │
│ 10. Record ML feedback (accuracyTracker)                              │
│                                                                        │
└───────────────────────────────┬────────────────────────────────────────┘
                                │
                    ┌───────────▼──────────┐
                    │ Return translation   │
                    │ + tokensUsed         │
                    │ + cost               │
                    │ + creditBalance      │
                    └──────────────────────┘
```

#### Asynchronous Translation Flow

```
┌──────────────┐
│ Client (WP)  │
└──────┬───────┘
       │ POST /v1/jobs/submit (async)
       │
┌──────▼────────────────────────────────────────────────────────────────┐
│ translationService.translateAsync()                                    │
│                                                                        │
│  1. Validate content length (≤ 50,000 chars)                          │
│  2. Estimate tokens & cost (TokenEstimator)                           │
│  3. Check credits (creditService.hasSufficientCredits)                │
│  4. Check for duplicate pending job                                   │
│  5. Create TranslationJob record (status: pending)                    │
│  6. Add job to Bull queue (translationQueue)                          │
│  7. Return jobId immediately                                          │
│                                                                        │
└───────────────────────────────┬────────────────────────────────────────┘
                                │
                    ┌───────────▼──────────┐
                    │ Return jobId         │
                    │ + status: pending    │
                    └──────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│ worker.ts (Bull Queue Processor)                                        │
│                                                                         │
│  1. Pick job from queue (FIFO)                                         │
│  2. Update job (status: processing)                                    │
│  3. Call Gemini API (geminiClient.translate)                           │
│  4. Atomic transaction:                                                │
│     - Deduct credits (creditService)                                   │
│     - Update job (status: completed, translation, tokens, cost)        │
│  5. Track metrics (Prometheus)                                         │
│  6. Deliver webhook (if callbackUrl provided)                          │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
                                │
                    ┌───────────▼──────────────────┐
                    │ Webhook: translation.completed│
                    │ or poll GET /v1/jobs/:jobId   │
                    └───────────────────────────────┘
```

---

## Core Operations

### 1. Synchronous Translation

#### `translateSync(userId: string, request: TranslationRequest): Promise<TranslationResponse>`

Processes a translation request **immediately** and returns the result in the HTTP response.

**Validation:**
- Content length ≤ 5,000 characters (configurable via `config.maxSyncChars`)
- Content not empty
- Sufficient credits available

**Deduplication:**
- Generates content hash from `(content + sourceLang + targetLang + model)`
- Checks for completed jobs with same hash in last 24 hours
- Returns cached translation if found (no credit deduction)

**Credit Flow:**
1. **Pre-check:** `hasSufficientCredits(userId, estimatedTokens)`
2. **Estimation:** `estimateTokens(content, sourceLang, targetLang)`
3. **Execution:** Gemini API call
4. **Deduction:** `deductCredits(userId, actualTokens, description, jobId)`

**Error Handling:**
- **Insufficient credits:** Throws error with current balance
- **API failure:** Updates job to `failed`, tracks metrics, throws error
- **No refund on failure** (unless tokens were actually consumed)

**Response:**
```typescript
{
  jobId: "uuid",
  status: "completed",
  translation: "Translated text...",
  tokensUsed: 234,
  cost: 0.234,
  processingTimeMs: 3421,
  creditBalance: 9766  // Updated balance after deduction
}
```

**Key Implementation Details:**

```typescript
// Content length validation
if (request.content.length > config.maxSyncChars) {
  throw new Error(
    `Content too long for synchronous translation. Maximum ${config.maxSyncChars} characters. ` +
    `Use async translation for larger content (up to ${config.maxAsyncChars} characters).`
  );
}

// Estimate tokens and cost
const estimatedTokens = estimateTokens(request.content, request.sourceLang, request.targetLang);
const estimatedCost = calculateCost(estimatedTokens, request.model);

// Check for sufficient credits
const sufficient = await hasSufficientCredits(userId, estimatedTokens);
const currentBalance = await getCurrentBalance(userId);

if (!sufficient) {
  throw new Error(
    `Insufficient credits. You need ${estimatedTokens} credits but only have ${currentBalance}. ` +
    `Please upgrade your plan or wait for your monthly allocation.`
  );
}

// Generate content hash for deduplication
const contentHash = generateContentHash(
  request.content,
  request.sourceLang,
  request.targetLang,
  request.model
);

// Check for duplicate job in last 24 hours
const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000);
const duplicateJob = await prisma.translationJob.findFirst({
  where: {
    user_id: userId,
    content_hash: contentHash,
    status: TranslationJobStatus.completed,
    created_at: { gte: oneDayAgo },
  },
});

if (duplicateJob && duplicateJob.translation) {
  return {
    jobId: duplicateJob.id,
    status: TranslationStatus.COMPLETED,
    translation: duplicateJob.translation,
    tokensUsed: duplicateJob.tokens_used,
    cost: parseFloat(duplicateJob.cost.toString()),
    processingTimeMs: duplicateJob.processing_time_ms || 0,
    creditBalance: currentBalance,
  };
}

// Create translation job record
const job = await prisma.translationJob.create({
  data: {
    user_id: userId,
    client_job_id: request.clientJobId,
    status: TranslationJobStatus.processing,
    source_lang: request.sourceLang,
    target_lang: request.targetLang,
    model: request.model === Model.MODEL_4B ? 'MODEL_4B' : 'MODEL_27B',
    tone: request.tone || Tone.NEUTRAL,
    content: request.content,
    content_hash: contentHash,
  },
});

// Call Google Gemini API
const geminiResponse = await geminiClient.translate(
  request.content,
  request.sourceLang,
  request.targetLang,
  request.model,
  request.tone || Tone.NEUTRAL
);

const processingTime = Date.now() - startTime;
const actualTokens = geminiResponse.tokens_used;
const actualCost = calculateCost(actualTokens, request.model);

// Deduct credits
const transaction = await deductCredits(
  userId,
  actualTokens,
  `Translation: ${request.sourceLang} → ${request.targetLang} (${request.model})`,
  job.id
);
const newBalance = transaction.balanceAfter;

// Update job to completed
await prisma.translationJob.update({
  where: { id: job.id },
  data: {
    status: TranslationJobStatus.completed,
    translation: geminiResponse.translation,
    tokens_used: actualTokens,
    cost: actualCost,
    processing_time_ms: processingTime,
    completed_at: new Date(),
  },
});

// Track metrics
trackTranslationJob(request.model, 'completed', 'sync', processingTime);
trackTokensProcessed(request.model, actualTokens);

// Record translation for ML learning (best-effort)
try {
  const { TokenEstimator } = await import('./TokenEstimator');
  const { accuracyTracker } = await import('./AccuracyTracker');
  
  const tokenEstimatorInstance = new TokenEstimator();
  const complexity = tokenEstimatorInstance.analyzeHTMLComplexity(request.content);
  
  await accuracyTracker.recordUsage(
    request.targetLang,
    estimatedTokens,
    actualTokens,
    complexity.level
  );
} catch (mlError) {
  // Silent failure - don't block translation success
  logger.warn('Failed to record ML data', { mlError });
}
```

---

### 2. Asynchronous Translation

#### `translateAsync(userId: string, request: JobSubmitRequest): Promise<JobSubmitResponse>`

Submits a translation job to the **Bull queue** for background processing. Returns immediately with a job ID.

**Validation:**
- Content length ≤ 50,000 characters (configurable via `config.maxAsyncChars`)
- Content not empty
- Sufficient credits available (pre-check only; actual deduction happens in worker)

**Deduplication:**
- Checks for duplicate **pending or processing** jobs with same content hash
- Returns existing job ID if found

**Queue Job Data:**
```typescript
{
  jobId: "uuid",
  clientJobId: "optional-client-reference",
  userId: "uuid",
  content: "Text to translate...",
  sourceLang: "en",
  targetLang: "es",
  model: "MODEL_4B",
  tone: "neutral",
  callbackUrl: "https://example.com/webhook",
  callbackSecret: "shared-secret"
}
```

**Response:**
```typescript
{
  jobId: "uuid",
  status: "pending",
  clientJobId: "optional-client-reference"
}
```

**Key Implementation Details:**

```typescript
// Validate content length
if (request.content.length > config.maxAsyncChars) {
  throw new Error(
    `Content too long. Maximum ${config.maxAsyncChars} characters allowed for async translation.`
  );
}

// Estimate tokens and cost
const estimatedTokens = estimateTokens(request.content, request.sourceLang, request.targetLang);
const estimatedCost = calculateCost(estimatedTokens, request.model);

// Check for sufficient credits
const sufficient = await hasSufficientCredits(userId, estimatedTokens);
const currentBalance = await getCurrentBalance(userId);

if (!sufficient) {
  throw new Error(
    `Insufficient credits. You need ${estimatedTokens} credits but only have ${currentBalance}. ` +
    `Please upgrade your plan or wait for your monthly allocation.`
  );
}

// Generate content hash for deduplication
const contentHash = generateContentHash(
  request.content,
  request.sourceLang,
  request.targetLang,
  request.model
);

// Check for duplicate pending job
const duplicatePendingJob = await prisma.translationJob.findFirst({
  where: {
    user_id: userId,
    content_hash: contentHash,
    status: {
      in: [TranslationJobStatus.pending, TranslationJobStatus.processing],
    },
  },
});

if (duplicatePendingJob) {
  return {
    jobId: duplicatePendingJob.id,
    status: duplicatePendingJob.status as TranslationStatus,
    clientJobId: request.clientJobId,
  };
}

// Create translation job record
const job = await prisma.translationJob.create({
  data: {
    user_id: userId,
    client_job_id: request.clientJobId,
    status: TranslationJobStatus.pending,
    source_lang: request.sourceLang,
    target_lang: request.targetLang,
    model: request.model === Model.MODEL_4B ? 'MODEL_4B' : 'MODEL_27B',
    tone: request.tone || Tone.NEUTRAL,
    content: request.content,
    content_hash: contentHash,
    callback_url: request.callbackUrl,
    callback_secret: request.callbackSecret,
  },
});

// Queue job in Bull for async processing
await translationQueue.add('translate', {
  jobId: job.id,
  clientJobId: request.clientJobId,
  userId,
  content: request.content,
  sourceLang: request.sourceLang,
  targetLang: request.targetLang,
  model: request.model,
  tone: request.tone || Tone.NEUTRAL,
  callbackUrl: request.callbackUrl,
  callbackSecret: request.callbackSecret,
});
```

---

### 3. Async Job Processor (Worker)

**File:** `/api/src/worker.ts`

The worker process consumes jobs from the Bull queue and processes them in the background.

**Processing Steps:**

1. **Update Status:** Set job to `processing`
2. **Translate:** Call `geminiClient.translate()`
3. **Atomic Transaction:**
   - Get current user balance
   - Create credit deduction transaction
   - Update user record
   - Update job with results (status: `completed`, translation, tokens, cost)
4. **Track Metrics:** Prometheus counters
5. **Deliver Webhook:** If `callbackUrl` provided (best-effort)

**Error Handling:**
- **Translation fails:** Update job to `failed`, track metrics, deliver failure webhook
- **Webhook fails:** Log error, but don't fail the job
- **Concurrent processing:** Bull ensures job uniqueness

**Key Implementation:**

```typescript
translationQueue.process(async (job) => {
  const startTime = Date.now();
  const data: TranslationJobData = job.data;

  try {
    // Update job status to processing
    await prisma.translationJob.update({
      where: { id: data.jobId },
      data: { status: 'processing' as TranslationStatus },
    });

    // Call Google Gemini API
    const result = await geminiClient.translate(
      data.content,
      data.sourceLang,
      data.targetLang,
      data.model,
      data.tone as any
    );

    const processingTime = Date.now() - startTime;
    const cost = calculateCost(result.tokens_used, data.model);

    // Atomic transaction: deduct credits + update job
    await prisma.$transaction(async (tx) => {
      // Get current user balance
      const latestTransaction = await tx.creditTransaction.findFirst({
        where: { user_id: data.userId },
        orderBy: { created_at: 'desc' },
        select: { balance_after: true },
      });

      const currentBalance = latestTransaction?.balance_after ?? 0;
      const newBalance = currentBalance - result.tokens_used;

      // Create credit transaction record
      await tx.creditTransaction.create({
        data: {
          user_id: data.userId,
          type: 'deduction',
          amount: -result.tokens_used,
          balance_after: newBalance,
          description: `Translation job ${data.jobId}: ${data.sourceLang} → ${data.targetLang}`,
          related_job_id: data.jobId,
        },
      });

      // Update user's updated_at timestamp
      await tx.user.update({
        where: { id: data.userId },
        data: { updated_at: new Date() },
      });

      // Update job with results
      await tx.translationJob.update({
        where: { id: data.jobId },
        data: {
          status: 'completed' as TranslationStatus,
          translation: result.translation,
          tokens_used: result.tokens_used,
          cost,
          processing_time_ms: processingTime,
          completed_at: new Date(),
        },
      });
    });

    // Track metrics
    trackTranslationJob(data.model, 'completed', 'async', processingTime);
    trackTokensProcessed(data.model, result.tokens_used);

    // Deliver webhook if callback URL provided
    if (data.callbackUrl && data.callbackSecret) {
      try {
        await deliverWebhook(
          data.jobId,
          {
            event: 'translation.completed',
            jobId: data.jobId,
            clientJobId: data.clientJobId,
            status: 'completed' as TranslationStatus,
            translation: result.translation,
            tokensUsed: result.tokens_used,
            cost,
            processingTimeMs: processingTime,
            timestamp: new Date().toISOString(),
          },
          data.callbackUrl,
          data.callbackSecret
        );
      } catch (webhookError) {
        logger.error('Webhook delivery failed', { jobId: data.jobId, error: webhookError });
        // Don't fail the job if webhook fails
      }
    }

    return result;
  } catch (error: any) {
    const processingTime = Date.now() - startTime;

    // Update job status to failed
    await prisma.translationJob.update({
      where: { id: data.jobId },
      data: {
        status: 'failed' as TranslationStatus,
        error_message: error.message,
        processing_time_ms: processingTime,
        completed_at: new Date(),
      },
    });

    // Track metrics
    trackTranslationJob(data.model, 'failed', 'async', processingTime);

    // Deliver failure webhook
    if (data.callbackUrl && data.callbackSecret) {
      try {
        await deliverWebhook(
          data.jobId,
          {
            event: 'translation.failed',
            jobId: data.jobId,
            clientJobId: data.clientJobId,
            status: 'failed' as TranslationStatus,
            errorMessage: error.message,
            processingTimeMs: processingTime,
            timestamp: new Date().toISOString(),
          },
          data.callbackUrl,
          data.callbackSecret
        );
      } catch (webhookError) {
        logger.error('Webhook delivery failed', { jobId: data.jobId, error: webhookError });
      }
    }

    throw error;
  }
});

// Queue event handlers
translationQueue.on('completed', (job) => {
  logger.info('Job completed', { jobId: job.id });
});

translationQueue.on('failed', (job, error) => {
  logger.error('Job failed', { jobId: job?.id, error: error.message });
});

translationQueue.on('error', (error) => {
  logger.error('Queue error', { error: error.message });
});
```

---

### 4. Job Status Query

#### `getJobStatus(userId: string, jobId: string): Promise<JobStatusResponse>`

Retrieves the current status and results of a translation job.

**Access Control:**
- Verifies job belongs to requesting user
- Returns 403 if unauthorized

**Response:**
```typescript
{
  jobId: "uuid",
  clientJobId: "optional",
  status: "completed" | "pending" | "processing" | "failed" | "cancelled",
  sourceLang: "en",
  targetLang: "es",
  model: "MODEL_4B",
  tone: "neutral",
  translation: "Translated text..." (if completed),
  tokensUsed: 234 (if completed),
  cost: 0.234 (if completed),
  errorMessage: "Error details" (if failed),
  processingTimeMs: 3421,
  createdAt: "2024-01-15T10:30:00Z",
  updatedAt: "2024-01-15T10:30:03Z",
  completedAt: "2024-01-15T10:30:03Z"
}
```

**Implementation:**

```typescript
async getJobStatus(userId: string, jobId: string): Promise<JobStatusResponse> {
  const job = await prisma.translationJob.findUnique({
    where: { id: jobId },
  });

  if (!job) {
    throw new Error('Translation job not found.');
  }

  if (job.user_id !== userId) {
    throw new Error('Access denied. This job does not belong to you.');
  }

  return {
    jobId: job.id,
    clientJobId: job.client_job_id || undefined,
    status: job.status as TranslationStatus,
    sourceLang: job.source_lang,
    targetLang: job.target_lang,
    model: job.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
    tone: job.tone as Tone,
    translation: job.translation || undefined,
    tokensUsed: job.tokens_used > 0 ? job.tokens_used : undefined,
    cost: job.cost.toNumber() > 0 ? job.cost.toNumber() : undefined,
    errorMessage: job.error_message || undefined,
    processingTimeMs: job.processing_time_ms || undefined,
    createdAt: job.created_at.toISOString(),
    updatedAt: job.updated_at.toISOString(),
    completedAt: job.completed_at?.toISOString(),
  };
}
```

---

### 5. Job Cancellation

#### `cancelJob(userId: string, jobId: string): Promise<JobStatusResponse>`

Cancels a **pending** translation job (before it starts processing).

**Restrictions:**
- Can only cancel jobs with status `pending`
- Cannot cancel jobs already `processing`, `completed`, `failed`, or `cancelled`

**No Credit Refund:**
- Since no credits were deducted for pending jobs (deduction happens after translation completes)

**Implementation:**

```typescript
async cancelJob(userId: string, jobId: string): Promise<JobStatusResponse> {
  const job = await prisma.translationJob.findUnique({
    where: { id: jobId },
  });

  if (!job) {
    throw new Error('Translation job not found.');
  }

  if (job.user_id !== userId) {
    throw new Error('Access denied. This job does not belong to you.');
  }

  // Can only cancel pending jobs
  if (job.status !== TranslationJobStatus.pending) {
    throw new Error(
      `Cannot cancel job with status '${job.status}'. Only pending jobs can be cancelled.`
    );
  }

  // Update job to cancelled
  const updatedJob = await prisma.translationJob.update({
    where: { id: jobId },
    data: {
      status: TranslationJobStatus.cancelled,
      completed_at: new Date(),
    },
  });

  // Track metrics
  trackTranslationJob(
    job.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
    'cancelled',
    'async'
  );

  return {
    jobId: updatedJob.id,
    clientJobId: updatedJob.client_job_id || undefined,
    status: updatedJob.status as TranslationStatus,
    sourceLang: updatedJob.source_lang,
    targetLang: updatedJob.target_lang,
    model: updatedJob.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
    tone: updatedJob.tone as Tone,
    createdAt: updatedJob.created_at.toISOString(),
    updatedAt: updatedJob.updated_at.toISOString(),
    completedAt: updatedJob.completed_at?.toISOString(),
  };
}
```

---

### 6. Job Retry

#### `retryJob(userId: string, jobId: string): Promise<JobStatusResponse>`

Re-queues a **failed** translation job for another processing attempt.

**Restrictions:**
- Can only retry jobs with status `failed`
- Resets job to `pending` status
- Clears error message, translation, tokens, cost
- Re-adds job to Bull queue

**Credit Flow:**
- No upfront credit check (will be validated when worker processes the job)

**Implementation:**

```typescript
async retryJob(userId: string, jobId: string): Promise<JobStatusResponse> {
  const job = await prisma.translationJob.findUnique({
    where: { id: jobId },
  });

  if (!job) {
    throw new Error('Translation job not found.');
  }

  if (job.user_id !== userId) {
    throw new Error('Access denied. This job does not belong to you.');
  }

  // Can only retry failed jobs
  if (job.status !== TranslationJobStatus.failed) {
    throw new Error(
      `Cannot retry job with status '${job.status}'. Only failed jobs can be retried.`
    );
  }

  // Reset job to pending status
  const updatedJob = await prisma.translationJob.update({
    where: { id: jobId },
    data: {
      status: TranslationJobStatus.pending,
      error_message: null,
      translation: null,
      tokens_used: 0,
      cost: 0,
      processing_time_ms: null,
      completed_at: null,
    },
  });

  // Re-queue job in Bull for async processing
  await translationQueue.add('translate', {
    jobId: updatedJob.id,
    clientJobId: updatedJob.client_job_id || undefined,
    userId,
    content: job.content,
    sourceLang: job.source_lang,
    targetLang: job.target_lang,
    model: job.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
    tone: job.tone as Tone,
    callbackUrl: job.callback_url || undefined,
    callbackSecret: job.callback_secret || undefined,
  });

  return {
    jobId: updatedJob.id,
    clientJobId: updatedJob.client_job_id || undefined,
    status: updatedJob.status as TranslationStatus,
    sourceLang: updatedJob.source_lang,
    targetLang: updatedJob.target_lang,
    model: updatedJob.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
    tone: updatedJob.tone as Tone,
    createdAt: updatedJob.created_at.toISOString(),
    updatedAt: updatedJob.updated_at.toISOString(),
    completedAt: updatedJob.completed_at?.toISOString(),
  };
}
```

---

### 7. Job List Query

#### `getJobs(userId: string, options): Promise<{ jobs: JobStatusResponse[], total: number, limit: number, offset: number }>`

Returns a paginated list of translation jobs for a user, with optional filtering.

**Options:**
```typescript
{
  status?: TranslationStatus | TranslationStatus[],  // Filter by status(es)
  limit?: number,                                    // Page size (default: 50)
  offset?: number,                                   // Skip N jobs (default: 0)
  sortBy?: 'created_at' | 'updated_at',             // Sort field (default: created_at)
  sortOrder?: 'asc' | 'desc'                        // Sort direction (default: desc)
}
```

**Response:**
```typescript
{
  jobs: [
    {
      jobId: "uuid",
      clientJobId: "optional",
      status: "completed",
      sourceLang: "en",
      targetLang: "es",
      model: "MODEL_4B",
      tone: "neutral",
      tokensUsed: 234,
      cost: 0.234,
      processingTimeMs: 3421,
      createdAt: "2024-01-15T10:30:00Z",
      updatedAt: "2024-01-15T10:30:03Z",
      completedAt: "2024-01-15T10:30:03Z"
    }
    // ... more jobs
  ],
  total: 142,
  limit: 50,
  offset: 0
}
```

**Implementation:**

```typescript
async getJobs(
  userId: string,
  options: {
    status?: TranslationStatus | TranslationStatus[];
    limit?: number;
    offset?: number;
    sortBy?: 'created_at' | 'updated_at';
    sortOrder?: 'asc' | 'desc';
  } = {}
) {
  const {
    status,
    limit = 50,
    offset = 0,
    sortBy = 'created_at',
    sortOrder = 'desc',
  } = options;

  const where: any = { user_id: userId };

  if (status) {
    where.status = Array.isArray(status)
      ? { in: status }
      : status;
  }

  const jobs = await prisma.translationJob.findMany({
    where,
    orderBy: { [sortBy]: sortOrder },
    take: limit,
    skip: offset,
    select: {
      id: true,
      client_job_id: true,
      status: true,
      source_lang: true,
      target_lang: true,
      model: true,
      tone: true,
      tokens_used: true,
      cost: true,
      error_message: true,
      processing_time_ms: true,
      created_at: true,
      updated_at: true,
      completed_at: true,
    },
  });

  const total = await prisma.translationJob.count({ where });

  return {
    jobs: jobs.map((job) => ({
      jobId: job.id,
      clientJobId: job.client_job_id || undefined,
      status: job.status as TranslationStatus,
      sourceLang: job.source_lang,
      targetLang: job.target_lang,
      model: job.model === 'MODEL_4B' ? Model.MODEL_4B : Model.MODEL_27B,
      tone: job.tone as Tone,
      tokensUsed: job.tokens_used > 0 ? job.tokens_used : undefined,
      cost: job.cost.toNumber() > 0 ? job.cost.toNumber() : undefined,
      errorMessage: job.error_message || undefined,
      processingTimeMs: job.processing_time_ms || undefined,
      createdAt: job.created_at.toISOString(),
      updatedAt: job.updated_at.toISOString(),
      completedAt: job.completed_at?.toISOString(),
    })),
    total,
    limit,
    offset,
  };
}
```

---

## Integration Points

### 1. Credit Service Integration

**Pre-Translation Validation:**
```typescript
// Check if user has enough credits
const estimatedTokens = estimateTokens(content, sourceLang, targetLang);
const sufficient = await hasSufficientCredits(userId, estimatedTokens);

if (!sufficient) {
  throw new Error('Insufficient credits');
}
```

**Post-Translation Deduction:**
```typescript
// Deduct actual tokens used
const transaction = await deductCredits(
  userId,
  actualTokens,
  `Translation: ${sourceLang} → ${targetLang} (${model})`,
  jobId
);

// Return new balance to client
const newBalance = transaction.balanceAfter;
```

**Refund Policy:**
- **No refund for sync translations** (even if API fails after credit check)
- **No refund for completed async jobs**
- **No deduction for cancelled pending jobs**
- **No deduction for failed async jobs** (if no tokens were consumed by Gemini)

---

### 2. Token Estimator Integration

**Estimation Before Translation:**
```typescript
const estimatedTokens = estimateTokens(
  request.content,
  request.sourceLang,
  request.targetLang
);
```

**ML Feedback After Translation:**
```typescript
// Record actual usage for ML model training (best-effort)
try {
  const tokenEstimatorInstance = new TokenEstimator();
  const complexity = tokenEstimatorInstance.analyzeHTMLComplexity(request.content);
  
  await accuracyTracker.recordUsage(
    request.targetLang,
    estimatedTokens,
    actualTokens,
    complexity.level
  );
} catch (mlError) {
  // Silent failure - don't block translation success
  logger.warn('Failed to record ML data', { mlError });
}
```

---

### 3. Gemini Client Integration

**Translation Call:**
```typescript
const geminiResponse = await geminiClient.translate(
  content,
  sourceLang,
  targetLang,
  model,
  tone
);

// Response structure:
// {
//   translation: string,
//   tokens_used: number
// }
```

**Error Handling:**
- **API errors:** Propagated to caller
- **Rate limiting:** Handled by Gemini client (exponential backoff)
- **Timeout:** Configured in Gemini client
- **Invalid response:** Throws error

---

### 4. Bull Queue Integration

**Job Submission:**
```typescript
await translationQueue.add('translate', {
  jobId: job.id,
  userId,
  content: request.content,
  sourceLang: request.sourceLang,
  targetLang: request.targetLang,
  model: request.model,
  tone: request.tone,
  callbackUrl: request.callbackUrl,
  callbackSecret: request.callbackSecret,
});
```

**Queue Configuration:**
- **Redis:** Bull stores jobs in Redis
- **Concurrency:** Configurable (default: 1 job at a time)
- **Retry:** Bull handles retries automatically
- **Priority:** FIFO by default
- **Job Data:** Serialized to JSON

---

### 5. Webhook Service Integration

**Success Webhook:**
```typescript
await deliverWebhook(
  jobId,
  {
    event: 'translation.completed',
    jobId,
    clientJobId,
    status: 'completed',
    translation: result.translation,
    tokensUsed: result.tokens_used,
    cost,
    processingTimeMs,
    timestamp: new Date().toISOString(),
  },
  callbackUrl,
  callbackSecret
);
```

**Failure Webhook:**
```typescript
await deliverWebhook(
  jobId,
  {
    event: 'translation.failed',
    jobId,
    clientJobId,
    status: 'failed',
    errorMessage: error.message,
    processingTimeMs,
    timestamp: new Date().toISOString(),
  },
  callbackUrl,
  callbackSecret
);
```

**Best-Effort Delivery:**
- Webhook failures **do not** fail the translation job
- Errors logged for monitoring
- Clients should poll job status as fallback

---

## Database Schema

### TranslationJob Table

```prisma
model TranslationJob {
  id                String                @id @default(uuid())
  user_id           String
  client_job_id     String?               // Client-provided reference ID
  status            TranslationJobStatus  @default(pending)
  source_lang       String
  target_lang       String
  model             String                // "MODEL_4B" or "MODEL_27B"
  tone              String                @default("neutral")
  content           String                @db.Text
  content_hash      String?               // For deduplication
  translation       String?               @db.Text
  tokens_used       Int                   @default(0)
  cost              Decimal               @default(0) @db.Decimal(10, 4)
  error_message     String?               @db.Text
  processing_time_ms Int?
  callback_url      String?
  callback_secret   String?
  created_at        DateTime              @default(now())
  updated_at        DateTime              @updatedAt
  completed_at      DateTime?

  user              User                  @relation(fields: [user_id], references: [id], onDelete: Cascade)
  credit_transactions CreditTransaction[] @relation("JobTransactions")

  @@index([user_id, status])
  @@index([content_hash])
  @@index([created_at])
}

enum TranslationJobStatus {
  pending
  processing
  completed
  failed
  cancelled
}
```

---

## Configuration

### Environment Variables

```bash
# Translation limits
MAX_SYNC_CHARS=5000           # Max chars for sync translation
MAX_ASYNC_CHARS=50000         # Max chars for async translation

# Bull queue
REDIS_URL=redis://localhost:6379

# Gemini API
GEMINI_API_KEY=your_api_key
GEMINI_TIMEOUT_MS=30000       # API timeout

# Webhook delivery
WEBHOOK_TIMEOUT_MS=10000      # Webhook delivery timeout
WEBHOOK_MAX_RETRIES=3         # Max retry attempts
```

### Config Object

```typescript
// api/src/config/index.ts
export const config = {
  maxSyncChars: parseInt(process.env.MAX_SYNC_CHARS || '5000'),
  maxAsyncChars: parseInt(process.env.MAX_ASYNC_CHARS || '50000'),
  redisUrl: process.env.REDIS_URL || 'redis://localhost:6379',
  geminiApiKey: process.env.GEMINI_API_KEY!,
  geminiTimeout: parseInt(process.env.GEMINI_TIMEOUT_MS || '30000'),
  webhookTimeout: parseInt(process.env.WEBHOOK_TIMEOUT_MS || '10000'),
  webhookMaxRetries: parseInt(process.env.WEBHOOK_MAX_RETRIES || '3'),
};
```

---

## Error Handling

### Error Types

| Error Type | HTTP Status | Description | Retry Strategy |
|------------|-------------|-------------|----------------|
| **Insufficient Credits** | 402 | User lacks credits for translation | User must add credits |
| **Content Too Long** | 400 | Content exceeds size limit | Use async mode or split content |
| **Job Not Found** | 404 | Job ID doesn't exist | Check job ID |
| **Access Denied** | 403 | Job belongs to different user | Check authentication |
| **Invalid Status** | 400 | Can't cancel/retry in current status | Check job status first |
| **Gemini API Error** | 500 | Translation service unavailable | Automatic retry (async) or manual retry |
| **Webhook Delivery Failed** | N/A | Callback endpoint unreachable | Logged, job still succeeds |

### Error Response Format

```typescript
{
  error: "Insufficient credits",
  message: "You need 234 credits but only have 150. Please upgrade your plan.",
  code: "INSUFFICIENT_CREDITS",
  timestamp: "2024-01-15T10:30:00Z"
}
```

### Sync Translation Error Handling

```typescript
try {
  const result = await translationService.translateSync(userId, request);
  res.json(result);
} catch (error) {
  if (error.message.includes('Insufficient credits')) {
    res.status(402).json({
      error: 'Insufficient credits',
      message: error.message,
      code: 'INSUFFICIENT_CREDITS',
    });
  } else if (error.message.includes('too long')) {
    res.status(400).json({
      error: 'Content too long',
      message: error.message,
      code: 'CONTENT_TOO_LONG',
    });
  } else {
    res.status(500).json({
      error: 'Translation failed',
      message: error.message,
      code: 'TRANSLATION_ERROR',
    });
  }
}
```

### Async Job Error Handling

```typescript
// Worker handles errors internally:
// - Updates job status to 'failed'
// - Records error_message
// - Delivers failure webhook
// - Logs error for monitoring

// Client checks job status:
const job = await translationService.getJobStatus(userId, jobId);

if (job.status === 'failed') {
  console.error('Translation failed:', job.errorMessage);
  
  // Option 1: Retry the job
  await translationService.retryJob(userId, jobId);
  
  // Option 2: Submit a new job with adjusted parameters
  await translationService.translateAsync(userId, newRequest);
}
```

---

## Monitoring & Metrics

### Prometheus Metrics

```typescript
// Translation job tracking
trackTranslationJob(
  model: Model,
  status: 'completed' | 'failed' | 'cancelled',
  mode: 'sync' | 'async',
  processingTimeMs?: number
);

// Token usage tracking
trackTokensProcessed(model: Model, tokensUsed: number);
```

**Metric Names:**
- `translation_jobs_total{model, status, mode}` - Counter
- `translation_processing_time_ms{model, mode}` - Histogram
- `translation_tokens_used_total{model}` - Counter

### Structured Logging

```typescript
// Start translation
logger.info('Starting synchronous translation', {
  userId,
  sourceLang,
  targetLang,
  model,
  tone,
  contentLength,
  estimatedTokens,
  estimatedCost,
});

// Translation completed
logger.info('Synchronous translation completed', {
  userId,
  jobId,
  tokensUsed,
  cost,
  processingTimeMs,
  newBalance,
});

// Translation failed
logger.error('Synchronous translation failed', {
  userId,
  jobId,
  error: errorMessage,
  processingTimeMs,
});

// Webhook failure (non-critical)
logger.error('Webhook delivery failed', {
  jobId,
  error: webhookError,
});

// ML tracking failure (non-critical)
logger.warn('Failed to record ML data', { mlError });
```

---

## Performance Optimization

### 1. Deduplication

**24-Hour Cache for Sync Translations:**
```typescript
const oneDayAgo = new Date(Date.now() - 24 * 60 * 60 * 1000);
const duplicateJob = await prisma.translationJob.findFirst({
  where: {
    user_id: userId,
    content_hash: contentHash,
    status: TranslationJobStatus.completed,
    created_at: { gte: oneDayAgo },
  },
});

if (duplicateJob && duplicateJob.translation) {
  // Return cached translation (no API call, no credit deduction)
  return {
    jobId: duplicateJob.id,
    status: TranslationStatus.COMPLETED,
    translation: duplicateJob.translation,
    tokensUsed: duplicateJob.tokens_used,
    cost: parseFloat(duplicateJob.cost.toString()),
    creditBalance: currentBalance,
  };
}
```

**Duplicate Pending Job Check for Async:**
```typescript
const duplicatePendingJob = await prisma.translationJob.findFirst({
  where: {
    user_id: userId,
    content_hash: contentHash,
    status: {
      in: [TranslationJobStatus.pending, TranslationJobStatus.processing],
    },
  },
});

if (duplicatePendingJob) {
  // Return existing job ID (no new queue entry)
  return {
    jobId: duplicatePendingJob.id,
    status: duplicatePendingJob.status,
  };
}
```

### 2. Database Indexing

**Optimized Queries:**
```sql
-- Job lookup by user and status
CREATE INDEX idx_user_status ON TranslationJob(user_id, status);

-- Deduplication lookup
CREATE INDEX idx_content_hash ON TranslationJob(content_hash);

-- Job history sorting
CREATE INDEX idx_created_at ON TranslationJob(created_at);
```

### 3. Async Processing Benefits

**Why Async Mode?**
- **Non-blocking:** API returns immediately (faster UX)
- **Scalable:** Bull queue handles backlog gracefully
- **Resilient:** Failed jobs can be retried without client intervention
- **Large content:** Supports up to 50,000 characters (10x sync limit)

**When to Use Async:**
- Content > 5,000 characters
- Batch translations
- Non-urgent translations
- Integration with background workflows

---

## Testing

### Unit Tests

```typescript
// translationService.test.ts
describe('TranslationService', () => {
  describe('translateSync', () => {
    it('should validate content length', async () => {
      const longContent = 'a'.repeat(5001);
      
      await expect(
        translationService.translateSync(userId, {
          content: longContent,
          sourceLang: 'en',
          targetLang: 'es',
          model: Model.MODEL_4B,
        })
      ).rejects.toThrow('Content too long for synchronous translation');
    });

    it('should check for sufficient credits', async () => {
      // Mock hasSufficientCredits to return false
      jest.spyOn(creditService, 'hasSufficientCredits').mockResolvedValue(false);
      
      await expect(
        translationService.translateSync(userId, {
          content: 'Hello world',
          sourceLang: 'en',
          targetLang: 'es',
          model: Model.MODEL_4B,
        })
      ).rejects.toThrow('Insufficient credits');
    });

    it('should return cached translation if available', async () => {
      // Create a completed job with same content
      const cachedJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.completed,
          content: 'Hello world',
          content_hash: generateContentHash('Hello world', 'en', 'es', Model.MODEL_4B),
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
          translation: 'Hola mundo',
          tokens_used: 10,
          cost: 0.01,
          created_at: new Date(),
        },
      });

      const result = await translationService.translateSync(userId, {
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: Model.MODEL_4B,
      });

      expect(result.jobId).toBe(cachedJob.id);
      expect(result.translation).toBe('Hola mundo');
      expect(geminiClient.translate).not.toHaveBeenCalled();
    });

    it('should deduct actual tokens used', async () => {
      jest.spyOn(geminiClient, 'translate').mockResolvedValue({
        translation: 'Hola mundo',
        tokens_used: 15,
      });

      const result = await translationService.translateSync(userId, {
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: Model.MODEL_4B,
      });

      expect(result.tokensUsed).toBe(15);
      
      const balance = await getCurrentBalance(userId);
      expect(balance).toBe(initialBalance - 15);
    });
  });

  describe('translateAsync', () => {
    it('should validate content length', async () => {
      const longContent = 'a'.repeat(50001);
      
      await expect(
        translationService.translateAsync(userId, {
          content: longContent,
          sourceLang: 'en',
          targetLang: 'es',
          model: Model.MODEL_4B,
        })
      ).rejects.toThrow('Content too long');
    });

    it('should return existing job if duplicate pending', async () => {
      const pendingJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.pending,
          content: 'Hello world',
          content_hash: generateContentHash('Hello world', 'en', 'es', Model.MODEL_4B),
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
        },
      });

      const result = await translationService.translateAsync(userId, {
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: Model.MODEL_4B,
      });

      expect(result.jobId).toBe(pendingJob.id);
      expect(translationQueue.add).not.toHaveBeenCalled();
    });

    it('should add job to queue', async () => {
      const result = await translationService.translateAsync(userId, {
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: Model.MODEL_4B,
      });

      expect(result.status).toBe(TranslationStatus.PENDING);
      expect(translationQueue.add).toHaveBeenCalledWith('translate', expect.objectContaining({
        jobId: result.jobId,
        userId,
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: Model.MODEL_4B,
      }));
    });
  });

  describe('cancelJob', () => {
    it('should cancel pending job', async () => {
      const pendingJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.pending,
          content: 'Hello world',
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
        },
      });

      const result = await translationService.cancelJob(userId, pendingJob.id);

      expect(result.status).toBe(TranslationStatus.CANCELLED);
    });

    it('should not cancel processing job', async () => {
      const processingJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.processing,
          content: 'Hello world',
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
        },
      });

      await expect(
        translationService.cancelJob(userId, processingJob.id)
      ).rejects.toThrow("Cannot cancel job with status 'processing'");
    });
  });

  describe('retryJob', () => {
    it('should retry failed job', async () => {
      const failedJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.failed,
          content: 'Hello world',
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
          error_message: 'API timeout',
        },
      });

      const result = await translationService.retryJob(userId, failedJob.id);

      expect(result.status).toBe(TranslationStatus.PENDING);
      expect(translationQueue.add).toHaveBeenCalledWith('translate', expect.objectContaining({
        jobId: failedJob.id,
      }));
    });

    it('should not retry completed job', async () => {
      const completedJob = await prisma.translationJob.create({
        data: {
          user_id: userId,
          status: TranslationJobStatus.completed,
          content: 'Hello world',
          source_lang: 'en',
          target_lang: 'es',
          model: 'MODEL_4B',
          translation: 'Hola mundo',
          tokens_used: 10,
          cost: 0.01,
        },
      });

      await expect(
        translationService.retryJob(userId, completedJob.id)
      ).rejects.toThrow("Cannot retry job with status 'completed'");
    });
  });
});
```

### Integration Tests

```typescript
// routes/translate.test.ts
describe('POST /v1/translate', () => {
  it('should translate synchronously', async () => {
    const apiKey = 'sk_test_123';

    const response = await request(app)
      .post('/v1/translate')
      .set('Authorization', `Bearer ${apiKey}`)
      .send({
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: 'MODEL_4B',
      });

    expect(response.status).toBe(200);
    expect(response.body).toMatchObject({
      jobId: expect.any(String),
      status: 'completed',
      translation: expect.any(String),
      tokensUsed: expect.any(Number),
      cost: expect.any(Number),
      processingTimeMs: expect.any(Number),
      creditBalance: expect.any(Number),
    });
  });

  it('should return 402 for insufficient credits', async () => {
    // Drain user credits
    await deductCredits(userId, 10000, 'Test deduction');

    const response = await request(app)
      .post('/v1/translate')
      .set('Authorization', `Bearer ${apiKey}`)
      .send({
        content: 'Hello world',
        sourceLang: 'en',
        targetLang: 'es',
        model: 'MODEL_4B',
      });

    expect(response.status).toBe(402);
    expect(response.body.error).toBe('Insufficient credits');
  });
});

describe('POST /v1/jobs/submit', () => {
  it('should submit async job', async () => {
    const apiKey = 'sk_test_123';

    const response = await request(app)
      .post('/v1/jobs/submit')
      .set('Authorization', `Bearer ${apiKey}`)
      .send({
        content: 'A'.repeat(10000),
        sourceLang: 'en',
        targetLang: 'es',
        model: 'MODEL_4B',
        callbackUrl: 'https://example.com/webhook',
        callbackSecret: 'secret123',
      });

    expect(response.status).toBe(200);
    expect(response.body).toMatchObject({
      jobId: expect.any(String),
      status: 'pending',
    });
  });
});
```

---

## Production Considerations

### 1. Scaling Async Workers

**Horizontal Scaling:**
```bash
# Run multiple worker processes
pm2 start api/dist/worker.js -i 4

# Or use systemd with multiple instances
systemctl start presszone-worker@1
systemctl start presszone-worker@2
systemctl start presszone-worker@3
systemctl start presszone-worker@4
```

**Bull Concurrency:**
```typescript
// Process N jobs concurrently per worker
translationQueue.process(5, async (job) => {
  // Process job
});
```

**Redis Scaling:**
- Use Redis Cluster for high availability
- Enable Redis persistence (AOF + RDB)
- Monitor Redis memory usage

### 2. Rate Limiting

**Gemini API Rate Limits:**
- Handled internally by `geminiClient` (exponential backoff)
- Configure timeout: `GEMINI_TIMEOUT_MS=30000`

**Queue Rate Limiting:**
```typescript
// Limit job submission rate per user
await rateLimiter.limit({
  key: `translate:${userId}`,
  maxRequests: 100,
  windowMs: 60000, // 100 requests per minute
});
```

### 3. Monitoring & Alerts

**Health Checks:**
```bash
# Check worker health
curl https://api.press.zone/health

# Check queue depth
redis-cli LLEN bull:translation:wait
```

**Alerting Thresholds:**
- Queue depth > 1000 jobs
- Failed job rate > 5%
- Average processing time > 30 seconds
- Redis memory > 80% capacity
- Worker process crashes

### 4. Disaster Recovery

**Job Recovery:**
```typescript
// Reprocess stuck jobs
const stuckJobs = await prisma.translationJob.findMany({
  where: {
    status: TranslationJobStatus.processing,
    updated_at: {
      lt: new Date(Date.now() - 60 * 60 * 1000), // > 1 hour old
    },
  },
});

for (const job of stuckJobs) {
  await translationService.retryJob(job.user_id, job.id);
}
```

**Queue Drain:**
```typescript
// Gracefully drain queue before maintenance
await translationQueue.pause();
await translationQueue.whenCurrentJobsFinished();
```

---

## Quick Reference

### Function Signatures

```typescript
// Synchronous translation
translateSync(userId: string, request: TranslationRequest): Promise<TranslationResponse>

// Asynchronous translation
translateAsync(userId: string, request: JobSubmitRequest): Promise<JobSubmitResponse>

// Job status query
getJobStatus(userId: string, jobId: string): Promise<JobStatusResponse>

// Job cancellation
cancelJob(userId: string, jobId: string): Promise<JobStatusResponse>

// Job retry
retryJob(userId: string, jobId: string): Promise<JobStatusResponse>

// Job list query
getJobs(userId: string, options?: {
  status?: TranslationStatus | TranslationStatus[];
  limit?: number;
  offset?: number;
  sortBy?: 'created_at' | 'updated_at';
  sortOrder?: 'asc' | 'desc';
}): Promise<{
  jobs: JobStatusResponse[];
  total: number;
  limit: number;
  offset: number;
}>
```

### Typical Workflow (Sync)

```typescript
1. Client → POST /v1/translate
2. translateSync() validates content length & credits
3. Estimate tokens via TokenEstimator
4. Check for duplicate translation (24h cache)
5. Create TranslationJob (status: processing)
6. Call geminiClient.translate()
7. Deduct actual tokens used via creditService
8. Update job (status: completed, translation, tokens, cost)
9. Track metrics & ML feedback
10. Return translation + updated balance
```

### Typical Workflow (Async)

```typescript
1. Client → POST /v1/jobs/submit
2. translateAsync() validates content length & credits
3. Check for duplicate pending job
4. Create TranslationJob (status: pending)
5. Add job to Bull queue
6. Return jobId immediately

[Background Worker]
7. Pick job from queue → update to processing
8. Call geminiClient.translate()
9. Atomic: deduct credits + update job (completed)
10. Deliver webhook (if provided)
11. Client polls GET /v1/jobs/:jobId or receives webhook
```

### Common Queries

```typescript
// Submit sync translation
const result = await translationService.translateSync(userId, {
  content: 'Hello world',
  sourceLang: 'en',
  targetLang: 'es',
  model: Model.MODEL_4B,
  tone: Tone.NEUTRAL,
});

// Submit async job
const job = await translationService.translateAsync(userId, {
  content: longArticle,
  sourceLang: 'en',
  targetLang: 'es',
  model: Model.MODEL_27B,
  callbackUrl: 'https://example.com/webhook',
  callbackSecret: 'secret123',
});

// Check job status
const status = await translationService.getJobStatus(userId, jobId);

// Cancel pending job
await translationService.cancelJob(userId, jobId);

// Retry failed job
await translationService.retryJob(userId, jobId);

// List user's jobs
const { jobs, total } = await translationService.getJobs(userId, {
  status: [TranslationStatus.COMPLETED, TranslationStatus.FAILED],
  limit: 20,
  offset: 0,
  sortBy: 'created_at',
  sortOrder: 'desc',
});
```

---

## Validation Checklist

Before deploying translation service:

- [ ] Sync translations enforce 5,000 character limit
- [ ] Async translations enforce 50,000 character limit
- [ ] Empty content rejected with 400 error
- [ ] Insufficient credits return 402 Payment Required
- [ ] Content hash deduplication working (24h cache for sync)
- [ ] Duplicate pending jobs detected (async)
- [ ] TranslationJob record created before API call
- [ ] Credits deducted **after** successful translation (actual tokens)
- [ ] No credit deduction for failed translations
- [ ] Job status correctly transitions: pending → processing → completed/failed
- [ ] Bull queue job data serialized correctly
- [ ] Worker processes jobs from queue
- [ ] Webhook delivery attempted (best-effort)
- [ ] Webhook failures don't fail translation job
- [ ] ML feedback recorded (best-effort, silent failure)
- [ ] Prometheus metrics tracked (jobs, tokens, processing time)
- [ ] Structured logging for all operations
- [ ] Job access control (users can only access their own jobs)
- [ ] Job cancellation only allowed for pending jobs
- [ ] Job retry only allowed for failed jobs
- [ ] Pagination implemented for job list
- [ ] Worker graceful shutdown on SIGTERM/SIGINT
- [ ] Redis connection handled gracefully
- [ ] Gemini API errors propagated correctly
- [ ] Unit tests cover edge cases
- [ ] Integration tests verify end-to-end flows

---

---

# Skill 19: Google Gemini API Integration

## Overview

The **Gemini Client** is the core translation engine that interfaces directly with Google's Gemini API. It handles all aspects of AI-powered translation including request formatting, HTML preservation, model selection, token counting, error handling, and dynamic configuration management.

**Purpose**: Execute high-quality neural machine translation using Google's state-of-the-art Gemini models while preserving HTML structure and handling edge cases gracefully.

**Location**: `api/src/services/geminiClient.ts`

**Dependencies**:
- `@google/generative-ai` - Official Google Gemini SDK
- `api/src/config` - Configuration management with dynamic refresh
- `api/src/utils/logger` - Structured logging
- `api/src/types` - Model, Tone, and response type definitions

**Key Features**:
- Direct integration with Google Gemini API
- HTML tag preservation during translation
- Dynamic configuration refresh (no restart required)
- Token counting for accurate billing
- Comprehensive error handling with retry logic
- Support for 35+ language pairs
- Tone/style customization (formal, casual, neutral)
- Health check monitoring
- Timeout protection (60s default)

---

## Architecture

### Class Structure

```typescript
class GeminiClient {
  // Singleton instance
  private client: GoogleGenerativeAI | null = null;
  private model: GenerativeModel | null = null;
  private readonly timeout = 60000; // 60 seconds
  private currentConfig: { apiKey: string } | null = null;

  constructor() {
    this.initializeClient();
    this.subscribeToConfigChanges();
  }

  // Core translation method
  async translate(
    content: string,
    sourceLang: string,
    targetLang: string,
    model: Model,
    tone: Tone = Tone.NEUTRAL
  ): Promise<GeminiTranslationResponse>

  // Health check
  async healthCheck(): Promise<boolean>

  // Configuration management
  refreshConfig(): void
  getCurrentConfig(): { hasKey: boolean } | null

  // Internal helpers
  private initializeClient(): void
  private extractHtmlTags(content: string): { text: string; tags: Map<string, string> }
  private restoreHtmlTags(text: string, tags: Map<string, string>): string
  private getToneInstruction(tone: Tone): string
  private getLanguageName(code: string): string
}

// Export singleton
export const geminiClient = new GeminiClient();
```

### Model Selection

**Current Model**: `gemini-3-flash-preview`

This model is hardcoded in the client (line 43) and provides:
- Fast translation responses (typically <5s)
- High-quality neural machine translation
- Support for 100+ languages
- HTML-aware context understanding
- Cost-effective pricing ($0.00015 per 1K tokens)

**Future Support**: While the `translate()` method accepts a `model` parameter, it currently ignores it. To support multiple models:

```typescript
private getModelName(model: Model): string {
  const modelMap: Record<Model, string> = {
    [Model.GEMINI_FLASH]: 'gemini-3-flash-preview',
    [Model.GEMINI_PRO]: 'gemini-1.5-pro',
    [Model.GEMINI_PRO_VISION]: 'gemini-pro-vision',
  };
  return modelMap[model] || 'gemini-3-flash-preview';
}

// In translate():
const selectedModel = this.client.getGenerativeModel({ 
  model: this.getModelName(model) 
});
```

---

## Core Features

### 1. HTML Tag Preservation

**Challenge**: Gemini might translate or corrupt HTML tags embedded in content.

**Solution**: Extract tags into placeholders before translation, restore after.

#### Extraction Process

```typescript
private extractHtmlTags(content: string): { text: string; tags: Map<string, string> } {
  const tags = new Map<string, string>();
  let counter = 0;

  // Regex matches opening, closing, and self-closing tags
  const htmlTagRegex = /<\/?[a-zA-Z][a-zA-Z0-9-]*(?:\s+[a-zA-Z][a-zA-Z0-9-]*(?:="[^"]*"|='[^']*'|=[^\s>]*)?)*\s*\/?>/g;

  const text = content.replace(htmlTagRegex, (match) => {
    const placeholder = `__TAG_${counter}__`;
    tags.set(placeholder, match);
    counter++;
    return placeholder;
  });

  return { text, tags };
}
```

**Example**:

```
Input:
  "Welcome to <strong>Press.Zone</strong>! Visit <a href='/forum'>our forum</a>."

After extraction:
  "Welcome to __TAG_0__Press.Zone__TAG_1__! Visit __TAG_2__our forum__TAG_3__."

Tags map:
  __TAG_0__ → <strong>
  __TAG_1__ → </strong>
  __TAG_2__ → <a href='/forum'>
  __TAG_3__ → </a>
```

#### Restoration Process

```typescript
private restoreHtmlTags(text: string, tags: Map<string, string>): string {
  let result = text;
  
  tags.forEach((tag, placeholder) => {
    result = result.replace(placeholder, tag);
  });
  
  return result;
}
```

**Why This Works**:
- Gemini sees placeholders as untranslatable tokens
- Placeholders maintain exact positions in translated text
- Original HTML structure preserved perfectly
- Attributes (href, class, id) remain unchanged

#### Prompt Instructions

The translation prompt includes explicit rules (lines 217-222):

```
IMPORTANT RULES:
1. Preserve all placeholders in the format __TAG_N__ exactly as they appear
2. Do not translate the placeholders themselves
3. Only translate the actual text content
4. Maintain the exact position and format of placeholders
5. Output ONLY the translated text, no explanations or additional text
```

---

### 2. Language Support

The client supports 35 major languages with full language name mapping:

```typescript
private getLanguageName(code: string): string {
  const languageMap: Record<string, string> = {
    'en': 'English',
    'es': 'Spanish',
    'fr': 'French',
    'de': 'German',
    'it': 'Italian',
    'pt': 'Portuguese',
    'nl': 'Dutch',
    'pl': 'Polish',
    'ru': 'Russian',
    'ja': 'Japanese',
    'ko': 'Korean',
    'zh': 'Chinese',
    'ar': 'Arabic',
    'hi': 'Hindi',
    'tr': 'Turkish',
    'vi': 'Vietnamese',
    'th': 'Thai',
    'id': 'Indonesian',
    'ms': 'Malay',
    'sv': 'Swedish',
    'da': 'Danish',
    'no': 'Norwegian',
    'fi': 'Finnish',
    'cs': 'Czech',
    'el': 'Greek',
    'he': 'Hebrew',
    'ro': 'Romanian',
    'hu': 'Hungarian',
    'uk': 'Ukrainian',
    'bg': 'Bulgarian',
    'sr': 'Serbian',
    'hr': 'Croatian',
    'sk': 'Slovak',
    'sl': 'Slovenian',
    'et': 'Estonian',
    'lv': 'Latvian',
    'lt': 'Lithuanian',
  };
  
  return languageMap[code] || code.toUpperCase();
}
```

**Why Use Full Names**: Gemini performs better with explicit language names ("Spanish") rather than ISO codes ("es") in prompts.

**Adding New Languages**: Simply extend the `languageMap` object. Gemini supports 100+ languages; the limitation is mapping ISO codes to names.

---

### 3. Tone/Style Customization

The client supports three translation tones:

```typescript
enum Tone {
  FORMAL = 'formal',
  CASUAL = 'casual',
  NEUTRAL = 'neutral',
}

private getToneInstruction(tone: Tone): string {
  const toneInstructions = {
    [Tone.FORMAL]: 'Use formal, professional language.',
    [Tone.CASUAL]: 'Use casual, conversational language.',
    [Tone.NEUTRAL]: 'Use neutral, standard language.',
  };
  
  return toneInstructions[tone] || toneInstructions[Tone.NEUTRAL];
}
```

**Use Cases**:
- **Formal**: Legal documents, academic papers, corporate communications
- **Casual**: Blog posts, social media, forum discussions
- **Neutral** (default): Product descriptions, news articles, general content

**Example Differences**:

```
English: "You need to complete the form."

Spanish (Formal):  "Debe completar el formulario."
Spanish (Casual):  "Tienes que completar el formulario."
Spanish (Neutral): "Necesita completar el formulario."
```

---

### 4. Dynamic Configuration Management

**Problem**: API keys may change via admin panel; restarting the API server is disruptive.

**Solution**: Subscribe to config change events and refresh client automatically.

#### Initialization

```typescript
constructor() {
  this.initializeClient();
  
  // Subscribe to Gemini-specific config changes
  onConfigChange('gemini', () => {
    logger.info('Gemini config changed, refreshing client...');
    this.refreshConfig();
  });
  
  // Subscribe to global config changes
  onConfigChange('all', () => {
    logger.info('All config changed, refreshing Gemini client...');
    this.refreshConfig();
  });
}

private initializeClient(): void {
  try {
    const geminiConfig = getGeminiConfig();
    this.currentConfig = geminiConfig;
    
    this.client = new GoogleGenerativeAI(geminiConfig.apiKey);
    this.model = this.client.getGenerativeModel({ model: 'gemini-3-flash-preview' });
    
    logger.info('Gemini client initialized', {
      model: 'gemini-3-flash-preview',
      hasKey: !!geminiConfig.apiKey,
    });
  } catch (error) {
    logger.error('Failed to initialize Gemini client', { error });
    throw error;
  }
}
```

#### Refresh Logic

```typescript
refreshConfig(): void {
  try {
    const newConfig = getGeminiConfig();
    
    // Only reinitialize if config has actually changed
    if (
      !this.currentConfig ||
      this.currentConfig.apiKey !== newConfig.apiKey
    ) {
      logger.info('Gemini config changed, reinitializing client...');
      this.initializeClient();
    } else {
      logger.debug('Gemini config unchanged, skipping reinitialize');
    }
  } catch (error) {
    logger.error('Failed to refresh Gemini client config', { error });
  }
}
```

**Behavior**:
1. Admin updates Gemini API key in settings panel
2. Settings service emits `gemini` config change event
3. Gemini client receives event → calls `refreshConfig()`
4. Client compares old vs new API key
5. If changed → reinitializes Google SDK client
6. All subsequent translations use new API key

**No Downtime**: In-flight translations continue with old client; new requests use new client.

---

### 5. Token Counting for Billing

Accurate token counting is critical for:
- Cost estimation (before translation)
- Credit deduction (after translation)
- Usage analytics
- ML model training (accuracy feedback)

#### Input Token Counting

```typescript
// Count tokens in prompt BEFORE translation
const tokenCountResult = await this.model.countTokens(prompt);
const inputTokens = tokenCountResult.totalTokens;
```

**What's Counted**: The full prompt including:
- System instructions
- Tone instructions
- HTML preservation rules
- Source and target language names
- The actual content (with placeholders)

**Typical Overhead**: System prompt adds ~150 tokens per request.

#### Output Token Counting

```typescript
// Count tokens in translation AFTER receiving response
const outputTokenCountResult = await this.model.countTokens(translatedText);
const outputTokens = outputTokenCountResult.totalTokens;
```

**What's Counted**: Only the translated text (with placeholders intact).

#### Total Billing

```typescript
const totalTokens = inputTokens + outputTokens;

return {
  translation: finalTranslation,
  tokens_used: totalTokens,
  processing_time_ms: processingTime,
  model,
};
```

**Cost Calculation**: Performed by `creditService.calculateCreditCost(totalTokens)`.

**Accuracy Tracking**: The `totalTokens` value is stored in `accuracy_stats` table to improve future estimates (see Skill 22).

---

### 6. Error Handling

The client implements robust error handling with specific error mapping:

```typescript
try {
  // Translation logic
} catch (error) {
  if (error instanceof Error) {
    const errorMessage = error.message;
    
    // Timeout
    if (errorMessage.includes('timed out')) {
      throw new Error('Translation request timed out. Please try again or use async translation for larger content.');
    }
    
    // Authentication
    if (errorMessage.includes('API key')) {
      throw new Error('Gemini API authentication failed.');
    }
    
    // Rate limiting / Quota
    if (errorMessage.includes('quota') || errorMessage.includes('rate limit')) {
      throw new Error('Gemini API rate limit exceeded. Please try again later.');
    }
    
    // Model unavailable
    if (errorMessage.includes('model') || errorMessage.includes('not found')) {
      throw new Error('Translation model temporarily unavailable. Please try again later.');
    }
    
    // Generic error
    throw new Error(`Translation failed: ${errorMessage}`);
  }
  
  // Unknown error type
  throw new Error('An unexpected error occurred during translation. Please try again.');
}
```

#### Error Categories

| Error Type | Cause | User-Facing Message | HTTP Status |
|------------|-------|---------------------|-------------|
| **Timeout** | Translation took >60s | "Translation request timed out. Please try again or use async translation for larger content." | 408 |
| **Authentication** | Invalid/expired API key | "Gemini API authentication failed." | 401 |
| **Rate Limit** | Quota exceeded | "Gemini API rate limit exceeded. Please try again later." | 429 |
| **Model Unavailable** | Model temporarily down | "Translation model temporarily unavailable. Please try again later." | 503 |
| **Generic** | Unknown API error | "Translation failed: [error message]" | 500 |
| **Unknown** | Non-Error thrown | "An unexpected error occurred during translation. Please try again." | 500 |

#### Timeout Protection

```typescript
const result = await Promise.race<GenerateContentResult>([
  this.model.generateContent(prompt),
  new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error('Translation request timed out')), this.timeout)
  ),
]);
```

**Behavior**:
- If translation completes within 60s → return result
- If exceeds 60s → reject with timeout error
- Prevents hanging requests from blocking API server

**Recommendation for Large Content**: Use async translation (queue-based) for content >5000 characters.

---

### 7. Health Check Monitoring

The client provides a health check method for monitoring systems:

```typescript
async healthCheck(): Promise<boolean> {
  try {
    if (!this.client || !this.model) {
      logger.warn('Gemini client not initialized');
      return false;
    }
    
    // Test with a minimal translation (5s timeout)
    const testPrompt = 'Translate "hello" to Spanish';
    const result = await Promise.race([
      this.model.generateContent(testPrompt),
      new Promise<never>((_, reject) =>
        setTimeout(() => reject(new Error('Health check timeout')), 5000)
      ),
    ]);
    
    const response = result.response;
    return !!response.text();
  } catch (error) {
    logger.warn('Gemini health check failed', {
      error: error instanceof Error ? error.message : 'Unknown error',
    });
    return false;
  }
}
```

**Used By**: `GET /health` endpoint (see Skill 11, Section 1).

**Health Check Flow**:
1. Verify client is initialized
2. Send minimal translation request ("hello" → Spanish)
3. Wait up to 5 seconds for response
4. Return `true` if successful, `false` otherwise

**Monitoring Integration**:

```typescript
// In health endpoint
const geminiHealthy = await geminiClient.healthCheck();

return {
  status: geminiHealthy ? 'healthy' : 'degraded',
  components: {
    gemini: {
      status: geminiHealthy ? 'up' : 'down',
      latency_ms: measureLatency(),
    },
  },
};
```

**Alerting**: Prometheus metrics track health check failures (`gemini_health_check_failures_total`).

---

## Translation Workflow

### Step-by-Step Execution

#### 1. Initialization Check

```typescript
async translate(
  content: string,
  sourceLang: string,
  targetLang: string,
  model: Model,
  tone: Tone = Tone.NEUTRAL
): Promise<GeminiTranslationResponse> {
  if (!this.client || !this.model) {
    throw new Error('Gemini client not initialized');
  }
  
  const startTime = Date.now();
  // ...
}
```

**Validation**: Ensures client is properly initialized before attempting translation.

#### 2. HTML Extraction

```typescript
const { text: cleanText, tags } = this.extractHtmlTags(content);
```

**Example**:

```
Input:
  "<p>Welcome to <strong>Press.Zone</strong>!</p>"

cleanText:
  "__TAG_0__Welcome to __TAG_1__Press.Zone__TAG_2__!__TAG_3__"

tags:
  Map {
    '__TAG_0__' => '<p>',
    '__TAG_1__' => '<strong>',
    '__TAG_2__' => '</strong>',
    '__TAG_3__' => '</p>'
  }
```

#### 3. Prompt Construction

```typescript
const sourceLanguage = this.getLanguageName(sourceLang); // "en" → "English"
const targetLanguage = this.getLanguageName(targetLang); // "es" → "Spanish"
const toneInstruction = this.getToneInstruction(tone);   // "Use neutral, standard language."

const prompt = `Translate the following text from ${sourceLanguage} to ${targetLanguage}.

${toneInstruction}

IMPORTANT RULES:
1. Preserve all placeholders in the format __TAG_N__ exactly as they appear
2. Do not translate the placeholders themselves
3. Only translate the actual text content
4. Maintain the exact position and format of placeholders
5. Output ONLY the translated text, no explanations or additional text

Text to translate:
${cleanText}`;
```

**Full Prompt Example**:

```
Translate the following text from English to Spanish.

Use neutral, standard language.

IMPORTANT RULES:
1. Preserve all placeholders in the format __TAG_N__ exactly as they appear
2. Do not translate the placeholders themselves
3. Only translate the actual text content
4. Maintain the exact position and format of placeholders
5. Output ONLY the translated text, no explanations or additional text

Text to translate:
__TAG_0__Welcome to __TAG_1__Press.Zone__TAG_2__!__TAG_3__
```

#### 4. Logging

```typescript
logger.info('Calling Gemini translation API', {
  sourceLang,
  targetLang,
  model: 'gemini-3-flash-preview',
  tone,
  contentLength: content.length,
  cleanTextLength: cleanText.length,
  tagCount: tags.size,
});
```

**Log Output**:

```json
{
  "level": "info",
  "message": "Calling Gemini translation API",
  "sourceLang": "en",
  "targetLang": "es",
  "model": "gemini-3-flash-preview",
  "tone": "neutral",
  "contentLength": 45,
  "cleanTextLength": 62,
  "tagCount": 4,
  "timestamp": "2026-01-27T14:30:00.000Z"
}
```

#### 5. Token Counting (Input)

```typescript
const tokenCountResult = await this.model.countTokens(prompt);
const inputTokens = tokenCountResult.totalTokens;
```

**Purpose**: Accurate billing and cost estimation.

#### 6. API Call with Timeout

```typescript
const result = await Promise.race<GenerateContentResult>([
  this.model.generateContent(prompt),
  new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error('Translation request timed out')), this.timeout)
  ),
]);

const response = result.response;
const translatedText = response.text();

if (!translatedText) {
  throw new Error('Empty translation response from Gemini API');
}
```

**Response Example**:

```
__TAG_0__Bienvenido a __TAG_1__Press.Zone__TAG_2__!__TAG_3__
```

#### 7. HTML Restoration

```typescript
const finalTranslation = this.restoreHtmlTags(translatedText, tags);
```

**Final Output**:

```html
<p>Bienvenido a <strong>Press.Zone</strong>!</p>
```

#### 8. Token Counting (Output)

```typescript
const outputTokenCountResult = await this.model.countTokens(translatedText);
const outputTokens = outputTokenCountResult.totalTokens;
const totalTokens = inputTokens + outputTokens;
```

#### 9. Success Response

```typescript
const processingTime = Date.now() - startTime;

logger.info('Gemini translation successful', {
  inputTokens,
  outputTokens,
  totalTokens,
  processingTimeMs: processingTime,
  tagsRestored: tags.size,
});

return {
  translation: finalTranslation,
  tokens_used: totalTokens,
  processing_time_ms: processingTime,
  model,
};
```

**Response Type**:

```typescript
interface GeminiTranslationResponse {
  translation: string;
  tokens_used: number;
  processing_time_ms: number;
  model: Model;
}
```

---

## Integration Points

### 1. Translation Service

The primary consumer of `geminiClient`:

```typescript
// api/src/services/translationService.ts
import { geminiClient } from './geminiClient';

export class TranslationService {
  async translateSync(request: TranslationRequest): Promise<TranslationResponse> {
    // Validation, credit check, etc.
    
    // Call Gemini
    const result = await geminiClient.translate(
      request.content,
      request.source_language,
      request.target_language,
      request.model || Model.GEMINI_FLASH,
      request.tone || Tone.NEUTRAL
    );
    
    // Create job record, deduct credits, etc.
    return {
      job_id: job.id,
      translated_content: result.translation,
      tokens_used: result.tokens_used,
      processing_time_ms: result.processing_time_ms,
    };
  }
}
```

**See**: Skill 18 (Translation Service Orchestration) for full workflow.

### 2. Config System

The client depends on the config system for API key management:

```typescript
// api/src/config/index.ts
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();
const configCache = new Map<string, any>();
const configSubscribers = new Map<string, Set<() => void>>();

export function getGeminiConfig(): { apiKey: string } {
  const cached = configCache.get('gemini');
  if (cached) return cached;
  
  // Load from database
  const setting = await prisma.systemSetting.findUnique({
    where: { key: 'gemini_api_key' },
  });
  
  const config = { apiKey: setting?.value || process.env.GEMINI_API_KEY };
  configCache.set('gemini', config);
  
  return config;
}

export function onConfigChange(category: string, callback: () => void): void {
  if (!configSubscribers.has(category)) {
    configSubscribers.set(category, new Set());
  }
  configSubscribers.get(category)!.add(callback);
}

export function notifyConfigChange(category: string): void {
  const subscribers = configSubscribers.get(category);
  if (subscribers) {
    subscribers.forEach(callback => callback());
  }
  
  // Also notify 'all' subscribers
  const allSubscribers = configSubscribers.get('all');
  if (allSubscribers) {
    allSubscribers.forEach(callback => callback());
  }
}
```

**See**: Skill 16 (System Settings Management) for full config system.

### 3. Queue Worker

For async translations (long-running jobs):

```typescript
// api/src/worker.ts
import { Queue, Worker } from 'bullmq';
import { geminiClient } from './services/geminiClient';

const worker = new Worker('translation-queue', async (job) => {
  const { content, sourceLang, targetLang, model, tone, jobId } = job.data;
  
  try {
    // Call Gemini
    const result = await geminiClient.translate(
      content,
      sourceLang,
      targetLang,
      model,
      tone
    );
    
    // Update job in database
    await prisma.translationJob.update({
      where: { id: jobId },
      data: {
        status: 'completed',
        translated_content: result.translation,
        tokens_used: result.tokens_used,
        processing_time_ms: result.processing_time_ms,
        completed_at: new Date(),
      },
    });
    
    // Deduct credits
    await creditService.deductCredits(
      job.user_id,
      calculateCreditCost(result.tokens_used),
      `Translation job ${jobId}`,
      jobId
    );
    
    // Send webhook notification
    await webhookService.notifyJobComplete(jobId);
    
    return { success: true, tokens: result.tokens_used };
  } catch (error) {
    // Handle failure
    await prisma.translationJob.update({
      where: { id: jobId },
      data: {
        status: 'failed',
        error_message: error.message,
      },
    });
    
    throw error;
  }
});
```

**See**: Skill 8 (Job Queue Architecture) for full queue system.

---

## Performance Optimization

### 1. Singleton Pattern

```typescript
export const geminiClient = new GeminiClient();
```

**Benefits**:
- Single client instance shared across all requests
- Persistent HTTP connection pooling
- Reduced memory footprint
- Config changes affect all consumers instantly

### 2. Token Counting Efficiency

Token counting requires an API call. Optimize by:

```typescript
// BAD: Count twice for same text
const inputCount = await model.countTokens(prompt);
const outputCount = await model.countTokens(translatedText);

// GOOD: Only count unique texts
const inputCount = await model.countTokens(prompt);
const outputCount = await model.countTokens(translatedText);
// Prompt and translation are different, so 2 calls necessary
```

**Best Practice**: Cache token counts for frequently used system prompts.

```typescript
private systemPromptTokens: number = 150; // Pre-counted

async translate(...) {
  const contentTokens = await this.model.countTokens(cleanText);
  const estimatedInputTokens = contentTokens + this.systemPromptTokens;
  // ...
}
```

### 3. Concurrent Request Handling

The Gemini SDK handles connection pooling automatically. For high concurrency:

```typescript
// Process multiple translations in parallel
const results = await Promise.all([
  geminiClient.translate(content1, 'en', 'es', Model.GEMINI_FLASH),
  geminiClient.translate(content2, 'en', 'fr', Model.GEMINI_FLASH),
  geminiClient.translate(content3, 'en', 'de', Model.GEMINI_FLASH),
]);
```

**Limitation**: Google's rate limits apply (60 requests/minute for free tier, 600/min for paid).

### 4. Caching Strategy

For repeated translations (e.g., common UI strings):

```typescript
// In translationService.ts
const translationCache = new Map<string, GeminiTranslationResponse>();

function getCacheKey(content: string, sourceLang: string, targetLang: string, tone: Tone): string {
  return `${sourceLang}:${targetLang}:${tone}:${content}`;
}

async translateWithCache(content: string, sourceLang: string, targetLang: string, tone: Tone) {
  const key = getCacheKey(content, sourceLang, targetLang, tone);
  
  const cached = translationCache.get(key);
  if (cached) {
    logger.info('Cache hit for translation', { key });
    return cached;
  }
  
  const result = await geminiClient.translate(content, sourceLang, targetLang, Model.GEMINI_FLASH, tone);
  translationCache.set(key, result);
  
  return result;
}
```

**Caution**: Cache only short, static content. Never cache personalized or dynamic content.

---

## Error Scenarios & Troubleshooting

### Scenario 1: "Gemini client not initialized"

**Cause**: Config system failed to load Gemini API key.

**Check**:
1. Verify `GEMINI_API_KEY` in `.env`
2. Check database for `gemini_api_key` system setting
3. Review startup logs for initialization errors

**Fix**:

```bash
# Set via environment variable
echo "GEMINI_API_KEY=your_key_here" >> .env

# OR set via admin panel
curl -X PUT https://api.press.zone/admin/settings/gemini_api_key \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"value": "your_key_here"}'
```

### Scenario 2: "Translation request timed out"

**Cause**: Content too large for 60s timeout.

**Symptoms**:
- Large HTML documents (>10,000 characters)
- Complex nested tags
- Rare language pairs

**Fix**: Use async translation:

```typescript
// Instead of sync translation
const result = await translationService.translateSync(request);

// Use async translation
const job = await translationService.translateAsync(request);
// Poll job status or wait for webhook
```

### Scenario 3: "Gemini API rate limit exceeded"

**Cause**: Too many concurrent requests.

**Rate Limits**:
- **Free tier**: 60 requests/minute
- **Paid tier**: 600 requests/minute

**Fix**: Implement rate limiting at application level:

```typescript
// In rateLimiter.ts
const geminiLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 550, // Leave buffer below 600 limit
  keyGenerator: (req) => 'gemini-global',
  handler: (req, res) => {
    res.status(429).json({
      error: 'Too many translation requests. Please try again later.',
    });
  },
});

// In routes/translate.ts
router.post('/sync', geminiLimiter, translateSyncHandler);
```

### Scenario 4: HTML tags corrupted in translation

**Symptom**: Output contains malformed HTML like `<fuerte>` (Spanish for `<strong>`).

**Cause**: HTML extraction regex failed or Gemini ignored placeholder rules.

**Debug**:

```typescript
// Add debug logging
logger.debug('HTML extraction', {
  originalContent: content,
  cleanText,
  tags: Array.from(tags.entries()),
});

logger.debug('HTML restoration', {
  translatedText,
  finalTranslation,
});
```

**Fix**: Verify regex matches all tag types:

```typescript
// Test regex
const testContent = '<div class="foo" id="bar"><span>text</span></div>';
const { text, tags } = extractHtmlTags(testContent);

console.log(text);  // Should be: __TAG_0____TAG_1__text__TAG_2____TAG_3__
console.log(tags);  // Should map all 4 tags
```

### Scenario 5: Empty translation response

**Symptom**: `Empty translation response from Gemini API` error.

**Causes**:
1. Gemini refused to translate (content policy violation)
2. Model returned structured output instead of plain text
3. Network error truncated response

**Debug**:

```typescript
const response = result.response;
console.log('Full response:', JSON.stringify(response, null, 2));

const translatedText = response.text();
console.log('Extracted text:', translatedText);
```

**Fix**: Check Gemini API response structure and content policy.

---

## Testing

### Unit Tests

```typescript
// __tests__/unit/services/geminiClient.test.ts
import { geminiClient } from '../../../src/services/geminiClient';
import { Model, Tone } from '../../../src/types';

describe('GeminiClient', () => {
  describe('HTML Tag Preservation', () => {
    it('should preserve HTML tags in translation', async () => {
      const content = '<p>Hello <strong>world</strong>!</p>';
      
      const result = await geminiClient.translate(
        content,
        'en',
        'es',
        Model.GEMINI_FLASH
      );
      
      // Check that HTML structure is preserved
      expect(result.translation).toMatch(/<p>.*<strong>.*<\/strong>.*<\/p>/);
      // Check that content is translated
      expect(result.translation).not.toContain('Hello');
      expect(result.translation).toMatch(/Hola|Bienvenido/);
    });
    
    it('should handle nested HTML tags', async () => {
      const content = '<div><p>Outer <span class="inner">Inner</span> text</p></div>';
      
      const result = await geminiClient.translate(
        content,
        'en',
        'fr',
        Model.GEMINI_FLASH
      );
      
      expect(result.translation).toContain('<div>');
      expect(result.translation).toContain('<span class="inner">');
      expect(result.translation).toContain('</div>');
    });
  });
  
  describe('Token Counting', () => {
    it('should return accurate token counts', async () => {
      const content = 'This is a test sentence.';
      
      const result = await geminiClient.translate(
        content,
        'en',
        'es',
        Model.GEMINI_FLASH
      );
      
      expect(result.tokens_used).toBeGreaterThan(0);
      expect(result.tokens_used).toBeLessThan(500); // Reasonable upper bound
    });
  });
  
  describe('Error Handling', () => {
    it('should throw timeout error for slow requests', async () => {
      const largeContent = 'A'.repeat(50000); // Very large content
      
      await expect(
        geminiClient.translate(largeContent, 'en', 'es', Model.GEMINI_FLASH)
      ).rejects.toThrow('timed out');
    }, 70000); // Test timeout > client timeout
    
    it('should handle invalid API key', async () => {
      // Temporarily break API key
      process.env.GEMINI_API_KEY = 'invalid_key';
      geminiClient.refreshConfig();
      
      await expect(
        geminiClient.translate('Test', 'en', 'es', Model.GEMINI_FLASH)
      ).rejects.toThrow('authentication failed');
      
      // Restore
      process.env.GEMINI_API_KEY = originalKey;
      geminiClient.refreshConfig();
    });
  });
  
  describe('Tone Customization', () => {
    it('should apply formal tone', async () => {
      const content = 'You need to finish this.';
      
      const result = await geminiClient.translate(
        content,
        'en',
        'es',
        Model.GEMINI_FLASH,
        Tone.FORMAL
      );
      
      // Formal Spanish uses "usted" form
      expect(result.translation).toMatch(/debe|necesita/i);
    });
    
    it('should apply casual tone', async () => {
      const content = 'You need to finish this.';
      
      const result = await geminiClient.translate(
        content,
        'en',
        'es',
        Model.GEMINI_FLASH,
        Tone.CASUAL
      );
      
      // Casual Spanish uses "tú" form
      expect(result.translation).toMatch(/tienes|necesitas/i);
    });
  });
});
```

### Integration Tests

```typescript
// __tests__/integration/geminiClient.test.ts
describe('Gemini Integration', () => {
  it('should translate real WordPress content', async () => {
    const wordpressPost = `
      <h1>Welcome to Press.Zone</h1>
      <p>This is a <strong>powerful</strong> translation service.</p>
      <ul>
        <li>Fast translations</li>
        <li>HTML preservation</li>
      </ul>
    `;
    
    const result = await geminiClient.translate(
      wordpressPost,
      'en',
      'de',
      Model.GEMINI_FLASH
    );
    
    expect(result.translation).toContain('<h1>');
    expect(result.translation).toContain('<strong>');
    expect(result.translation).toContain('<ul>');
    expect(result.translation).not.toContain('Welcome to Press.Zone');
    expect(result.tokens_used).toBeGreaterThan(0);
  });
  
  it('should handle config refresh', async () => {
    // Initial translation
    const result1 = await geminiClient.translate('Test', 'en', 'es', Model.GEMINI_FLASH);
    expect(result1.translation).toBeDefined();
    
    // Simulate config change
    await settingsService.updateSetting('gemini_api_key', newApiKey);
    
    // Config should auto-refresh
    await new Promise(resolve => setTimeout(resolve, 100));
    
    // New translation should work with new key
    const result2 = await geminiClient.translate('Test', 'en', 'es', Model.GEMINI_FLASH);
    expect(result2.translation).toBeDefined();
  });
});
```

### Health Check Test

```typescript
describe('Health Check', () => {
  it('should return true when Gemini is available', async () => {
    const healthy = await geminiClient.healthCheck();
    expect(healthy).toBe(true);
  });
  
  it('should return false when API key is invalid', async () => {
    process.env.GEMINI_API_KEY = 'invalid';
    geminiClient.refreshConfig();
    
    const healthy = await geminiClient.healthCheck();
    expect(healthy).toBe(false);
    
    // Restore
    process.env.GEMINI_API_KEY = validKey;
    geminiClient.refreshConfig();
  });
});
```

---

## Monitoring & Observability

### Structured Logging

```typescript
// Translation start
logger.info('Calling Gemini translation API', {
  sourceLang,
  targetLang,
  model: 'gemini-3-flash-preview',
  tone,
  contentLength: content.length,
  cleanTextLength: cleanText.length,
  tagCount: tags.size,
});

// Translation success
logger.info('Gemini translation successful', {
  inputTokens,
  outputTokens,
  totalTokens,
  processingTimeMs: processingTime,
  tagsRestored: tags.size,
});

// Translation failure
logger.error('Gemini translation failed', {
  error: errorMessage,
  processingTime,
});
```

**Query Examples** (using log aggregation tool):

```
# Find slow translations
level="info" message="Gemini translation successful" processingTimeMs > 5000

# Find rate limit errors
level="error" message="Gemini translation failed" error~"rate limit"

# Average tokens per translation
avg(totalTokens) WHERE message="Gemini translation successful"
```

### Prometheus Metrics

```typescript
// In utils/metrics.ts
export const geminiTranslationDuration = new Histogram({
  name: 'gemini_translation_duration_seconds',
  help: 'Duration of Gemini translation requests',
  labelNames: ['status', 'source_lang', 'target_lang'],
  buckets: [0.5, 1, 2, 5, 10, 30, 60],
});

export const geminiTokensUsed = new Counter({
  name: 'gemini_tokens_used_total',
  help: 'Total tokens consumed by Gemini API',
  labelNames: ['source_lang', 'target_lang'],
});

export const geminiErrors = new Counter({
  name: 'gemini_errors_total',
  help: 'Total Gemini API errors',
  labelNames: ['error_type'],
});

// In geminiClient.ts
import { geminiTranslationDuration, geminiTokensUsed, geminiErrors } from '../utils/metrics';

async translate(...) {
  const timer = geminiTranslationDuration.startTimer({
    source_lang: sourceLang,
    target_lang: targetLang,
  });
  
  try {
    // ... translation logic
    
    timer({ status: 'success' });
    
    geminiTokensUsed.inc({
      source_lang: sourceLang,
      target_lang: targetLang,
    }, totalTokens);
    
    return result;
  } catch (error) {
    timer({ status: 'error' });
    
    geminiErrors.inc({
      error_type: getErrorType(error),
    });
    
    throw error;
  }
}
```

**Grafana Queries**:

```promql
# Translation success rate
rate(gemini_translation_duration_seconds_count{status="success"}[5m])
/ 
rate(gemini_translation_duration_seconds_count[5m])

# P95 latency
histogram_quantile(0.95, rate(gemini_translation_duration_seconds_bucket[5m]))

# Token consumption rate
rate(gemini_tokens_used_total[5m])

# Error rate by type
rate(gemini_errors_total[5m])
```

---

## Configuration Reference

### Environment Variables

```bash
# Required
GEMINI_API_KEY=your_gemini_api_key_here

# Optional (defaults shown)
GEMINI_TIMEOUT_MS=60000           # Translation timeout
GEMINI_MODEL=gemini-3-flash-preview   # Model name
```

### Database Settings

The client uses the `SystemSetting` table for dynamic configuration:

```typescript
// Admin updates via API
PUT /admin/settings/gemini_api_key
{
  "value": "new_api_key_here"
}

// Database structure
{
  key: 'gemini_api_key',
  value: 'AIzaSy...',
  category: 'gemini',
  type: 'string',
  description: 'Google Gemini API key for translation',
  is_secret: true,
}
```

**Security**: Secret values are encrypted at rest (see Skill 12).

---

## Future Enhancements

### 1. Multi-Model Support

Currently hardcoded to `gemini-3-flash-preview`. To support multiple models:

```typescript
enum GeminiModel {
  FLASH = 'gemini-3-flash-preview',
  PRO = 'gemini-1.5-pro',
  PRO_VISION = 'gemini-pro-vision',
}

private getModelInstance(model: GeminiModel): GenerativeModel {
  return this.client.getGenerativeModel({ model });
}

async translate(..., model: GeminiModel) {
  const modelInstance = this.getModelInstance(model);
  // ... rest of translation logic with modelInstance
}
```

**Use Cases**:
- **Flash**: Fast, cost-effective translations (current)
- **Pro**: Higher quality for professional documents
- **Pro Vision**: Translate text in images (screenshots, PDFs)

### 2. Streaming Translations

For very long content, stream translation in chunks:

```typescript
async *translateStream(content: string, sourceLang: string, targetLang: string) {
  const chunks = splitIntoChunks(content, 5000); // 5000 char chunks
  
  for (const chunk of chunks) {
    const result = await this.translate(chunk, sourceLang, targetLang, Model.GEMINI_FLASH);
    yield result.translation;
  }
}

// Usage
for await (const chunk of geminiClient.translateStream(largeContent, 'en', 'es')) {
  console.log(chunk); // Process each chunk as it arrives
}
```

### 3. Context-Aware Translation

Maintain conversation context across multiple requests:

```typescript
async translateWithContext(
  content: string,
  sourceLang: string,
  targetLang: string,
  context: string[] // Previous sentences
): Promise<GeminiTranslationResponse> {
  const contextPrompt = context.length > 0
    ? `Context from previous translations:\n${context.join('\n')}\n\n`
    : '';
  
  const prompt = `${contextPrompt}Translate the following...`;
  // ... rest of translation logic
}
```

**Use Case**: Maintain consistent terminology across multi-page documents.

### 4. Custom Glossaries

Allow users to define translation rules:

```typescript
interface GlossaryEntry {
  term: string;
  translation: string;
  caseSensitive: boolean;
}

async translateWithGlossary(
  content: string,
  sourceLang: string,
  targetLang: string,
  glossary: GlossaryEntry[]
): Promise<GeminiTranslationResponse> {
  const glossaryPrompt = glossary.length > 0
    ? `Use these specific translations:\n${glossary.map(e => `"${e.term}" → "${e.translation}"`).join('\n')}\n\n`
    : '';
  
  const prompt = `${glossaryPrompt}Translate the following...`;
  // ... rest of translation logic
}
```

**Use Case**: Brand names, technical terms, product names should not be translated.

### 5. Batch Translation

Translate multiple pieces of content in a single API call:

```typescript
async translateBatch(
  items: Array<{ content: string; sourceLang: string; targetLang: string }>,
  model: Model
): Promise<GeminiTranslationResponse[]> {
  const batchPrompt = items.map((item, i) => 
    `[${i}] Translate from ${item.sourceLang} to ${item.targetLang}:\n${item.content}`
  ).join('\n\n---\n\n');
  
  const result = await this.model.generateContent(batchPrompt);
  
  // Parse response and split into individual translations
  const translations = parseIndexedResponse(result.response.text());
  
  return translations.map((translation, i) => ({
    translation,
    tokens_used: result.tokens_used / items.length, // Approximate
    processing_time_ms: result.processing_time_ms,
    model,
  }));
}
```

**Use Case**: Translate all strings in a UI file at once.

---

## Quick Reference

### Function Signature

```typescript
geminiClient.translate(
  content: string,
  sourceLang: string,
  targetLang: string,
  model: Model,
  tone?: Tone
): Promise<GeminiTranslationResponse>
```

### Parameters

| Parameter | Type | Description | Example |
|-----------|------|-------------|---------|
| `content` | string | Content to translate (may include HTML) | `"<p>Hello world</p>"` |
| `sourceLang` | string | Source language ISO 639-1 code | `"en"` |
| `targetLang` | string | Target language ISO 639-1 code | `"es"` |
| `model` | Model | ML model (currently ignored, always uses flash) | `Model.GEMINI_FLASH` |
| `tone` | Tone | Translation tone (optional, defaults to neutral) | `Tone.FORMAL` |

### Return Value

```typescript
interface GeminiTranslationResponse {
  translation: string;           // Translated content with HTML restored
  tokens_used: number;            // Total tokens (input + output)
  processing_time_ms: number;     // API call duration
  model: Model;                   // Model used (echo)
}
```

### Example Usage

```typescript
import { geminiClient } from './services/geminiClient';
import { Model, Tone } from './types';

// Basic translation
const result = await geminiClient.translate(
  '<p>Welcome to <strong>Press.Zone</strong>!</p>',
  'en',
  'es',
  Model.GEMINI_FLASH
);

console.log(result.translation);
// Output: "<p>¡Bienvenido a <strong>Press.Zone</strong>!</p>"

console.log(result.tokens_used);
// Output: 245

// Formal tone translation
const formal = await geminiClient.translate(
  'You need to complete the form.',
  'en',
  'es',
  Model.GEMINI_FLASH,
  Tone.FORMAL
);

console.log(formal.translation);
// Output: "Debe completar el formulario."

// Casual tone translation
const casual = await geminiClient.translate(
  'You need to complete the form.',
  'en',
  'es',
  Model.GEMINI_FLASH,
  Tone.CASUAL
);

console.log(casual.translation);
// Output: "Tienes que completar el formulario."
```

---

## Validation Checklist

Before deploying Gemini client changes:

- [ ] Client initializes successfully on startup
- [ ] Config refresh triggered when admin updates settings
- [ ] HTML tags preserved in translation (all types: opening, closing, self-closing)
- [ ] Token counting accurate for input and output
- [ ] Timeout protection prevents hanging requests
- [ ] Error messages user-friendly and actionable
- [ ] Health check returns accurate status
- [ ] Structured logging includes all relevant context
- [ ] Prometheus metrics tracked for translations, tokens, errors
- [ ] Supported languages include all 35 in language map
- [ ] Tone customization applies correctly (formal/casual/neutral)
- [ ] Empty responses handled gracefully
- [ ] Rate limit errors caught and reported
- [ ] Authentication errors surface immediately
- [ ] Model unavailable errors retry with exponential backoff
- [ ] Unit tests cover HTML preservation, token counting, error handling
- [ ] Integration tests verify end-to-end translation flow
- [ ] Load tests confirm throughput meets requirements (>100 translations/min)
- [ ] Documentation updated with any new features or changes


---

# Skill 20: PayPal Subscription Management

## Overview

The PayPal Subscription Management system handles the complete subscription lifecycle for translate.press.zone's SaaS monetization. It integrates with PayPal's Subscriptions API to create checkout sessions, process webhook events, manage subscription states, and coordinate with the credit management system to allocate translation credits.

### Key Capabilities

- **Subscription Creation**: Generate PayPal checkout sessions for three-tier plans (starter, professional, enterprise)
- **Billing Cycles**: Support monthly and annual billing with automatic renewal
- **Webhook Processing**: Handle 6 subscription lifecycle events with cryptographic signature verification
- **Credit Allocation**: Automatically allocate credits on activation and renewal
- **Status Synchronization**: Keep local database in sync with PayPal's subscription state
- **OAuth Authentication**: Generate PayPal access tokens for API requests
- **Payment Recording**: Track completed payments with transaction history

### Files

- **api/src/services/paypalService.ts** (647 lines): Core PayPal integration with subscription creation, webhook verification, and event handlers
- **api/src/routes/webhooks.ts** (462 lines): Webhook endpoint routing and event processing logic
- **api/src/utils/paypalCert.ts**: Certificate-based webhook signature verification (referenced)
- **api/src/config/index.ts**: PayPal configuration (client ID, secret, plan IDs, webhook ID)

### Database Models

```prisma
model Subscription {
  id                      String             @id @default(uuid())
  user_id                 String             @unique
  plan_tier               PlanTier           // starter, professional, enterprise
  billing_cycle           BillingCycle       // monthly, annual
  status                  SubscriptionStatus // active, cancelled, suspended
  paypal_subscription_id  String?            @unique
  current_period_start    DateTime
  current_period_end      DateTime
  cancel_at_period_end    Boolean            @default(false)
  created_at              DateTime           @default(now())
  updated_at              DateTime           @updatedAt

  user                    User               @relation(fields: [user_id], references: [id])
  payments                Payment[]

  @@index([user_id])
  @@index([paypal_subscription_id])
}

model Payment {
  id                  String        @id @default(uuid())
  user_id             String
  paypal_payment_id   String?       @unique
  amount              Decimal       @db.Decimal(10, 2)
  currency            String        @default("USD")
  status              PaymentStatus // pending, completed, failed, refunded
  type                PaymentType   // subscription_payment, one_time_purchase
  subscription_id     String?
  created_at          DateTime      @default(now())
  updated_at          DateTime      @updatedAt

  user                User          @relation(fields: [user_id], references: [id])
  subscription        Subscription? @relation(fields: [subscription_id], references: [id])
  credit_transactions CreditTransaction[]

  @@index([user_id])
  @@index([paypal_payment_id])
  @@index([subscription_id])
}

enum PlanTier {
  starter
  professional
  enterprise
}

enum BillingCycle {
  monthly
  annual
}

enum SubscriptionStatus {
  active
  cancelled
  suspended
  past_due
}

enum PaymentStatus {
  pending
  completed
  failed
  refunded
}

enum PaymentType {
  subscription_payment
  one_time_purchase
}
```

## Architecture

### Subscription Flow

```
┌─────────────┐    1. Create subscription    ┌──────────────┐
│   User      │────────────────────────────>│   API Server │
│  (Plugin)   │                               │              │
└─────────────┘                               └──────┬───────┘
                                                     │
                                                     │ 2. POST /v1/subscriptions
                                                     │    plan_id, subscriber email
                                                     ▼
                                              ┌──────────────┐
                                              │   PayPal API │
                                              │              │
                                              └──────┬───────┘
                                                     │
                                                     │ 3. Return approval_url
                                                     ▼
┌─────────────┐    4. Redirect to PayPal     ┌──────────────┐
│    User     │◄────────────────────────────│   API Server │
│  (Browser)  │                               │              │
└─────────────┘                               └──────────────┘
      │
      │ 5. User completes checkout
      ▼
┌──────────────┐
│  PayPal Site │
│              │
└──────┬───────┘
       │
       │ 6. BILLING.SUBSCRIPTION.ACTIVATED webhook
       ▼
┌──────────────┐    7. Verify signature      ┌──────────────┐
│   API Server │────────────────────────────>│   PayPal API │
│   (Webhook)  │◄────────────────────────────│   (Cert URL) │
└──────┬───────┘    8. Signature valid       └──────────────┘
       │
       │ 9. Update Subscription table
       │ 10. Allocate credits via allocateCredits()
       ▼
┌──────────────┐
│   Database   │
│              │
└──────────────┘
```

### Webhook Event Handling

```
┌──────────────┐         Webhook Event         ┌──────────────┐
│   PayPal     │──────────────────────────────>│   Nginx      │
│              │   POST /v1/webhooks/paypal     │   Proxy      │
└──────────────┘   Headers + JSON payload       └──────┬───────┘
                                                       │
                                                       │ Route to API
                                                       ▼
                                                ┌──────────────┐
                                                │   Express    │
                                                │   Middleware │
                                                └──────┬───────┘
                                                       │
                                                       │ 1. Parse JSON body
                                                       │ 2. Extract headers
                                                       ▼
                                                ┌──────────────────────┐
                                                │ verifyWebhookSignature()│
                                                └──────┬───────────────┘
                                                       │
                   ┌───────────────────────────────────┼───────────────────────────────────┐
                   │                                   │                                   │
                   │ INVALID                           │ VALID                             │
                   ▼                                   ▼                                   │
            ┌──────────────┐              ┌────────────────────────┐                     │
            │  Return 401  │              │ Route by event_type    │                     │
            │  Unauthorized│              │                        │                     │
            └──────────────┘              └────────┬───────────────┘                     │
                                                   │                                     │
                 ┌─────────────────────────────────┼─────────────────────────────────┐   │
                 │                                 │                                 │   │
                 ▼                                 ▼                                 ▼   │
    ┌─────────────────────────┐    ┌─────────────────────────┐    ┌─────────────────────────┐
    │ SUBSCRIPTION.ACTIVATED  │    │ PAYMENT.SALE.COMPLETED  │    │ SUBSCRIPTION.CANCELLED  │
    │                         │    │                         │    │                         │
    │ handleSubscriptionActivated│ │ handlePaymentCompleted │    │ handleSubscriptionCancelled│
    └────────┬────────────────┘    └────────┬────────────────┘    └────────┬────────────────┘
             │                              │                              │
             │ 1. Find user by email        │ 1. Find subscription         │ 1. Find subscription
             │ 2. Parse plan ID             │ 2. Create Payment record     │ 2. Set status=cancelled
             │ 3. Upsert Subscription       │ 3. Update period dates       │ 3. Set cancel_at_period_end
             │ 4. Allocate credits          │ 4. Allocate renewal credits  │
             ▼                              ▼                              ▼
      ┌──────────────┐              ┌──────────────┐              ┌──────────────┐
      │   Database   │              │   Database   │              │   Database   │
      │              │              │              │              │              │
      └──────────────┘              └──────────────┘              └──────────────┘
```

## Core Functions

### 1. Subscription Creation

#### `createSubscription()`

Creates a new PayPal subscription checkout session.

```typescript
export async function createSubscription(
  userId: string,
  planTier: SubscriptionPlan,
  billingCycle: BillingCycleType
): Promise<{ approvalUrl: string; subscriptionId: string }>
```

**Implementation** (paypalService.ts:49-125):

```typescript
export async function createSubscription(
  userId: string,
  planTier: SubscriptionPlan,
  billingCycle: BillingCycleType
): Promise<{ approvalUrl: string; subscriptionId: string }> {
  try {
    // Get user details
    const user = await prisma.user.findUnique({
      where: { id: userId },
      select: { email: true },
    });

    if (!user) {
      throw new Error('User not found');
    }

    // Get PayPal plan ID
    const planId = getPayPalPlanId(planTier, billingCycle);

    if (!planId) {
      throw new Error(`Invalid plan tier or billing cycle: ${planTier}/${billingCycle}`);
    }

    // Create subscription request
    const request = new paypal.subscriptions.SubscriptionsCreateRequest();
    request.requestBody({
      plan_id: planId,
      subscriber: {
        email_address: user.email,
      },
      application_context: {
        brand_name: 'translate.press.zone',
        locale: 'en-US',
        shipping_preference: 'NO_SHIPPING',
        user_action: 'SUBSCRIBE_NOW',
        return_url: `${config.frontendUrl}/subscription/success`,
        cancel_url: `${config.frontendUrl}/subscription/cancel`,
      },
      custom_id: userId, // Store user ID for webhook processing
    });

    // Execute request
    const client = getPayPalClient();
    const response = await client.execute(request);

    // Extract approval URL
    const approvalLink = response.result.links?.find((link: any) => link.rel === 'approve');

    if (!approvalLink) {
      throw new Error('No approval URL returned from PayPal');
    }

    logger.info('PayPal subscription created', {
      userId,
      planTier,
      billingCycle,
      subscriptionId: response.result.id,
    });

    // Track metrics
    subscriptionEventsTotal.inc({ event_type: 'subscription_created' });

    return {
      approvalUrl: approvalLink.href,
      subscriptionId: response.result.id,
    };
  } catch (error) {
    logger.error('Failed to create PayPal subscription', {
      userId,
      planTier,
      billingCycle,
      error,
    });
    trackError('paypal', 'subscription_creation_failed');
    throw error;
  }
}
```

**Key Points**:

- **Plan ID Mapping**: Uses `getPayPalPlanId()` to map tier + cycle to PayPal plan ID
- **Custom ID**: Embeds `userId` in `custom_id` field for webhook correlation
- **Approval URL**: Returns HATEOAS link for user redirect to PayPal checkout
- **Return URLs**: Configures success/cancel redirect URLs
- **Brand Name**: Displays "translate.press.zone" during checkout
- **Metrics**: Tracks subscription creation events
- **Error Handling**: Logs failures and throws for upper-layer retry logic

### 2. Webhook Signature Verification

#### `verifyWebhookSignature()`

Validates PayPal webhook signatures using certificate-based verification.

```typescript
export async function verifyWebhookSignature(
  payload: string,
  headers: Record<string, string>
): Promise<boolean>
```

**Implementation** (paypalService.ts:134-180):

```typescript
export async function verifyWebhookSignature(
  payload: string,
  headers: Record<string, string>
): Promise<boolean> {
  try {
    const transmissionId = headers['paypal-transmission-id'];
    const transmissionTime = headers['paypal-transmission-time'];
    const certUrl = headers['paypal-cert-url'];
    const authAlgo = headers['paypal-auth-algo'];
    const transmissionSig = headers['paypal-transmission-sig'];

    if (!transmissionId || !transmissionTime || !certUrl || !authAlgo || !transmissionSig) {
      logger.warn('Missing PayPal webhook signature headers');
      return false;
    }

    // Create verification request
    const request = new paypal.notifications.WebhooksVerifySignatureRequest();
    request.requestBody({
      transmission_id: transmissionId,
      transmission_time: transmissionTime,
      cert_url: certUrl,
      auth_algo: authAlgo,
      transmission_sig: transmissionSig,
      webhook_id: config.paypalWebhookId,
      webhook_event: JSON.parse(payload),
    });

    // Verify signature
    const client = getPayPalClient();
    const response = await client.execute(request);

    const isValid = response.result.verification_status === 'SUCCESS';

    if (!isValid) {
      logger.warn('PayPal webhook signature verification failed', {
        verificationStatus: response.result.verification_status,
      });
    }

    return isValid;
  } catch (error) {
    logger.error('Failed to verify PayPal webhook signature', { error });
    trackError('paypal', 'webhook_verification_failed');
    return false;
  }
}
```

**Security Headers**:

- `paypal-transmission-id`: Unique ID for webhook event
- `paypal-transmission-time`: Timestamp to prevent replay attacks
- `paypal-cert-url`: URL to PayPal's public certificate
- `paypal-auth-algo`: Algorithm used for signature (SHA256withRSA)
- `paypal-transmission-sig`: Base64-encoded signature

**Verification Process**:

1. Extract all 5 required headers
2. Send verification request to PayPal API
3. PayPal validates signature using its private key
4. Return `true` if `verification_status === 'SUCCESS'`
5. Log warnings for invalid signatures (potential security breach)

**Alternative Verification** (webhooks.ts):

The webhook route also uses a local verification utility:

```typescript
// webhooks.ts:27-47
async function verifyWebhookSignature(
  headers: Record<string, string | string[] | undefined>,
  body: string
): Promise<boolean> {
  try {
    // Get PayPal webhook ID from configuration
    const paypalConfig = getPayPalConfig();
    const webhookId = paypalConfig.webhookId;

    if (!webhookId) {
      logger.error('PayPal webhook ID not configured');
      return false;
    }

    // Delegate to the full signature verification utility
    return await verifyPayPalWebhookSignature(headers, body, webhookId);
  } catch (error) {
    logger.error('PayPal webhook signature verification failed', { error });
    return false;
  }
}
```

This delegates to `verifyPayPalWebhookSignature()` in `utils/paypalCert.ts`, which performs certificate-based cryptographic verification locally.

### 3. Webhook Event Handling

#### `handleWebhookEvent()`

Orchestrates webhook event processing with signature verification and routing.

```typescript
export async function handleWebhookEvent(
  payload: PayPalWebhookEvent,
  headers: Record<string, string>
): Promise<void>
```

**Implementation** (paypalService.ts:188-253):

```typescript
export async function handleWebhookEvent(
  payload: PayPalWebhookEvent,
  headers: Record<string, string>
): Promise<void> {
  try {
    // Verify webhook signature
    const isValid = await verifyWebhookSignature(JSON.stringify(payload), headers);

    if (!isValid) {
      logger.warn('Rejecting PayPal webhook with invalid signature', {
        eventId: payload.id,
        eventType: payload.event_type,
      });
      throw new Error('Invalid webhook signature');
    }

    logger.info('Processing PayPal webhook event', {
      eventId: payload.id,
      eventType: payload.event_type,
      subscriptionId: payload.resource.id,
    });

    // Track metrics
    subscriptionEventsTotal.inc({ event_type: payload.event_type });

    // Handle different event types
    switch (payload.event_type) {
      case 'BILLING.SUBSCRIPTION.ACTIVATED':
        await handleSubscriptionActivated(payload);
        break;

      case 'PAYMENT.SALE.COMPLETED':
        await handlePaymentCompleted(payload);
        break;

      case 'BILLING.SUBSCRIPTION.CANCELLED':
        await handleSubscriptionCancelled(payload);
        break;

      case 'BILLING.SUBSCRIPTION.SUSPENDED':
        await handleSubscriptionSuspended(payload);
        break;

      case 'BILLING.SUBSCRIPTION.EXPIRED':
        await handleSubscriptionExpired(payload);
        break;

      case 'BILLING.SUBSCRIPTION.UPDATED':
        await handleSubscriptionUpdated(payload);
        break;

      default:
        logger.info('Unhandled PayPal webhook event type', {
          eventType: payload.event_type,
        });
    }
  } catch (error) {
    logger.error('Failed to handle PayPal webhook event', {
      eventId: payload.id,
      eventType: payload.event_type,
      error,
    });
    trackError('paypal', 'webhook_processing_failed');
    throw error;
  }
}
```

**Event Types Handled**:

| Event Type | Handler | Description |
|------------|---------|-------------|
| `BILLING.SUBSCRIPTION.ACTIVATED` | `handleSubscriptionActivated()` | New subscription activated after checkout |
| `PAYMENT.SALE.COMPLETED` | `handlePaymentCompleted()` | Recurring payment processed successfully |
| `BILLING.SUBSCRIPTION.CANCELLED` | `handleSubscriptionCancelled()` | User or admin cancelled subscription |
| `BILLING.SUBSCRIPTION.SUSPENDED` | `handleSubscriptionSuspended()` | Payment failed, subscription paused |
| `BILLING.SUBSCRIPTION.EXPIRED` | `handleSubscriptionExpired()` | Subscription reached end date |
| `BILLING.SUBSCRIPTION.UPDATED` | `handleSubscriptionUpdated()` | Plan tier or billing cycle changed |

## Event Handlers

### 1. Subscription Activated

Handles new subscription activation after user completes PayPal checkout.

```typescript
async function handleSubscriptionActivated(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:258-330):

```typescript
async function handleSubscriptionActivated(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const subscriptionId = payload.resource.id;
    const userId = payload.resource.subscriber?.payer_id || (payload as any).custom_id;

    if (!userId) {
      throw new Error('User ID not found in webhook payload');
    }

    // Extract plan information
    const planId = payload.resource.plan_id;
    if (!planId) {
      throw new Error('Missing plan_id in subscription data');
    }
    const { planTier, billingCycle } = parsePlanId(planId);

    // Calculate period dates
    const currentPeriodStart = new Date();
    const currentPeriodEnd = new Date();
    if (billingCycle === BillingCycle.monthly) {
      currentPeriodEnd.setMonth(currentPeriodEnd.getMonth() + 1);
    } else {
      currentPeriodEnd.setFullYear(currentPeriodEnd.getFullYear() + 1);
    }

    // Create or update subscription in database
    await prisma.subscription.upsert({
      where: { user_id: userId },
      create: {
        user_id: userId,
        plan_tier: planTier,
        billing_cycle: billingCycle,
        status: SubscriptionStatus.active,
        paypal_subscription_id: subscriptionId,
        current_period_start: currentPeriodStart,
        current_period_end: currentPeriodEnd,
        cancel_at_period_end: false,
      },
      update: {
        plan_tier: planTier,
        billing_cycle: billingCycle,
        status: SubscriptionStatus.active,
        paypal_subscription_id: subscriptionId,
        current_period_start: currentPeriodStart,
        current_period_end: currentPeriodEnd,
        cancel_at_period_end: false,
      },
    });

    // Allocate credits for the subscription tier
    const creditAmount = getCreditAllocation(mapPlanTier(planTier));
    await allocateCredits(
      userId,
      creditAmount,
      `Subscription activated: ${planTier} (${billingCycle})`,
      undefined
    );

    logger.info('Subscription activated', {
      userId,
      subscriptionId,
      planTier,
      billingCycle,
      creditsAllocated: creditAmount,
    });
  } catch (error) {
    logger.error('Failed to handle subscription activation', {
      subscriptionId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Alternative Implementation** (webhooks.ts:160-263):

The webhook route also contains a parallel implementation with atomic transactions:

```typescript
async function handleSubscriptionActivated(event: PayPalWebhookEvent): Promise<void> {
  const subscriptionId = event.resource.id;
  const planId = event.resource.plan_id;
  const email = event.resource.subscriber?.email_address;

  if (!subscriptionId || !planId || !email) {
    logger.warn('Missing required fields in subscription activation event', {
      subscriptionId,
      planId,
      email,
    });
    return;
  }

  // Find user by email
  const user = await prisma.user.findUnique({
    where: { email },
  });

  if (!user) {
    logger.warn('User not found for subscription activation', { email });
    return;
  }

  // Determine plan tier and billing cycle from plan ID
  const { tier, cycle } = parsePlanId(planId);

  if (!tier || !cycle) {
    logger.warn('Unable to parse plan ID', { planId });
    return;
  }

  // Atomic transaction: activate subscription + allocate credits
  await prisma.$transaction(async (tx) => {
    // 1. Update or create subscription
    await tx.subscription.upsert({
      where: { user_id: user.id },
      update: {
        plan_tier: tier,
        billing_cycle: cycle,
        status: 'active',
        paypal_subscription_id: subscriptionId,
        current_period_start: new Date(),
        current_period_end: calculatePeriodEnd(cycle),
        cancel_at_period_end: false,
      },
      create: {
        user_id: user.id,
        plan_tier: tier,
        billing_cycle: cycle,
        status: 'active',
        paypal_subscription_id: subscriptionId,
        current_period_start: new Date(),
        current_period_end: calculatePeriodEnd(cycle),
      },
    });

    // 2. Get current balance from latest transaction
    const latestTransaction = await tx.creditTransaction.findFirst({
      where: { user_id: user.id },
      orderBy: { created_at: 'desc' },
      select: { balance_after: true },
    });

    const currentBalance = latestTransaction?.balance_after ?? 0;

    // 3. Calculate new balance
    const creditAllocation = getCreditAllocation(tier);
    const newBalance = currentBalance + creditAllocation;

    // 4. Create credit transaction
    await tx.creditTransaction.create({
      data: {
        user_id: user.id,
        type: 'allocation',
        amount: creditAllocation,
        balance_after: newBalance,
        description: `Monthly credit allocation for ${tier} plan`,
      },
    });

    // 5. Update user timestamp
    await tx.user.update({
      where: { id: user.id },
      data: {
        updated_at: new Date(),
      },
    });
  });

  logger.info('Subscription activated successfully', {
    userId: user.id,
    subscriptionId,
    tier,
    cycle,
    creditsAllocated: getCreditAllocation(tier),
  });
}
```

**Key Differences**:

- **Atomicity**: webhooks.ts uses `prisma.$transaction()` to ensure all-or-nothing execution
- **User Lookup**: webhooks.ts finds user by email (from PayPal subscriber data)
- **Balance Calculation**: webhooks.ts manually computes `balance_after` inline
- **Credit Allocation**: paypalService.ts delegates to `allocateCredits()` function

**Credit Allocations**:

```typescript
// config/index.ts
export const getCreditAllocation = (tier: 'starter' | 'professional' | 'enterprise'): number => {
  return {
    starter: 100000,       // 100K credits/month
    professional: 500000,   // 500K credits/month
    enterprise: 2000000,    // 2M credits/month
  }[tier];
};
```

### 2. Payment Completed

Handles recurring payment success and allocates renewal credits.

```typescript
async function handlePaymentCompleted(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:335-414):

```typescript
async function handlePaymentCompleted(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const paymentId = payload.resource.id;
    const subscriptionId = (payload.resource as any).billing_agreement_id;
    const amount = parseFloat((payload.resource as any).amount?.total || '0');
    const currency = (payload.resource as any).amount?.currency || 'USD';

    // Find subscription
    const subscription = await prisma.subscription.findUnique({
      where: { paypal_subscription_id: subscriptionId },
    });

    if (!subscription) {
      logger.warn('Subscription not found for payment', { subscriptionId, paymentId });
      return;
    }

    // Create payment record
    const payment = await prisma.payment.create({
      data: {
        user_id: subscription.user_id,
        paypal_payment_id: paymentId,
        amount,
        currency,
        status: PaymentStatus.completed,
        type: PaymentType.subscription_payment,
        subscription_id: subscription.id,
      },
    });

    // Update subscription period
    const newPeriodStart = new Date();
    const newPeriodEnd = new Date();
    if (subscription.billing_cycle === BillingCycle.monthly) {
      newPeriodEnd.setMonth(newPeriodEnd.getMonth() + 1);
    } else {
      newPeriodEnd.setFullYear(newPeriodEnd.getFullYear() + 1);
    }

    await prisma.subscription.update({
      where: { id: subscription.id },
      data: {
        current_period_start: newPeriodStart,
        current_period_end: newPeriodEnd,
        status: SubscriptionStatus.active,
      },
    });

    // Allocate credits for renewal
    const creditAmount = getCreditAllocation(mapPlanTier(subscription.plan_tier));
    await allocateCredits(
      subscription.user_id,
      creditAmount,
      `Subscription renewed: ${subscription.plan_tier} (${subscription.billing_cycle})`,
      payment.id
    );

    // Track metrics
    paymentsTotal.inc({ status: 'completed', type: 'subscription_payment' });
    revenueTotal.inc(
      { tier: subscription.plan_tier, cycle: subscription.billing_cycle },
      amount
    );

    logger.info('Payment completed', {
      userId: subscription.user_id,
      paymentId,
      subscriptionId,
      amount,
      currency,
      creditsAllocated: creditAmount,
    });
  } catch (error) {
    logger.error('Failed to handle payment completion', {
      paymentId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Actions**:

1. **Find Subscription**: Lookup by `paypal_subscription_id` (from `billing_agreement_id`)
2. **Record Payment**: Create Payment record with amount, currency, status
3. **Update Period**: Roll forward `current_period_start` and `current_period_end`
4. **Allocate Credits**: Call `allocateCredits()` with renewal description and payment ID
5. **Track Metrics**: Increment `paymentsTotal` and `revenueTotal` Prometheus counters

**Metrics**:

- `paymentsTotal{status="completed", type="subscription_payment"}`: Total completed payments
- `revenueTotal{tier="starter|professional|enterprise", cycle="monthly|annual"}`: Revenue by plan

### 3. Subscription Cancelled

Handles user-initiated or admin-initiated subscription cancellation.

```typescript
async function handleSubscriptionCancelled(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:419-443):

```typescript
async function handleSubscriptionCancelled(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const subscriptionId = payload.resource.id;

    // Update subscription status
    const subscription = await prisma.subscription.update({
      where: { paypal_subscription_id: subscriptionId },
      data: {
        status: SubscriptionStatus.cancelled,
        cancel_at_period_end: true,
      },
    });

    logger.info('Subscription cancelled', {
      userId: subscription.user_id,
      subscriptionId,
    });
  } catch (error) {
    logger.error('Failed to handle subscription cancellation', {
      subscriptionId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Behavior**:

- **Status Update**: Sets `status = 'cancelled'`
- **Graceful Termination**: Sets `cancel_at_period_end = true` (user retains access until end of paid period)
- **No Credit Deduction**: User keeps existing credits until period expires
- **No Refund**: PayPal handles refund logic separately

### 4. Subscription Suspended

Handles payment failure leading to subscription suspension.

```typescript
async function handleSubscriptionSuspended(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:448-471):

```typescript
async function handleSubscriptionSuspended(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const subscriptionId = payload.resource.id;

    // Update subscription status
    const subscription = await prisma.subscription.update({
      where: { paypal_subscription_id: subscriptionId },
      data: {
        status: SubscriptionStatus.suspended,
      },
    });

    logger.info('Subscription suspended', {
      userId: subscription.user_id,
      subscriptionId,
    });
  } catch (error) {
    logger.error('Failed to handle subscription suspension', {
      subscriptionId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Behavior**:

- **Temporary Suspension**: User may reactivate by updating payment method in PayPal
- **No Credit Deduction**: Existing credits remain available
- **Access Control**: Application may restrict translation access for suspended users
- **Reactivation**: PayPal will send `BILLING.SUBSCRIPTION.ACTIVATED` if payment succeeds

### 5. Subscription Expired

Handles subscription expiration after period end.

```typescript
async function handleSubscriptionExpired(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:476-499):

```typescript
async function handleSubscriptionExpired(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const subscriptionId = payload.resource.id;

    // Update subscription status
    const subscription = await prisma.subscription.update({
      where: { paypal_subscription_id: subscriptionId },
      data: {
        status: SubscriptionStatus.cancelled,
      },
    });

    logger.info('Subscription expired', {
      userId: subscription.user_id,
      subscriptionId,
    });
  } catch (error) {
    logger.error('Failed to handle subscription expiration', {
      subscriptionId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Behavior**:

- **Permanent Cancellation**: Sets `status = 'cancelled'` (no reactivation)
- **End of Service**: User loses subscription benefits
- **Credits Retained**: Unused credits remain in account (not deducted)
- **Resubscription Required**: User must create new subscription for continued service

### 6. Subscription Updated

Handles plan tier or billing cycle changes (upgrades/downgrades).

```typescript
async function handleSubscriptionUpdated(payload: PayPalWebhookEvent): Promise<void>
```

**Implementation** (paypalService.ts:504-538):

```typescript
async function handleSubscriptionUpdated(payload: PayPalWebhookEvent): Promise<void> {
  try {
    const subscriptionId = payload.resource.id;
    const planId = payload.resource.plan_id;

    if (!planId) {
      logger.warn('No plan ID in subscription update', { subscriptionId });
      return;
    }

    const { planTier, billingCycle } = parsePlanId(planId);

    // Update subscription
    const subscription = await prisma.subscription.update({
      where: { paypal_subscription_id: subscriptionId },
      data: {
        plan_tier: planTier,
        billing_cycle: billingCycle,
      },
    });

    logger.info('Subscription updated', {
      userId: subscription.user_id,
      subscriptionId,
      planTier,
      billingCycle,
    });
  } catch (error) {
    logger.error('Failed to handle subscription update', {
      subscriptionId: payload.resource.id,
      error,
    });
    throw error;
  }
}
```

**Behavior**:

- **Plan Change**: Updates `plan_tier` and `billing_cycle` based on new plan ID
- **No Immediate Credit Allocation**: Credits allocated on next billing cycle (PAYMENT.SALE.COMPLETED)
- **Prorated Charges**: PayPal handles billing adjustments
- **Graceful Transition**: User retains access during transition

## Subscription Management API

### Cancel Subscription

Allows users to cancel their subscription programmatically.

```typescript
export async function cancelSubscription(
  userId: string,
  subscriptionId: string,
  reason?: string
): Promise<void>
```

**Implementation** (paypalService.ts:547-600):

```typescript
export async function cancelSubscription(
  userId: string,
  subscriptionId: string,
  reason?: string
): Promise<void> {
  try {
    // Verify subscription belongs to user
    const subscription = await prisma.subscription.findFirst({
      where: {
        user_id: userId,
        paypal_subscription_id: subscriptionId,
      },
    });

    if (!subscription) {
      throw new Error('Subscription not found or does not belong to user');
    }

    // Cancel subscription in PayPal
    const request = new paypal.subscriptions.SubscriptionsCancelRequest(subscriptionId);
    request.requestBody({
      reason: reason || 'User requested cancellation',
    });

    const client = getPayPalClient();
    await client.execute(request);

    // Update subscription status
    await prisma.subscription.update({
      where: { id: subscription.id },
      data: {
        status: SubscriptionStatus.cancelled,
        cancel_at_period_end: true,
      },
    });

    logger.info('Subscription cancelled', {
      userId,
      subscriptionId,
      reason,
    });

    // Track metrics
    subscriptionEventsTotal.inc({ event_type: 'subscription_cancelled' });
  } catch (error) {
    logger.error('Failed to cancel subscription', {
      userId,
      subscriptionId,
      error,
    });
    trackError('paypal', 'subscription_cancellation_failed');
    throw error;
  }
}
```

**Security**:

- **Ownership Verification**: Checks `user_id` matches before cancellation
- **Authorization**: Prevents users from cancelling other users' subscriptions
- **Audit Trail**: Logs cancellation reason for analytics

**Cancellation Flow**:

1. Verify subscription ownership
2. Send cancellation request to PayPal API
3. Update local database status
4. Wait for webhook confirmation (BILLING.SUBSCRIPTION.CANCELLED)

### Get Subscription Details

Fetches current subscription details from PayPal.

```typescript
export async function getSubscriptionDetails(subscriptionId: string): Promise<any>
```

**Implementation** (paypalService.ts:632-646):

```typescript
export async function getSubscriptionDetails(subscriptionId: string): Promise<any> {
  try {
    const request = new paypal.subscriptions.SubscriptionsGetRequest(subscriptionId);
    const client = getPayPalClient();
    const response = await client.execute(request);

    return response.result;
  } catch (error) {
    logger.error('Failed to get subscription details', {
      subscriptionId,
      error,
    });
    throw error;
  }
}
```

**Response Data**:

```json
{
  "id": "I-BW452GLLEP1G",
  "plan_id": "P-5ML4271244454362WXNWU5NQ",
  "status": "ACTIVE",
  "status_update_time": "2023-03-15T10:00:00Z",
  "start_time": "2023-03-15T10:00:00Z",
  "subscriber": {
    "email_address": "user@example.com",
    "payer_id": "PAYERID123"
  },
  "billing_info": {
    "outstanding_balance": {
      "currency_code": "USD",
      "value": "0.00"
    },
    "cycle_executions": [
      {
        "tenure_type": "REGULAR",
        "sequence": 1,
        "cycles_completed": 3,
        "cycles_remaining": 0
      }
    ],
    "last_payment": {
      "amount": {
        "currency_code": "USD",
        "value": "29.00"
      },
      "time": "2023-05-15T10:00:00Z"
    },
    "next_billing_time": "2023-06-15T10:00:00Z"
  },
  "links": [
    {
      "href": "https://api.paypal.com/v1/billing/subscriptions/I-BW452GLLEP1G",
      "rel": "self",
      "method": "GET"
    },
    {
      "href": "https://www.paypal.com/webapps/billing/subscriptions?ba_token=BA-123",
      "rel": "approve",
      "method": "GET"
    }
  ]
}
```

**Use Cases**:

- **Admin Dashboard**: Display subscription status and billing history
- **User Account Page**: Show next billing date and payment method
- **Debugging**: Investigate webhook processing issues

## Plan Configuration

### Plan ID Mapping

PayPal plan IDs are stored in environment variables and mapped by tier + cycle.

```typescript
// config/index.ts
export const getPayPalPlanId = (
  tier: 'starter' | 'professional' | 'enterprise',
  cycle: 'monthly' | 'annual'
): string | undefined => {
  const plans = {
    starter: {
      monthly: config.paypalPlanStarterMonthly,
      annual: config.paypalPlanStarterAnnual,
    },
    professional: {
      monthly: config.paypalPlanProfessionalMonthly,
      annual: config.paypalPlanProfessionalAnnual,
    },
    enterprise: {
      monthly: config.paypalPlanEnterpriseMonthly,
      annual: config.paypalPlanEnterpriseAnnual,
    },
  };

  return plans[tier]?.[cycle];
};
```

**Environment Variables**:

```env
PAYPAL_PLAN_STARTER_MONTHLY=P-1A23BC4D5E6F7G8H9
PAYPAL_PLAN_STARTER_ANNUAL=P-9H8G7F6E5D4C3B2A1
PAYPAL_PLAN_PROFESSIONAL_MONTHLY=P-1B23CD4E5F6G7H8I9
PAYPAL_PLAN_PROFESSIONAL_ANNUAL=P-9I8H7G6F5E4D3C2B1
PAYPAL_PLAN_ENTERPRISE_MONTHLY=P-1C23DE4F5G6H7I8J9
PAYPAL_PLAN_ENTERPRISE_ANNUAL=P-9J8I7H6G5F4E3D2C1
```

**Plan Parsing**:

```typescript
// paypalService.ts:605-624
function parsePlanId(planId: string): { planTier: PlanTier; billingCycle: BillingCycle } {
  // Match against configured plan IDs
  if (planId === config.paypalPlanStarterMonthly) {
    return { planTier: PlanTier.starter, billingCycle: BillingCycle.monthly };
  } else if (planId === config.paypalPlanStarterAnnual) {
    return { planTier: PlanTier.starter, billingCycle: BillingCycle.annual };
  } else if (planId === config.paypalPlanProfessionalMonthly) {
    return { planTier: PlanTier.professional, billingCycle: BillingCycle.monthly };
  } else if (planId === config.paypalPlanProfessionalAnnual) {
    return { planTier: PlanTier.professional, billingCycle: BillingCycle.annual };
  } else if (planId === config.paypalPlanEnterpriseMonthly) {
    return { planTier: PlanTier.enterprise, billingCycle: BillingCycle.monthly };
  } else if (planId === config.paypalPlanEnterpriseAnnual) {
    return { planTier: PlanTier.enterprise, billingCycle: BillingCycle.annual };
  }

  // Default fallback
  logger.warn('Unknown PayPal plan ID', { planId });
  return { planTier: PlanTier.starter, billingCycle: BillingCycle.monthly };
}
```

**Alternative Parsing** (webhooks.ts:430-442):

```typescript
function parsePlanId(planId: string): { tier: 'starter' | 'professional' | 'enterprise' | null; cycle: 'monthly' | 'annual' | null } {
  const paypalConfig = getPayPalConfig();
  const plans = {
    [paypalConfig.plans.starterMonthly]: { tier: 'starter' as const, cycle: 'monthly' as const },
    [paypalConfig.plans.starterAnnual]: { tier: 'starter' as const, cycle: 'annual' as const },
    [paypalConfig.plans.professionalMonthly]: { tier: 'professional' as const, cycle: 'monthly' as const },
    [paypalConfig.plans.professionalAnnual]: { tier: 'professional' as const, cycle: 'annual' as const },
    [paypalConfig.plans.enterpriseMonthly]: { tier: 'enterprise' as const, cycle: 'monthly' as const },
    [paypalConfig.plans.enterpriseAnnual]: { tier: 'enterprise' as const, cycle: 'annual' as const },
  };

  return plans[planId] || { tier: null, cycle: null };
}
```

## Webhook Endpoint

### Route Handler

```typescript
// webhooks.ts:61-155
router.post('/paypal', async (req: Request, res: Response) => {
  const startTime = Date.now();
  const requestId = req.requestId || 'unknown';

  try {
    // Get raw body for signature verification
    const rawBody = JSON.stringify(req.body);

    // Verify webhook signature
    const isValid = await verifyWebhookSignature(req.headers as Record<string, string>, rawBody);

    if (!isValid) {
      logger.warn('PayPal webhook signature verification failed', {
        requestId,
        eventType: req.body?.event_type,
        headers: {
          hasTransmissionId: !!req.headers['paypal-transmission-id'],
          hasTransmissionTime: !!req.headers['paypal-transmission-time'],
          hasTransmissionSig: !!req.headers['paypal-transmission-sig'],
          hasCertUrl: !!req.headers['paypal-cert-url'],
          hasAuthAlgo: !!req.headers['paypal-auth-algo'],
        },
      });
      // Return 401 Unauthorized for invalid signatures to indicate authentication failure
      return res.status(401).json({
        error: {
          code: 'INVALID_SIGNATURE',
          message: 'Webhook signature verification failed'
        }
      });
    }

    const event = req.body as PayPalWebhookEvent;

    logger.info('PayPal webhook received', {
      requestId,
      eventId: event.id,
      eventType: event.event_type,
      resourceType: event.resource_type,
      createTime: event.create_time,
    });

    // Handle different event types
    switch (event.event_type) {
      case 'BILLING.SUBSCRIPTION.ACTIVATED':
        await handleSubscriptionActivated(event);
        break;

      case 'BILLING.SUBSCRIPTION.CANCELLED':
        await handleSubscriptionCancelled(event);
        break;

      case 'BILLING.SUBSCRIPTION.SUSPENDED':
        await handleSubscriptionSuspended(event);
        break;

      case 'BILLING.SUBSCRIPTION.UPDATED':
        await handleSubscriptionUpdated(event);
        break;

      case 'PAYMENT.SALE.COMPLETED':
        await handlePaymentCompleted(event);
        break;

      default:
        logger.info('Unhandled PayPal webhook event type', {
          requestId,
          eventType: event.event_type,
        });
    }

    const duration = Date.now() - startTime;
    logger.info('PayPal webhook processed successfully', {
      requestId,
      eventId: event.id,
      eventType: event.event_type,
      duration,
    });

    // Always return 200 to acknowledge receipt
    return res.status(200).json({ received: true });
  } catch (error) {
    const duration = Date.now() - startTime;
    logger.error('PayPal webhook processing failed', {
      requestId,
      error: error instanceof Error ? error.message : 'Unknown error',
      stack: error instanceof Error ? error.stack : undefined,
      duration,
    });

    // Return 200 to prevent retries for processing errors
    // The event is logged and can be manually reprocessed if needed
    return res.status(200).json({ received: true, error: 'Processing error' });
  }
});
```

**Key Points**:

- **Signature Verification First**: Reject invalid signatures with 401
- **Idempotency**: Always return 200 to prevent PayPal retries (even on processing errors)
- **Structured Logging**: Log all events with request ID, event type, duration
- **Error Handling**: Catch processing errors, log, return 200 with error flag
- **Performance Tracking**: Measure webhook processing duration

### Webhook Configuration

**PayPal Developer Dashboard**:

1. Navigate to PayPal Developer Dashboard → Apps & Credentials
2. Select your app → Add Webhook
3. Set webhook URL: `https://api.press.zone/v1/webhooks/paypal`
4. Subscribe to events:
   - `BILLING.SUBSCRIPTION.ACTIVATED`
   - `BILLING.SUBSCRIPTION.CANCELLED`
   - `BILLING.SUBSCRIPTION.SUSPENDED`
   - `BILLING.SUBSCRIPTION.UPDATED`
   - `PAYMENT.SALE.COMPLETED`
5. Copy Webhook ID to environment variable: `PAYPAL_WEBHOOK_ID`

**Environment Variables**:

```env
PAYPAL_MODE=sandbox  # or 'live'
PAYPAL_CLIENT_ID=Abc123_ClientId
PAYPAL_CLIENT_SECRET=Xyz789_Secret
PAYPAL_WEBHOOK_ID=1A2B3C4D5E6F7G8H9
```

## Types & Interfaces

### PayPalWebhookEvent

```typescript
// types/index.ts
export interface PayPalWebhookEvent {
  id: string;
  event_type: string;
  resource_type: string;
  create_time: string;
  resource: {
    id: string;
    plan_id?: string;
    status?: string;
    subscriber?: {
      email_address?: string;
      payer_id?: string;
    };
    billing_agreement_id?: string;
    amount?: {
      total?: string;
      currency?: string;
    };
  };
}
```

### SubscriptionPlan

```typescript
export type SubscriptionPlan = 'starter' | 'professional' | 'enterprise';
```

### BillingCycle

```typescript
export type BillingCycleType = 'monthly' | 'annual';
```

## Security Considerations

### 1. Webhook Signature Verification

**Requirement**: ALL webhook events MUST be verified before processing.

**Implementation**:

- Extract 5 signature headers from request
- Send verification request to PayPal API
- Reject with 401 if verification fails
- Log warnings for suspicious requests

**Attack Prevention**:

- **Spoofed Webhooks**: Signature verification prevents attackers from forging webhook events
- **Replay Attacks**: Transmission timestamp prevents reuse of old events
- **Man-in-the-Middle**: HTTPS + signature ensures payload integrity

### 2. User Authorization

**Requirement**: Users can only cancel their own subscriptions.

**Implementation**:

```typescript
// Verify subscription belongs to user
const subscription = await prisma.subscription.findFirst({
  where: {
    user_id: userId,
    paypal_subscription_id: subscriptionId,
  },
});

if (!subscription) {
  throw new Error('Subscription not found or does not belong to user');
}
```

### 3. Idempotency

**Requirement**: Webhook handlers must be idempotent to handle duplicate events.

**Implementation**:

- Use `upsert()` for subscription creation/updates
- Check if payment already exists before creating record
- Return 200 even for duplicate events (prevents PayPal retries)

### 4. Data Validation

**Requirement**: Validate all webhook payload fields before processing.

**Implementation**:

```typescript
if (!subscriptionId || !planId || !email) {
  logger.warn('Missing required fields in subscription activation event', {
    subscriptionId,
    planId,
    email,
  });
  return;
}
```

**Validated Fields**:

- `subscription_id`: Must be non-empty string
- `plan_id`: Must match configured plan IDs
- `email`: Must exist in User table
- `amount`: Must be valid decimal number
- `currency`: Must be valid currency code (USD)

## Error Handling

### Webhook Processing Errors

**Strategy**: Log error but return 200 to prevent retries.

**Rationale**:

- Transient errors (DB connection lost) may resolve on retry
- Permanent errors (invalid plan ID) will never succeed
- Returning 4xx/5xx triggers PayPal retry mechanism (exponential backoff, up to 24 hours)
- For permanent errors, retries waste resources and flood logs

**Implementation**:

```typescript
try {
  // Process webhook event
  await handleSubscriptionActivated(event);
  return res.status(200).json({ received: true });
} catch (error) {
  logger.error('PayPal webhook processing failed', {
    requestId,
    error: error instanceof Error ? error.message : 'Unknown error',
    stack: error instanceof Error ? error.stack : undefined,
  });

  // Return 200 to prevent retries
  return res.status(200).json({ received: true, error: 'Processing error' });
}
```

### Manual Reprocessing

**Scenario**: Webhook processing failed due to transient error.

**Solution**: Fetch subscription details from PayPal and manually replay event.

```typescript
// Fetch current subscription state from PayPal
const details = await getSubscriptionDetails(subscriptionId);

// Construct synthetic webhook event
const event: PayPalWebhookEvent = {
  id: 'manual-replay-' + Date.now(),
  event_type: 'BILLING.SUBSCRIPTION.ACTIVATED',
  resource_type: 'subscription',
  create_time: new Date().toISOString(),
  resource: {
    id: details.id,
    plan_id: details.plan_id,
    status: details.status,
    subscriber: details.subscriber,
  },
};

// Process event manually
await handleSubscriptionActivated(event);
```

## Monitoring & Metrics

### Prometheus Metrics

```typescript
// Subscription events
subscriptionEventsTotal.inc({ event_type: 'subscription_created' });
subscriptionEventsTotal.inc({ event_type: 'subscription_cancelled' });
subscriptionEventsTotal.inc({ event_type: payload.event_type });

// Payments
paymentsTotal.inc({ status: 'completed', type: 'subscription_payment' });

// Revenue
revenueTotal.inc(
  { tier: subscription.plan_tier, cycle: subscription.billing_cycle },
  amount
);

// Errors
trackError('paypal', 'subscription_creation_failed');
trackError('paypal', 'webhook_verification_failed');
trackError('paypal', 'webhook_processing_failed');
trackError('paypal', 'subscription_cancellation_failed');
```

### Log Queries

**Find Failed Webhook Events**:

```bash
grep "PayPal webhook processing failed" /var/log/translate-backend/api.log | jq '.eventId'
```

**Find Subscription Activations**:

```bash
grep "Subscription activated" /var/log/translate-backend/api.log | jq '{userId: .userId, tier: .planTier, cycle: .billingCycle, credits: .creditsAllocated}'
```

**Find Revenue Events**:

```bash
grep "Payment completed" /var/log/translate-backend/api.log | jq '{amount: .amount, currency: .currency, tier: .subscription.plan_tier}'
```

### Alerting Rules

**Webhook Verification Failures**:

```yaml
- alert: PayPalWebhookVerificationFailure
  expr: rate(paypal_webhook_verification_failures[5m]) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High rate of PayPal webhook verification failures"
    description: "{{ $value }} webhook verification failures per second over 5 minutes"
```

**Subscription Processing Failures**:

```yaml
- alert: PayPalSubscriptionProcessingFailure
  expr: rate(paypal_subscription_processing_failures[10m]) > 0.05
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "PayPal subscription processing failures detected"
    description: "{{ $value }} subscription processing failures per second over 10 minutes"
```

## Testing

### Unit Tests

```typescript
// __tests__/unit/services/paypalService.test.ts
import { createSubscription, verifyWebhookSignature, parsePlanId } from '../../../src/services/paypalService';

describe('PayPal Service', () => {
  describe('createSubscription', () => {
    it('should create subscription with correct plan ID', async () => {
      const userId = 'user-123';
      const planTier = 'professional';
      const billingCycle = 'monthly';

      const result = await createSubscription(userId, planTier, billingCycle);

      expect(result).toHaveProperty('approvalUrl');
      expect(result).toHaveProperty('subscriptionId');
      expect(result.approvalUrl).toContain('paypal.com');
    });

    it('should throw error for invalid plan', async () => {
      const userId = 'user-123';
      const planTier = 'invalid' as any;
      const billingCycle = 'monthly';

      await expect(createSubscription(userId, planTier, billingCycle)).rejects.toThrow('Invalid plan tier');
    });
  });

  describe('verifyWebhookSignature', () => {
    it('should return true for valid signature', async () => {
      const payload = JSON.stringify({ event_type: 'BILLING.SUBSCRIPTION.ACTIVATED' });
      const headers = {
        'paypal-transmission-id': 'abc123',
        'paypal-transmission-time': '2023-03-15T10:00:00Z',
        'paypal-cert-url': 'https://api.paypal.com/cert',
        'paypal-auth-algo': 'SHA256withRSA',
        'paypal-transmission-sig': 'signature123',
      };

      const isValid = await verifyWebhookSignature(payload, headers);

      expect(isValid).toBe(true);
    });

    it('should return false for missing headers', async () => {
      const payload = JSON.stringify({ event_type: 'BILLING.SUBSCRIPTION.ACTIVATED' });
      const headers = {};

      const isValid = await verifyWebhookSignature(payload, headers);

      expect(isValid).toBe(false);
    });
  });

  describe('parsePlanId', () => {
    it('should parse starter monthly plan', () => {
      const planId = process.env.PAYPAL_PLAN_STARTER_MONTHLY;
      const result = parsePlanId(planId);

      expect(result).toEqual({
        planTier: 'starter',
        billingCycle: 'monthly',
      });
    });

    it('should return default for unknown plan', () => {
      const planId = 'UNKNOWN_PLAN';
      const result = parsePlanId(planId);

      expect(result).toEqual({
        planTier: 'starter',
        billingCycle: 'monthly',
      });
    });
  });
});
```

### Integration Tests

```typescript
// __tests__/integration/webhooks/paypal.test.ts
import request from 'supertest';
import app from '../../../src/server';

describe('POST /v1/webhooks/paypal', () => {
  it('should process subscription activation webhook', async () => {
    const event = {
      id: 'WH-123',
      event_type: 'BILLING.SUBSCRIPTION.ACTIVATED',
      resource_type: 'subscription',
      create_time: '2023-03-15T10:00:00Z',
      resource: {
        id: 'I-BW452GLLEP1G',
        plan_id: process.env.PAYPAL_PLAN_STARTER_MONTHLY,
        status: 'ACTIVE',
        subscriber: {
          email_address: 'user@example.com',
          payer_id: 'PAYERID123',
        },
      },
    };

    const response = await request(app)
      .post('/v1/webhooks/paypal')
      .set('paypal-transmission-id', 'abc123')
      .set('paypal-transmission-time', '2023-03-15T10:00:00Z')
      .set('paypal-cert-url', 'https://api.paypal.com/cert')
      .set('paypal-auth-algo', 'SHA256withRSA')
      .set('paypal-transmission-sig', 'signature123')
      .send(event);

    expect(response.status).toBe(200);
    expect(response.body).toEqual({ received: true });
  });

  it('should reject webhook with invalid signature', async () => {
    const event = {
      id: 'WH-123',
      event_type: 'BILLING.SUBSCRIPTION.ACTIVATED',
      resource_type: 'subscription',
      create_time: '2023-03-15T10:00:00Z',
      resource: {
        id: 'I-BW452GLLEP1G',
        plan_id: process.env.PAYPAL_PLAN_STARTER_MONTHLY,
        status: 'ACTIVE',
      },
    };

    const response = await request(app)
      .post('/v1/webhooks/paypal')
      .send(event);

    expect(response.status).toBe(401);
    expect(response.body.error.code).toBe('INVALID_SIGNATURE');
  });
});
```

## Quick Reference

### Function Signatures

```typescript
// Subscription creation
createSubscription(userId: string, planTier: SubscriptionPlan, billingCycle: BillingCycleType): Promise<{ approvalUrl: string; subscriptionId: string }>

// Webhook verification
verifyWebhookSignature(payload: string, headers: Record<string, string>): Promise<boolean>

// Webhook event handling
handleWebhookEvent(payload: PayPalWebhookEvent, headers: Record<string, string>): Promise<void>

// Subscription management
cancelSubscription(userId: string, subscriptionId: string, reason?: string): Promise<void>
getSubscriptionDetails(subscriptionId: string): Promise<any>

// Plan utilities
getPayPalPlanId(tier: 'starter' | 'professional' | 'enterprise', cycle: 'monthly' | 'annual'): string | undefined
parsePlanId(planId: string): { planTier: PlanTier; billingCycle: BillingCycle }
mapPlanTier(tier: PlanTier): 'starter' | 'professional' | 'enterprise'
```

### Typical Subscription Flow

```
1. User initiates subscription → POST /v1/subscription/create
2. API calls createSubscription() → Returns approval URL
3. User redirected to PayPal → Completes checkout
4. PayPal sends BILLING.SUBSCRIPTION.ACTIVATED webhook
5. API verifies signature → handleSubscriptionActivated()
6. Upsert Subscription record → Allocate credits
7. User receives confirmation email (if enabled)
8. Recurring payments trigger PAYMENT.SALE.COMPLETED webhook
9. API allocates renewal credits → Updates subscription period
10. User cancellation triggers BILLING.SUBSCRIPTION.CANCELLED webhook
11. API sets status=cancelled, cancel_at_period_end=true
```

### Common Queries

```typescript
// Create new subscription
const { approvalUrl, subscriptionId } = await createSubscription(
  userId,
  'professional',
  'monthly'
);

// Cancel subscription
await cancelSubscription(userId, subscriptionId, 'User requested cancellation');

// Get subscription details from PayPal
const details = await getSubscriptionDetails(subscriptionId);

// Parse plan ID
const { planTier, billingCycle } = parsePlanId(details.plan_id);

// Get credit allocation for tier
const credits = getCreditAllocation('professional'); // 500000
```

## Validation Checklist

Before deploying PayPal integration:

- [ ] PayPal client ID and secret configured
- [ ] Webhook ID configured and verified
- [ ] All 6 plan IDs configured (3 tiers × 2 cycles)
- [ ] Webhook URL set to `https://api.press.zone/v1/webhooks/paypal`
- [ ] Webhook subscribed to 5 event types (ACTIVATED, CANCELLED, SUSPENDED, UPDATED, PAYMENT.SALE.COMPLETED)
- [ ] Signature verification working (test with PayPal simulator)
- [ ] Return URLs configured (success/cancel)
- [ ] Credit allocation amounts verified for each tier
- [ ] Database indexes on `paypal_subscription_id` and `paypal_payment_id`
- [ ] Subscription upsert logic idempotent
- [ ] Payment record creation idempotent
- [ ] Error handling returns 200 for processing errors
- [ ] Structured logging for all events
- [ ] Prometheus metrics tracked
- [ ] Alerting rules configured for failures
- [ ] Unit tests cover all event handlers
- [ ] Integration tests verify signature validation
- [ ] Manual reprocessing procedure documented
- [ ] Sandbox testing complete before production

## Monitoring & Alerting Setup

### Overview

The Press.Zone Backend API exposes comprehensive Prometheus metrics for monitoring, observability, and alerting. This section documents the metrics collection system, health check endpoints, alerting recommendations, and integration with monitoring tools like Prometheus and Grafana.

---

### Health Check Endpoints

The API provides three health check endpoints for different monitoring scenarios:

#### 1. Basic Health Check
**Endpoint:** `GET /health`
**Authentication:** None
**Purpose:** Verifies that the API process is running

**Response:**
```json
{
  "status": "healthy",
  "service": "translate-api",
  "version": "1.0.0",
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Use Cases:**
- Load balancer health checks
- Simple uptime monitoring
- Docker health check directive

---

#### 2. Readiness Check
**Endpoint:** `GET /health/ready`
**Authentication:** None
**Purpose:** Verifies that the API is ready to accept traffic (checks dependencies)

**Checks Performed:**
- PostgreSQL database connectivity (`SELECT 1` query)
- Redis connectivity (`PING` command)

**Response (Healthy):**
```json
{
  "status": "ready",
  "checks": {
    "database": {
      "status": "healthy"
    },
    "redis": {
      "status": "healthy"
    }
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Response (Unhealthy - 503):**
```json
{
  "status": "not ready",
  "checks": {
    "database": {
      "status": "healthy"
    },
    "redis": {
      "status": "unhealthy",
      "error": "Connection timeout"
    }
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Use Cases:**
- Kubernetes readiness probes
- Pre-deployment validation
- Dependency health monitoring
- Traffic routing decisions

---

#### 3. Liveness Check
**Endpoint:** `GET /health/live`
**Authentication:** None
**Purpose:** Verifies that the API process is alive (but might not be ready)

**Response:**
```json
{
  "status": "alive",
  "uptime": 86400.5,
  "memory": {
    "rss": 52428800,
    "heapTotal": 28311552,
    "heapUsed": 22456784,
    "external": 1024000,
    "arrayBuffers": 512000
  },
  "timestamp": "2026-01-27T12:00:00.000Z"
}
```

**Use Cases:**
- Kubernetes liveness probes
- Process crash detection
- Memory leak monitoring

---

### Prometheus Metrics Endpoint

**Endpoint:** `GET /metrics`
**Authentication:** Admin JWT required
**Format:** Prometheus text format
**Source:** `api/src/utils/metrics.ts`

**Access:**
```bash
curl -H "Authorization: Bearer <ADMIN_JWT_TOKEN>" \
  https://api.press.zone/metrics
```

---

### Available Metrics

#### HTTP Request Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `http_requests_total` | Counter | method, route, status_code | Total number of HTTP requests |
| `http_request_duration_seconds` | Histogram | method, route, status_code | Duration of HTTP requests (buckets: 0.1, 0.5, 1, 2, 5, 10s) |

**Example:**
```prometheus
# Total requests to /v1/translate endpoint
http_requests_total{method="POST",route="/v1/translate",status_code="200"} 1500

# 95th percentile response time
http_request_duration_seconds{method="POST",route="/v1/translate",status_code="200",quantile="0.95"} 2.5
```

---

#### Translation Job Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `translation_jobs_total` | Counter | model, status, type | Total number of translation jobs |
| `translation_job_duration_seconds` | Histogram | model, type | Duration of translation jobs (buckets: 1, 5, 10, 30, 60, 120s) |
| `tokens_processed_total` | Counter | model | Total number of tokens processed |

**Labels:**
- `model`: `gemini-3-flash-preview`, `gemini-pro`
- `status`: `pending`, `processing`, `completed`, `failed`, `cancelled`
- `type`: `sync`, `async`

**Example:**
```prometheus
# Successful async translations
translation_jobs_total{model="gemini-3-flash-preview",status="completed",type="async"} 5000

# Average job duration
translation_job_duration_seconds_sum{model="gemini-3-flash-preview",type="async"} 25000
translation_job_duration_seconds_count{model="gemini-3-flash-preview",type="async"} 5000
# Average = 25000 / 5000 = 5 seconds per job

# Total tokens processed
tokens_processed_total{model="gemini-3-flash-preview"} 10000000
```

---

#### Webhook Delivery Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `webhook_deliveries_total` | Counter | success | Total webhook delivery attempts |
| `webhook_delivery_duration_seconds` | Histogram | (none) | Duration of webhook delivery (buckets: 0.1, 0.5, 1, 2, 5s) |

**Example:**
```prometheus
# Successful webhook deliveries
webhook_deliveries_total{success="true"} 4500

# Failed webhook deliveries
webhook_deliveries_total{success="false"} 500

# Webhook success rate = 4500 / (4500 + 500) = 90%
```

---

#### Subscription Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `active_subscriptions` | Gauge | tier | Current number of active subscriptions |
| `subscription_events_total` | Counter | event_type | Total subscription lifecycle events |

**Labels:**
- `tier`: `starter`, `professional`, `enterprise`
- `event_type`: `ACTIVATED`, `CANCELLED`, `SUSPENDED`, `UPDATED`

**Example:**
```prometheus
# Active subscriptions by tier
active_subscriptions{tier="starter"} 200
active_subscriptions{tier="professional"} 150
active_subscriptions{tier="enterprise"} 50

# Total cancellations
subscription_events_total{event_type="CANCELLED"} 25
```

---

#### Credit Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `credits_allocated_total` | Counter | tier | Total credits allocated to users |
| `credits_deducted_total` | Counter | tier, model | Total credits deducted for translations |

**Example:**
```prometheus
# Total credits allocated to professional tier
credits_allocated_total{tier="professional"} 50000000

# Credits consumed by gemini-3-flash-preview
credits_deducted_total{tier="professional",model="gemini-3-flash-preview"} 35000000

# Utilization rate = 35M / 50M = 70%
```

---

#### Payment Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `payments_total` | Counter | status, type | Total number of payments |
| `revenue_total` | Counter | tier, cycle | Total revenue in USD |

**Labels:**
- `status`: `completed`, `failed`, `pending`
- `type`: `subscription`, `one_time`
- `cycle`: `monthly`, `annual`

**Example:**
```prometheus
# Total completed payments
payments_total{status="completed",type="subscription"} 400

# Monthly revenue from professional tier
revenue_total{tier="professional",cycle="monthly"} 8000.00
```

---

#### Error Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `errors_total` | Counter | type, code | Total number of errors |

**Labels:**
- `type`: `ValidationError`, `AuthenticationError`, `InsufficientCreditsError`, `InternalServerError`, etc.
- `code`: HTTP status code or custom error code

**Example:**
```prometheus
# Total authentication failures
errors_total{type="AuthenticationError",code="401"} 125

# Total insufficient credit errors
errors_total{type="InsufficientCreditsError",code="402"} 45

# Error rate = errors / total requests
```

---

#### Database Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `database_query_duration_seconds` | Histogram | operation | Duration of database queries (buckets: 0.01, 0.05, 0.1, 0.5, 1, 2s) |
| `database_connections_active` | Gauge | (none) | Number of active database connections |

**Example:**
```prometheus
# 95th percentile query time
database_query_duration_seconds{operation="select",quantile="0.95"} 0.15

# Active DB connections
database_connections_active 8
```

---

#### Queue Metrics

| Metric Name | Type | Labels | Description |
|-------------|------|--------|-------------|
| `queue_jobs_total` | Counter | queue, status | Total queue jobs processed |
| `queue_job_duration_seconds` | Histogram | queue | Duration of queue jobs (buckets: 1, 5, 10, 30, 60, 120s) |
| `queue_length` | Gauge | queue | Current number of jobs in queue |

**Labels:**
- `queue`: `translation-queue`, `webhook-queue`
- `status`: `completed`, `failed`, `active`, `delayed`

**Example:**
```prometheus
# Translation queue depth
queue_length{queue="translation-queue"} 25

# Completed translation jobs
queue_jobs_total{queue="translation-queue",status="completed"} 5000
```

---

### Alerting Recommendations

Configure alerts in Prometheus Alertmanager or your monitoring tool based on these thresholds:

#### Critical Alerts (Immediate Action Required)

```yaml
# 1. API Unavailable
- alert: APIDown
  expr: up{job="translate-api"} == 0
  for: 1m
  severity: critical
  summary: "API service is down"

# 2. Database Connection Failure
- alert: DatabaseUnhealthy
  expr: probe_success{job="health-check",endpoint="/health/ready"} == 0
  for: 2m
  severity: critical
  summary: "Database connectivity issues detected"

# 3. High Error Rate
- alert: HighErrorRate
  expr: (rate(errors_total[5m]) / rate(http_requests_total[5m])) > 0.05
  for: 5m
  severity: critical
  summary: "Error rate above 5% for 5 minutes"

# 4. Memory Leak Suspected
- alert: MemoryUsageHigh
  expr: process_resident_memory_bytes > 2000000000  # 2GB
  for: 10m
  severity: critical
  summary: "API memory usage exceeds 2GB"

# 5. Queue Backup
- alert: QueueBacklog
  expr: queue_length{queue="translation-queue"} > 100
  for: 10m
  severity: critical
  summary: "Translation queue has over 100 pending jobs"
```

#### Warning Alerts (Investigation Recommended)

```yaml
# 1. Slow API Response
- alert: SlowAPIResponse
  expr: histogram_quantile(0.95, http_request_duration_seconds_bucket) > 5
  for: 10m
  severity: warning
  summary: "95th percentile response time exceeds 5 seconds"

# 2. Webhook Delivery Failure Rate
- alert: WebhookFailureRate
  expr: (rate(webhook_deliveries_total{success="false"}[10m]) / rate(webhook_deliveries_total[10m])) > 0.10
  for: 10m
  severity: warning
  summary: "Webhook failure rate above 10%"

# 3. Disk Space Low
- alert: DiskSpaceLow
  expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.20
  for: 5m
  severity: warning
  summary: "Disk space below 20%"

# 4. Translation Job Failures
- alert: TranslationFailureRate
  expr: (rate(translation_jobs_total{status="failed"}[10m]) / rate(translation_jobs_total[10m])) > 0.05
  for: 10m
  severity: warning
  summary: "Translation failure rate above 5%"

# 5. Subscription Churn
- alert: HighSubscriptionCancellations
  expr: rate(subscription_events_total{event_type="CANCELLED"}[1h]) > 5
  for: 1h
  severity: warning
  summary: "More than 5 subscription cancellations per hour"
```

---

### Prometheus Configuration

**Scrape Config:**
```yaml
# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'translate-api'
    scrape_interval: 15s
    scrape_timeout: 10s
    scheme: https
    bearer_token: '<ADMIN_JWT_TOKEN>'
    static_configs:
      - targets:
          - 'api.press.zone:443'
    metrics_path: /metrics

  - job_name: 'health-check'
    scrape_interval: 30s
    http_sd_configs:
      - url: 'https://api.press.zone/health/ready'
    probe: blackbox_exporter
```

**Storage Retention:**
```yaml
# Keep metrics for 90 days
global:
  retention: 90d
```

---

### Grafana Dashboard Setup

#### 1. Import Dashboards

Pre-built dashboards for common metrics (create JSON files):

**API Overview Dashboard:**
- Request rate (requests/sec)
- Error rate percentage
- 95th percentile response time
- Active connections

**Translation Service Dashboard:**
- Jobs per minute (by status)
- Average job duration
- Tokens processed per minute
- Queue depth over time

**Business Metrics Dashboard:**
- Active subscriptions by tier
- Revenue (daily/monthly)
- Credit allocation vs utilization
- Subscription churn rate

**System Health Dashboard:**
- CPU usage
- Memory usage
- Database connection pool
- Queue worker health

#### 2. Data Source Configuration

```yaml
# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false
```

#### 3. Alert Notification Channels

Configure notification channels in Grafana:
- **Slack:** For critical alerts
- **Email:** For warning alerts
- **PagerDuty:** For on-call rotation
- **Webhook:** For custom integrations

---

### Log Aggregation

#### Structured Logging Format

All logs are emitted in JSON format via Winston logger:

```json
{
  "level": "info",
  "message": "Translation job completed",
  "timestamp": "2026-01-27T12:00:00.000Z",
  "service": "translate-api",
  "jobId": "job-uuid-12345",
  "userId": "user-uuid-67890",
  "model": "gemini-3-flash-preview",
  "tokensUsed": 500,
  "duration": 2500
}
```

#### Log Levels

| Level | Use Case |
|-------|----------|
| `error` | Application errors, exceptions, failures |
| `warn` | Deprecation warnings, recoverable errors |
| `info` | Business events (job completion, payments) |
| `http` | HTTP request/response logging |
| `debug` | Detailed debugging (disabled in production) |

#### Centralized Logging (Recommended Tools)

**Option 1: ELK Stack**
```yaml
# docker-compose.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
```

**Option 2: Loki + Grafana**
```yaml
# docker-compose.yml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
```

#### Log Queries (Examples)

**Find all errors in last hour:**
```promql
{job="translate-api", level="error"} |= "" | line_format "{{.timestamp}} {{.message}}"
```

**Translation job failures:**
```promql
{job="translate-api"} |= "Translation job failed" | json
```

**High-latency requests:**
```promql
{job="translate-api", level="http"} | json | duration > 5000
```

---

### Performance Monitoring

#### Key Performance Indicators (KPIs)

| KPI | Target | Critical Threshold | Query |
|-----|--------|-------------------|-------|
| API Availability | >99.9% | <99.5% | `avg_over_time(up{job="translate-api"}[7d])` |
| Average Response Time | <1s | >3s | `rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])` |
| Error Rate | <1% | >5% | `rate(errors_total[5m]) / rate(http_requests_total[5m])` |
| Translation Success Rate | >98% | <95% | `rate(translation_jobs_total{status="completed"}[1h]) / rate(translation_jobs_total[1h])` |
| Webhook Success Rate | >95% | <90% | `rate(webhook_deliveries_total{success="true"}[1h]) / rate(webhook_deliveries_total[1h])` |
| Queue Processing Time | <10s | >60s | `histogram_quantile(0.95, rate(queue_job_duration_seconds_bucket[5m]))` |

#### Latency Percentiles

Monitor response time distribution:
- **P50 (Median):** Should be <500ms
- **P95:** Should be <2s
- **P99:** Should be <5s
- **P99.9:** Should be <10s

---

### Incident Response Runbook

#### 1. High Error Rate Alert

**Investigation Steps:**
1. Check `/metrics` endpoint for `errors_total` by type
2. Review recent logs: `tail -f /var/log/translate-api/error.log`
3. Check database connectivity: `curl https://api.press.zone/health/ready`
4. Identify affected endpoint from `http_requests_total` metrics
5. Review recent deployments or configuration changes

**Remediation:**
- If database issue: Restart database, check connection pool settings
- If external API issue (Gemini): Check quota limits, implement circuit breaker
- If code bug: Rollback deployment, hot-fix and redeploy

---

#### 2. Queue Backup Alert

**Investigation Steps:**
1. Check queue length: `queue_length{queue="translation-queue"}`
2. Check worker health: `systemctl status presszone-backend-worker.service`
3. Review failed jobs: Query database for `status='failed'`
4. Check Redis connectivity: `redis-cli PING`

**Remediation:**
- Scale up workers: Add more worker instances
- Clear stuck jobs: `npx bull-board` for queue management
- Investigate job failures: Fix underlying issue (API rate limit, etc.)

---

#### 3. Memory Leak Alert

**Investigation Steps:**
1. Check process memory: `ps aux | grep node`
2. Generate heap snapshot: `node --inspect` + Chrome DevTools
3. Review memory metrics in Grafana
4. Check for memory-intensive operations (large translations)

**Remediation:**
- Restart service: `systemctl restart presszone-backend-api.service`
- Investigate code: Look for memory leaks (event listeners, large buffers)
- Implement fixes: Add memory limits, improve garbage collection

---

### Monitoring Best Practices

1. **Set Up Synthetic Monitoring:**
   - External uptime monitoring (UptimeRobot, Pingdom)
   - Periodic health check pings
   - Test critical user flows

2. **Establish Baselines:**
   - Record normal traffic patterns
   - Document expected metrics ranges
   - Set alerts based on deviation from baseline

3. **Regular Review:**
   - Weekly metrics review meetings
   - Monthly capacity planning
   - Quarterly alert tuning

4. **Document Incidents:**
   - Post-mortem analysis for outages
   - Root cause identification
   - Action items for prevention

5. **Test Alert Delivery:**
   - Monthly test of alert channels
   - Verify on-call rotation
   - Practice incident response procedures

---

### Integration with External Tools

#### Datadog Integration
```bash
# Install Datadog agent
DD_API_KEY=<your-api-key> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

# Configure API monitoring
cat <<EOF > /etc/datadog-agent/conf.d/prometheus.yaml
init_config:
instances:
  - prometheus_url: https://api.press.zone/metrics
    namespace: translate_api
    metrics:
      - '*'
EOF
```

#### New Relic Integration
```javascript
// Add to api/src/server.ts
require('newrelic');

// newrelic.js configuration
exports.config = {
  app_name: ['Translate API'],
  license_key: process.env.NEW_RELIC_LICENSE_KEY,
  logging: {
    level: 'info'
  }
};
```

#### Sentry Error Tracking
```javascript
// Add to api/src/server.ts
import * as Sentry from '@sentry/node';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 0.1,
});

app.use(Sentry.Handlers.requestHandler());
app.use(Sentry.Handlers.errorHandler());
```

---

This monitoring and alerting setup provides comprehensive observability into the Press.Zone Backend API, enabling proactive issue detection and rapid incident response.

---


---

# Skill 21: Webhook Delivery System

## Identity
- **Skill ID**: `webhook-delivery`
- **Domain**: Async Notifications, Webhook Management, Retry Logic
- **Technologies**: Node.js, Axios, HMAC Signatures, Exponential Backoff
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Webhook callback delivery to WordPress sites
- HMAC signature generation/verification
- Retry logic with exponential backoff
- Webhook delivery tracking and monitoring
- SSRF prevention and URL validation
- Circuit breaker patterns
- Webhook timeout handling
- Asynchronous job completion notifications

**File patterns:**
- `api/src/services/webhookService.ts`
- `api/src/routes/webhooks.ts`
- Webhook-related database models

## Core Patterns

### 1. Webhook Delivery Lifecycle

```
Translation Job Completion
         ↓
   Webhook Payload Created
         ↓
   HMAC Signature Generated
         ↓
   POST to Callback URL (10s timeout)
         ↓
   ┌─────────────┴─────────────┐
   ↓                           ↓
Success (200-299)          Failure
   ↓                           ↓
Record Success          Retry with Backoff
   ↓                      (max 5 attempts)
Done                            ↓
                    ┌───────────┴───────────┐
                    ↓                       ↓
              Success               Max Retries Exceeded
                    ↓                       ↓
            Record Success          Record Permanent Failure
                    ↓                       ↓
                  Done              Circuit Breaker Triggered
```

### 2. HMAC Signature Generation

**Algorithm**: SHA-256 HMAC

```typescript
// Signature generation (backend)
const body = JSON.stringify(payload);
const signature = crypto
  .createHmac('sha256', callbackSecret)
  .update(body)
  .digest('hex');

// Headers sent with webhook
{
  'Content-Type': 'application/json',
  'X-TPZ-Signature': signature,       // HMAC for verification
  'X-TPZ-Timestamp': timestamp,       // Replay attack prevention
  'User-Agent': 'TranslatePressZone-Webhook/1.0',
  'X-Retry-Attempt': attemptNumber    // Only on retries
}
```

**WordPress verification**:
```php
// WordPress plugin receives webhook
$receivedSignature = $_SERVER['HTTP_X_TPZ_SIGNATURE'];
$body = file_get_contents('php://input');
$expectedSignature = hash_hmac('sha256', $body, $callbackSecret);

if (!hash_equals($expectedSignature, $receivedSignature)) {
    http_response_code(401);
    die('Invalid signature');
}
```

### 3. Retry Strategy with Exponential Backoff

**Configuration** (`api/src/config/index.ts`):
```typescript
{
  webhookMaxRetries: 5,           // Maximum retry attempts
  webhookRetryDelayMs: 1000      // Base delay in milliseconds
}
```

**Retry Schedule**:
```
Attempt 1: Immediate (initial delivery)
Attempt 2: 1000ms  delay (1s)
Attempt 3: 2000ms  delay (2s)
Attempt 4: 4000ms  delay (4s)
Attempt 5: 8000ms  delay (8s)
Attempt 6: 16000ms delay (16s)
```

**Formula**: `delay = baseDelay * (2 ^ (attemptNumber - 1))`

### 4. Webhook Payload Structure

**Event Types**:
- `translation.completed` - Job completed successfully
- `translation.failed` - Job failed with error

**Payload Schema** (from `api/src/types/index.ts`):
```typescript
interface WebhookPayload {
  event: 'translation.completed' | 'translation.failed';
  jobId: string;                    // Translation job UUID
  clientJobId?: string;             // WordPress post/page ID
  status: TranslationStatus;        // 'completed' or 'failed'
  translation?: string;             // Translated content (if completed)
  tokensUsed?: number;              // Tokens consumed (if completed)
  cost?: number;                    // Cost in USD (if completed)
  errorMessage?: string;            // Error details (if failed)
  processingTimeMs?: number;        // Processing time
  timestamp: string;                // ISO 8601 completion timestamp
}
```

**Example - Successful Translation**:
```json
{
  "event": "translation.completed",
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "clientJobId": "post_123",
  "status": "completed",
  "translation": "<p>Traduzido com sucesso!</p>",
  "tokensUsed": 1234,
  "cost": 0.00247,
  "processingTimeMs": 2456,
  "timestamp": "2026-01-27T12:34:56.789Z"
}
```

**Example - Failed Translation**:
```json
{
  "event": "translation.failed",
  "jobId": "550e8400-e29b-41d4-a716-446655440001",
  "clientJobId": "post_456",
  "status": "failed",
  "errorMessage": "Gemini API rate limit exceeded",
  "timestamp": "2026-01-27T12:35:10.123Z"
}
```

## Database Schema

### WebhookDelivery Model

**Table**: `webhook_deliveries`

```prisma
model WebhookDelivery {
  id             String   @id @default(uuid()) @db.Uuid
  job_id         String   @db.Uuid
  attempt_number Int      @default(1)
  success        Boolean
  http_status    Int?
  response_body  String?  @db.Text
  error_message  String?  @db.Text
  attempted_at   DateTime @default(now())

  // Relations
  job TranslationJob @relation(fields: [job_id], references: [id], onDelete: Cascade)

  @@index([job_id])
  @@index([attempted_at])
  @@map("webhook_deliveries")
}
```

**Indexes**:
- `job_id` - Fast lookup of all delivery attempts for a job
- `attempted_at` - Time-series analysis of webhook deliveries

**Data Retention**:
- `response_body` truncated to 1000 characters to prevent DB bloat
- Records persist until parent job is deleted (cascade delete)

## Service Implementation

### Core Functions (`api/src/services/webhookService.ts`)

#### 1. deliverWebhook()

**Purpose**: Deliver webhook notification to callback URL

**Signature**:
```typescript
async function deliverWebhook(
  jobId: string,
  payload: WebhookPayload,
  callbackUrl: string,
  callbackSecret: string
): Promise<WebhookDelivery>
```

**Logic**:
1. Serialize payload to JSON
2. Generate timestamp for replay attack prevention
3. Compute HMAC-SHA256 signature using `callbackSecret`
4. Prepare HTTP headers:
   - `Content-Type: application/json`
   - `X-TPZ-Signature: <hmac>`
   - `X-TPZ-Timestamp: <timestamp>`
   - `User-Agent: TranslatePressZone-Webhook/1.0`
5. POST to `callbackUrl` with 10-second timeout
6. Track metrics (success rate, delivery duration)
7. Record delivery attempt in `webhook_deliveries` table
8. Return delivery record

**Error Handling**:
- Axios errors → Extract HTTP status, response body, error message
- Network errors → Log timeout/connection details
- Database errors → Throw exception (critical failure)

**Timeout**: 10 seconds (prevents hanging on slow WordPress sites)

#### 2. retryFailedWebhook()

**Purpose**: Retry failed webhook with exponential backoff

**Signature**:
```typescript
async function retryFailedWebhook(
  jobId: string,
  attemptNumber: number
): Promise<WebhookDelivery>
```

**Logic**:
1. Validate `attemptNumber <= webhookMaxRetries`
2. Fetch job details from database (callback_url, callback_secret)
3. Reconstruct webhook payload from job data
4. Calculate exponential backoff delay:
   ```typescript
   const delayMs = config.webhookRetryDelayMs * Math.pow(2, attemptNumber - 1);
   ```
5. Wait for backoff period (`setTimeout`)
6. Attempt delivery with same logic as `deliverWebhook()`
7. Add `X-Retry-Attempt: <attemptNumber>` header
8. Record delivery attempt with `attempt_number = attemptNumber`

**Retry Trigger**:
- Called by worker process (BullMQ queue)
- Scheduled after initial delivery failure
- Continues until success or max retries exceeded

#### 3. getWebhookDeliveries()

**Purpose**: Fetch delivery history for a job

**Signature**:
```typescript
async function getWebhookDeliveries(jobId: string): Promise<WebhookDelivery[]>
```

**Returns**: Array of delivery attempts ordered by `attempt_number` ASC

**Use Case**: Admin panel debugging, webhook delivery troubleshooting

#### 4. shouldRetryWebhook()

**Purpose**: Check if webhook should be retried

**Signature**:
```typescript
async function shouldRetryWebhook(jobId: string): Promise<boolean>
```

**Returns**: `true` if retry is needed, `false` otherwise

**Logic**:
1. Fetch last delivery attempt for job
2. Return `false` if:
   - No delivery attempts exist
   - Last attempt succeeded
   - `attempt_number >= webhookMaxRetries`
3. Return `true` otherwise

#### 5. getNextRetryAttempt()

**Purpose**: Calculate next retry attempt number

**Signature**:
```typescript
async function getNextRetryAttempt(jobId: string): Promise<number | null>
```

**Returns**: Next attempt number (2-6) or `null` if max retries exceeded

**Logic**:
1. Fetch last delivery attempt
2. If no attempts exist, return `1` (initial delivery)
3. Calculate `nextAttempt = lastAttempt.attempt_number + 1`
4. Return `null` if `nextAttempt > webhookMaxRetries`
5. Return `nextAttempt` otherwise

## Security Considerations

### 1. SSRF Prevention

**Threat**: Attacker submits malicious `callbackUrl` pointing to internal services

**Mitigations**:
- **URL validation**: Ensure valid HTTP/HTTPS URLs only
- **Private IP blocking**: Reject localhost, 127.0.0.1, 10.x.x.x, 192.168.x.x, 172.16-31.x.x
- **DNS rebinding protection**: Resolve domain before HTTP request
- **Redirect following disabled**: Prevent redirect to internal IPs
- **Timeout enforcement**: 10-second timeout prevents slow loris attacks

**Implementation** (recommended pattern):
```typescript
// SSRF validation function (to be added)
function validateCallbackUrl(url: string): void {
  const parsed = new URL(url);
  
  // Only allow HTTP/HTTPS
  if (!['http:', 'https:'].includes(parsed.protocol)) {
    throw new Error('Invalid protocol. Only HTTP/HTTPS allowed.');
  }
  
  // Block private IPs
  const hostname = parsed.hostname;
  if (
    hostname === 'localhost' ||
    hostname === '127.0.0.1' ||
    hostname.startsWith('10.') ||
    hostname.startsWith('192.168.') ||
    /^172\.(1[6-9]|2[0-9]|3[0-1])\./.test(hostname)
  ) {
    throw new Error('Private IP addresses not allowed');
  }
  
  // Block metadata endpoints (cloud providers)
  if (hostname === '169.254.169.254') {
    throw new Error('Metadata endpoints not allowed');
  }
}
```

**Current Status**: URL validation not implemented in code (security gap)

**Recommendation**: Add validation before initial job submission AND before each retry

### 2. Replay Attack Prevention

**Mechanism**: `X-TPZ-Timestamp` header

**WordPress Verification**:
```php
$timestamp = $_SERVER['HTTP_X_TPZ_TIMESTAMP'];
$now = time() * 1000; // Convert to milliseconds
$maxAge = 5 * 60 * 1000; // 5 minutes

if (abs($now - $timestamp) > $maxAge) {
    http_response_code(400);
    die('Request too old');
}
```

**Window**: 5 minutes (prevents replay of old webhooks)

### 3. HMAC Signature Security

**Key Management**:
- `callbackSecret` stored encrypted in database (AES-256-GCM)
- Unique secret per WordPress site
- Minimum 32-character length enforced
- Generated with `crypto.randomBytes(32)`

**Timing-Safe Comparison**:
```php
// WordPress MUST use hash_equals() to prevent timing attacks
if (!hash_equals($expectedSignature, $receivedSignature)) {
    // NEVER use $expectedSignature === $receivedSignature
}
```

## Monitoring & Metrics

### Prometheus Metrics (tracked in `utils/metrics.ts`)

**Webhook Delivery Metrics**:
```typescript
// Success rate
webhook_delivery_total{status="success"}
webhook_delivery_total{status="failure"}

// Delivery duration histogram
webhook_delivery_duration_ms_bucket{le="1000"}
webhook_delivery_duration_ms_bucket{le="2000"}
webhook_delivery_duration_ms_bucket{le="5000"}
webhook_delivery_duration_ms_bucket{le="10000"}

// Retry attempts
webhook_retry_attempts_total{attempt="2"}
webhook_retry_attempts_total{attempt="3"}
webhook_retry_attempts_total{attempt="4"}
webhook_retry_attempts_total{attempt="5"}
webhook_retry_attempts_total{attempt="6"}

// Permanent failures (max retries exceeded)
webhook_permanent_failures_total
```

**Alerting Rules**:
```yaml
# High webhook failure rate
- alert: WebhookFailureRateHigh
  expr: rate(webhook_delivery_total{status="failure"}[5m]) > 0.1
  for: 5m
  annotations:
    summary: "Webhook failure rate above 10%"

# Webhook delivery timeouts
- alert: WebhookTimeouts
  expr: rate(webhook_delivery_duration_ms_bucket{le="10000"}[5m]) < 0.95
  for: 5m
  annotations:
    summary: "More than 5% of webhooks timing out"
```

### Logging Strategy

**All webhook operations logged with**:
- Job ID
- Callback URL (truncated for security)
- Attempt number
- HTTP status code
- Error message (if failed)
- Processing duration

**Log Levels**:
- `INFO` - Successful deliveries
- `WARN` - Retry attempts
- `ERROR` - Permanent failures (max retries exceeded)

**Example Log Entries**:
```json
{
  "level": "info",
  "message": "Webhook delivered successfully",
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "callbackUrl": "https://example.com/wp-json/...",
  "httpStatus": 200,
  "duration": 234
}

{
  "level": "warn",
  "message": "Retrying webhook delivery",
  "jobId": "550e8400-e29b-41d4-a716-446655440001",
  "attemptNumber": 3,
  "delayMs": 4000
}

{
  "level": "error",
  "message": "Webhook delivery permanently failed",
  "jobId": "550e8400-e29b-41d4-a716-446655440002",
  "attemptNumber": 6,
  "errorMessage": "Max retries exceeded"
}
```

## Integration with Translation Service

### Webhook Trigger Points

**1. Job Completion** (`api/src/services/translationService.ts`):
```typescript
// After successful translation
if (job.callback_url && job.callback_secret) {
  const payload: WebhookPayload = {
    event: 'translation.completed',
    jobId: job.id,
    clientJobId: job.client_job_id,
    status: 'completed',
    translation: translatedContent,
    tokensUsed: tokensUsed,
    cost: cost,
    processingTimeMs: duration,
    timestamp: new Date().toISOString()
  };
  
  await deliverWebhook(job.id, payload, job.callback_url, job.callback_secret);
}
```

**2. Job Failure** (after error):
```typescript
// After translation error
if (job.callback_url && job.callback_secret) {
  const payload: WebhookPayload = {
    event: 'translation.failed',
    jobId: job.id,
    clientJobId: job.client_job_id,
    status: 'failed',
    errorMessage: error.message,
    timestamp: new Date().toISOString()
  };
  
  await deliverWebhook(job.id, payload, job.callback_url, job.callback_secret);
}
```

### Async Queue Processing (BullMQ)

**Retry Queue** (`api/src/queue/webhookQueue.ts` - to be implemented):
```typescript
// Add failed webhook to retry queue
if (!delivery.success && delivery.attempt_number < config.webhookMaxRetries) {
  const nextAttempt = delivery.attempt_number + 1;
  const delayMs = config.webhookRetryDelayMs * Math.pow(2, nextAttempt - 1);
  
  await webhookQueue.add(
    'retry-webhook',
    { jobId: job.id, attemptNumber: nextAttempt },
    { delay: delayMs }
  );
}
```

**Worker Process**:
```typescript
webhookQueue.process('retry-webhook', async (job) => {
  const { jobId, attemptNumber } = job.data;
  
  try {
    const delivery = await retryFailedWebhook(jobId, attemptNumber);
    
    if (!delivery.success) {
      // Schedule next retry if not max attempts
      const shouldRetry = await shouldRetryWebhook(jobId);
      if (shouldRetry) {
        const nextAttempt = await getNextRetryAttempt(jobId);
        if (nextAttempt) {
          // Re-queue for next retry
          await webhookQueue.add(
            'retry-webhook',
            { jobId, attemptNumber: nextAttempt },
            { delay: config.webhookRetryDelayMs * Math.pow(2, nextAttempt - 1) }
          );
        }
      }
    }
  } catch (error) {
    logger.error('Webhook retry worker failed', { jobId, attemptNumber, error });
    throw error; // BullMQ will handle retry of worker itself
  }
});
```

## WordPress Plugin Integration

### Webhook Endpoint Setup

**WordPress Route** (`translate-press-zone/includes/api/webhooks.php`):
```php
add_action('rest_api_init', function() {
    register_rest_route('translate-press-zone/v1', '/webhook/callback', [
        'methods' => 'POST',
        'callback' => 'tpz_handle_webhook_callback',
        'permission_callback' => 'tpz_verify_webhook_signature',
    ]);
});
```

**Signature Verification**:
```php
function tpz_verify_webhook_signature(WP_REST_Request $request) {
    $signature = $request->get_header('X-TPZ-Signature');
    $timestamp = $request->get_header('X-TPZ-Timestamp');
    $body = $request->get_body();
    
    // Verify timestamp (prevent replay attacks)
    $now = time() * 1000;
    if (abs($now - $timestamp) > 300000) { // 5 minutes
        return new WP_Error('expired', 'Webhook timestamp expired', ['status' => 400]);
    }
    
    // Verify HMAC signature
    $callbackSecret = get_option('tpz_callback_secret');
    $expectedSignature = hash_hmac('sha256', $body, $callbackSecret);
    
    if (!hash_equals($expectedSignature, $signature)) {
        return new WP_Error('invalid_signature', 'Invalid webhook signature', ['status' => 401]);
    }
    
    return true;
}
```

**Callback Handler**:
```php
function tpz_handle_webhook_callback(WP_REST_Request $request) {
    $payload = $request->get_json_params();
    
    $jobId = sanitize_text_field($payload['jobId']);
    $clientJobId = sanitize_text_field($payload['clientJobId']);
    $event = sanitize_text_field($payload['event']);
    
    if ($event === 'translation.completed') {
        $translation = wp_kses_post($payload['translation']);
        $postId = (int) str_replace('post_', '', $clientJobId);
        
        // Update post with translated content
        wp_update_post([
            'ID' => $postId,
            'post_content' => $translation,
        ]);
        
        // Mark as translated
        update_post_meta($postId, '_tpz_translation_status', 'completed');
        update_post_meta($postId, '_tpz_job_id', $jobId);
        
        return new WP_REST_Response([
            'success' => true,
            'message' => 'Translation applied successfully',
        ], 200);
        
    } elseif ($event === 'translation.failed') {
        $errorMessage = sanitize_text_field($payload['errorMessage']);
        $postId = (int) str_replace('post_', '', $clientJobId);
        
        // Mark as failed
        update_post_meta($postId, '_tpz_translation_status', 'failed');
        update_post_meta($postId, '_tpz_error_message', $errorMessage);
        
        return new WP_REST_Response([
            'success' => true,
            'message' => 'Translation failure recorded',
        ], 200);
    }
    
    return new WP_REST_Response([
        'success' => false,
        'message' => 'Unknown event type',
    ], 400);
}
```

## Testing Strategy

### Unit Tests

**Test File**: `api/src/services/__tests__/webhookService.test.ts`

**Test Cases**:

```typescript
describe('deliverWebhook', () => {
  it('should deliver webhook successfully', async () => {
    // Mock axios.post to return 200 OK
    // Assert delivery record created with success=true
  });
  
  it('should handle HTTP errors', async () => {
    // Mock axios.post to return 500 error
    // Assert delivery record created with success=false
  });
  
  it('should handle network timeouts', async () => {
    // Mock axios.post to timeout after 10s
    // Assert delivery record contains timeout error
  });
  
  it('should truncate response_body to 1000 chars', async () => {
    // Mock axios.post to return 5000 char response
    // Assert delivery.response_body.length === 1000
  });
  
  it('should generate correct HMAC signature', async () => {
    // Capture signature from headers
    // Verify matches expected HMAC-SHA256
  });
});

describe('retryFailedWebhook', () => {
  it('should calculate correct exponential backoff', async () => {
    // Test attempt 2: 1000ms delay
    // Test attempt 3: 2000ms delay
    // Test attempt 4: 4000ms delay
  });
  
  it('should throw error if max retries exceeded', async () => {
    // Call with attemptNumber=6
    // Assert error thrown
  });
  
  it('should include X-Retry-Attempt header', async () => {
    // Mock axios.post
    // Assert header X-Retry-Attempt === '3'
  });
});

describe('shouldRetryWebhook', () => {
  it('should return false if last attempt succeeded', async () => {
    // Create delivery with success=true
    // Assert shouldRetryWebhook returns false
  });
  
  it('should return false if max retries exceeded', async () => {
    // Create delivery with attempt_number=5, success=false
    // Assert shouldRetryWebhook returns false
  });
  
  it('should return true if retry is needed', async () => {
    // Create delivery with attempt_number=2, success=false
    // Assert shouldRetryWebhook returns true
  });
});
```

### Integration Tests

**Test File**: `api/src/__tests__/integration/webhooks.test.ts`

**Test Cases**:

```typescript
describe('Webhook Integration', () => {
  it('should deliver webhook on job completion', async () => {
    // 1. Submit translation job with callbackUrl
    // 2. Wait for job to complete
    // 3. Mock webhook endpoint to capture request
    // 4. Assert webhook received with correct payload
    // 5. Assert signature verification passes
  });
  
  it('should retry failed webhooks', async () => {
    // 1. Submit job with callbackUrl pointing to failing endpoint
    // 2. Wait for initial delivery failure
    // 3. Mock endpoint to succeed on retry
    // 4. Assert retry succeeds with correct attempt_number
  });
  
  it('should stop retrying after max attempts', async () => {
    // 1. Submit job with callbackUrl pointing to always-failing endpoint
    // 2. Wait for all 5 retries to complete
    // 3. Assert no more retries scheduled
    // 4. Assert permanent failure logged
  });
});
```

### Manual Testing Checklist

**Webhook Delivery**:
- [ ] Successful webhook delivery returns 200-299 status
- [ ] Failed webhook creates retry job in queue
- [ ] HMAC signature generated correctly
- [ ] Timestamp header included in all requests
- [ ] User-Agent header set to `TranslatePressZone-Webhook/1.0`

**Retry Logic**:
- [ ] Exponential backoff delays calculated correctly
- [ ] Retry attempts increment sequentially (1, 2, 3, 4, 5)
- [ ] Max retries (5) enforced
- [ ] X-Retry-Attempt header included on retries
- [ ] Permanent failure logged after max retries

**Security**:
- [ ] HMAC signature verification works in WordPress
- [ ] Timestamp validation prevents replay attacks
- [ ] Private IP addresses rejected (SSRF prevention)
- [ ] Timeout enforced (10 seconds)
- [ ] Signature timing-safe comparison used

**Database**:
- [ ] Delivery attempts recorded in webhook_deliveries table
- [ ] Cascade delete removes deliveries when job deleted
- [ ] Response body truncated to 1000 characters
- [ ] Indexes on job_id and attempted_at exist

**Monitoring**:
- [ ] Prometheus metrics exported correctly
- [ ] Logs include all required fields
- [ ] Alerts trigger on high failure rate
- [ ] Webhook delivery duration histogram tracked

## Common Issues & Solutions

### Issue 1: WordPress Site Returns 403 Forbidden

**Symptoms**:
- Webhook delivery fails with HTTP 403
- WordPress security plugin blocks requests

**Root Cause**: Security plugins (Wordfence, iThemes Security) block requests from unknown User-Agent

**Solution**:
```php
// WordPress: Whitelist webhook User-Agent
add_filter('wordfence_is_allowed_user_agent', function($allowed, $userAgent) {
    if (strpos($userAgent, 'TranslatePressZone-Webhook') !== false) {
        return true;
    }
    return $allowed;
}, 10, 2);
```

### Issue 2: Webhook Timeouts on Large Translations

**Symptoms**:
- Webhooks timeout after 10 seconds
- WordPress site slow to respond

**Root Cause**: WordPress performs expensive operations (image processing, cache clearing) on post update

**Solution**:
```php
// WordPress: Defer expensive operations
add_action('tpz_translation_completed', function($postId, $translation) {
    // Queue expensive operations for later
    wp_schedule_single_event(time() + 60, 'tpz_process_translation', [$postId]);
    
    // Return response immediately
    return true;
}, 10, 2);
```

### Issue 3: Duplicate Webhook Deliveries

**Symptoms**:
- Same translation applied twice
- WordPress post updated multiple times

**Root Cause**: Webhook retries after transient network error

**Solution**:
```php
// WordPress: Idempotency check
function tpz_handle_webhook_callback(WP_REST_Request $request) {
    $jobId = $request['jobId'];
    
    // Check if already processed
    $processedJobId = get_transient('tpz_processed_webhook_' . $jobId);
    if ($processedJobId) {
        return new WP_REST_Response(['success' => true, 'message' => 'Already processed'], 200);
    }
    
    // Process webhook...
    
    // Mark as processed (expires in 24 hours)
    set_transient('tpz_processed_webhook_' . $jobId, true, DAY_IN_SECONDS);
    
    return new WP_REST_Response(['success' => true], 200);
}
```

### Issue 4: SSRF Attack Attempt

**Symptoms**:
- Webhook delivery to `http://localhost:9000/admin`
- Attempt to access internal services

**Root Cause**: Malicious user submits internal URL as callbackUrl

**Solution**:
```typescript
// Backend: Validate URL before job submission
function validateCallbackUrl(url: string): void {
  const parsed = new URL(url);
  
  if (!['http:', 'https:'].includes(parsed.protocol)) {
    throw new Error('Invalid protocol');
  }
  
  const hostname = parsed.hostname;
  const privateRanges = [
    'localhost',
    '127.0.0.1',
    '10.',
    '192.168.',
    /^172\.(1[6-9]|2[0-9]|3[0-1])\./
  ];
  
  for (const range of privateRanges) {
    if (typeof range === 'string' && hostname.startsWith(range)) {
      throw new Error('Private IP not allowed');
    }
    if (range instanceof RegExp && range.test(hostname)) {
      throw new Error('Private IP not allowed');
    }
  }
}
```

## Performance Optimization

### 1. Batch Webhook Deliveries

**Problem**: High-volume sites generate hundreds of webhooks per minute

**Solution**: Batch multiple completed jobs into single webhook

```typescript
// Batch webhook payload
interface BatchWebhookPayload {
  event: 'translation.batch_completed';
  jobs: Array<{
    jobId: string;
    clientJobId: string;
    translation: string;
    tokensUsed: number;
    cost: number;
  }>;
  timestamp: string;
}
```

### 2. Webhook Delivery Queue Priority

**Problem**: Critical webhooks delayed by retry queue

**Solution**: Use BullMQ priority levels

```typescript
// High priority for initial delivery
await webhookQueue.add('deliver-webhook', payload, {
  priority: 1 // Highest priority
});

// Lower priority for retries
await webhookQueue.add('retry-webhook', payload, {
  priority: 5, // Lower priority
  delay: delayMs
});
```

### 3. Connection Pooling

**Problem**: Creating new HTTP connections for each webhook is slow

**Solution**: Use Axios HTTP agent with connection pooling

```typescript
import http from 'http';
import https from 'https';

const httpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: 50
});

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 50
});

await axios.post(callbackUrl, payload, {
  headers,
  timeout: 10000,
  httpAgent,
  httpsAgent
});
```

## Verification Checklist

- [ ] HMAC signature generation implemented correctly
- [ ] Exponential backoff calculated with correct formula
- [ ] Max retry limit (5) enforced
- [ ] Timeout (10s) enforced on all requests
- [ ] SSRF validation blocks private IPs
- [ ] Replay attack prevention with timestamp validation
- [ ] Webhook delivery attempts recorded in database
- [ ] Response body truncated to 1000 characters
- [ ] Prometheus metrics tracked for all deliveries
- [ ] Structured logging includes all required fields
- [ ] WordPress plugin verifies HMAC signature
- [ ] WordPress plugin checks timestamp freshness
- [ ] Idempotency checks prevent duplicate processing
- [ ] Unit tests cover success/failure/timeout scenarios
- [ ] Integration tests verify end-to-end flow
- [ ] Alerts configured for high failure rate
- [ ] Documentation includes WordPress integration examples
- [ ] Security review completed for SSRF vulnerabilities

# Skill: Email Notifications (SendGrid)

## Identity
- **Skill ID**: `email-notifications-sendgrid`
- **Domain**: Email Service, Transactional Emails, User Notifications
- **Technologies**: SendGrid API, HTML Email Templates
- **Source Agent**: `backend-app-agent.md`

## When to Load This Skill

Load this skill when working on:
- Email notification features
- SendGrid integration
- User communication flows
- Transactional email templates
- Email delivery troubleshooting
- Graceful degradation for dev environments

**File patterns:**
- `api/src/services/emailService.ts`
- Email template modifications
- SendGrid configuration
- Email-related environment variables

## Core Patterns

### 1. Service Architecture

```typescript
// api/src/services/emailService.ts
import sgMail from '@sendgrid/mail';
import { config } from '../config';
import { logger } from '../utils/logger';

// Initialize SendGrid at module load
if (config.sendgridApiKey) {
  sgMail.setApiKey(config.sendgridApiKey);
} else if (config.nodeEnv === 'production') {
  logger.warn('SendGrid API key not configured - emails will fail in production');
}
```

**Key Characteristics:**
- **Singleton Pattern:** SendGrid client initialized once at module load
- **Graceful Degradation:** Fails silently in development if not configured
- **Stateless:** All methods are independent, no shared state
- **Non-Blocking:** Returns boolean success/failure (doesn't throw on send failures)

---

### 2. Configuration Requirements

**Environment Variables:**
```bash
# Required for email functionality
SENDGRID_API_KEY=SG.xxxxxxxxxxxxxxxxxxxx
SENDGRID_FROM_EMAIL=noreply@translate.press.zone
SENDGRID_FROM_NAME=TranslatePress.zone

# Used for constructing links in emails
FRONTEND_URL=https://translate.press.zone
```

**Configuration Validation:**
- If `SENDGRID_API_KEY` is missing:
  - **Development:** Logs warning, emails skipped, tokens logged to console
  - **Production:** Logs critical warning on startup, email sends will fail

---

### 3. Available Email Methods

#### 3.1 Welcome Email

**Function:** `sendWelcomeEmail(email: string, name: string): Promise<boolean>`

**Purpose:** Sent immediately after user registration to onboard new users.

**Content:**
- Greeting with user's name
- Quick start guide (Generate API key, Install plugin, Start translating)
- Call-to-action button to dashboard

**Template Variables:**
- `${name}` - User's display name
- `${config.frontendUrl}/dashboard` - Dashboard link

**Returns:** `true` if sent successfully, `false` if SendGrid not configured or send failed

**Example Usage:**
```typescript
import { sendWelcomeEmail } from '../services/emailService';

// After user registration
const user = await createUser(email, password);
await sendWelcomeEmail(user.email, user.name || 'there');
```

---

#### 3.2 Email Verification

**Function:** `sendVerificationEmail(email: string, token: string): Promise<boolean>`

**Purpose:** Sent after registration to verify user's email address (required before API access).

**Content:**
- Verification button with token link
- Fallback plain text link
- Security notice ("If you didn't create an account...")

**Template Variables:**
- `${config.frontendUrl}/verify-email?token=${token}` - Verification URL

**Token Handling:**
- **Production:** Sends email with token link
- **Development (no SendGrid):** Logs token to logger for manual verification

**Throws:** `Error('Failed to send verification email')` if SendGrid configured but send fails (critical for auth flow)

**Example Usage:**
```typescript
import { sendVerificationEmail } from '../services/emailService';
import crypto from 'crypto';

// Generate verification token
const token = crypto.randomBytes(32).toString('hex');
await prisma.user.update({
  where: { id: userId },
  data: {
    email_verification_token: token,
    email_verification_expires: new Date(Date.now() + 3600000) // 1 hour
  }
});

try {
  await sendVerificationEmail(user.email, token);
} catch (error) {
  logger.error('Critical: Verification email failed', { userId, error });
  // Handle failure (retry, queue, alert admin)
}
```

---

#### 3.3 Password Reset Email

**Function:** `sendPasswordResetEmail(email: string, token: string): Promise<boolean>`

**Purpose:** Sent when user requests password reset via "Forgot Password" flow.

**Content:**
- Password reset button with token link
- Expiration warning (⏰ 1 hour)
- Security notice ("If you didn't request this...")

**Template Variables:**
- `${config.frontendUrl}/reset-password?token=${token}` - Reset URL

**Token Handling:**
- **Production:** Sends email with token link
- **Development (no SendGrid):** Logs token to logger for testing

**Throws:** `Error('Failed to send password reset email')` if SendGrid configured but send fails

**Example Usage:**
```typescript
import { sendPasswordResetEmail } from '../services/emailService';
import crypto from 'crypto';

// Generate reset token
const token = crypto.randomBytes(32).toString('hex');
const hashedToken = crypto.createHash('sha256').update(token).digest('hex');

await prisma.user.update({
  where: { email },
  data: {
    password_reset_token: hashedToken,
    password_reset_expires: new Date(Date.now() + 3600000) // 1 hour
  }
});

try {
  await sendPasswordResetEmail(email, token); // Send plain token, store hashed
} catch (error) {
  logger.error('Password reset email failed', { email, error });
  throw error; // Propagate to controller
}
```

---

#### 3.4 Low Credit Warning

**Function:** `sendLowCreditWarning(email: string, creditsRemaining: number): Promise<boolean>`

**Purpose:** Sent when user's credit balance drops below a threshold (e.g., 10% remaining).

**Content:**
- Warning icon and headline
- Highlighted remaining balance (e.g., "5,000 tokens remaining")
- Call-to-action to upgrade plan

**Template Variables:**
- `${creditsRemaining.toLocaleString()}` - Formatted credit balance
- `${config.frontendUrl}/account/subscription` - Upgrade link

**Returns:** `true` if sent, `false` if failed (non-blocking)

**Example Usage:**
```typescript
import { sendLowCreditWarning } from '../services/emailService';
import { getCurrentBalance } from './creditService';

// After credit deduction
const balance = await getCurrentBalance(userId);
const user = await prisma.user.findUnique({ where: { id: userId } });

if (balance < user.credit_allocation * 0.1) { // Less than 10% remaining
  await sendLowCreditWarning(user.email, balance);
}
```

---

#### 3.5 Subscription Receipt

**Function:** `sendSubscriptionReceipt(email: string, amount: number, planTier: string): Promise<boolean>`

**Purpose:** Sent after successful PayPal subscription payment for record-keeping.

**Content:**
- Payment confirmation
- Receipt table (Plan, Amount, Date)
- Dashboard link to start using credits

**Template Variables:**
- `${planTier}` - Plan name (starter, professional, enterprise)
- `${amount.toFixed(2)}` - Payment amount in USD
- `${new Date().toLocaleDateString()}` - Current date

**Returns:** `true` if sent, `false` if failed (non-blocking)

**Example Usage:**
```typescript
import { sendSubscriptionReceipt } from '../services/emailService';

// PayPal webhook handler
async function handlePaymentCompleted(event: PayPalWebhookEvent) {
  const { amount, planTier, userEmail } = parsePayPalEvent(event);

  // Allocate credits
  await allocateCredits(userId, creditAmount, 'Subscription payment');

  // Send receipt
  await sendSubscriptionReceipt(userEmail, amount.value, planTier);
}
```

---

#### 3.6 Job Completion Notification

**Function:** `sendJobCompletionEmail(email: string, jobId: string, sourceLang: string, targetLang: string): Promise<boolean>`

**Purpose:** Sent when an async translation job finishes processing.

**Content:**
- Success checkmark and headline
- Language pair (e.g., "EN → ES")
- Job ID (truncated for readability)
- Notice that content is available

**Template Variables:**
- `${sourceLang.toUpperCase()}` - Source language code
- `${targetLang.toUpperCase()}` - Target language code
- `${jobId.substring(0, 16)}...` - Truncated job ID

**Returns:** `true` if sent, `false` if failed (non-blocking)

**Example Usage:**
```typescript
import { sendJobCompletionEmail } from '../services/emailService';

// Translation worker (after job completes)
async function processTranslationJob(job: Job) {
  const { userId, jobId, sourceLang, targetLang } = job.data;

  // Perform translation
  const result = await translateWithGemini(content, sourceLang, targetLang);

  // Update job status
  await prisma.translationJob.update({
    where: { id: jobId },
    data: { status: 'completed', result, completed_at: new Date() }
  });

  // Notify user
  const user = await prisma.user.findUnique({ where: { id: userId } });
  await sendJobCompletionEmail(user.email, jobId, sourceLang, targetLang);
}
```

---

### 4. Email Template Design

**Design System:**
- **Max Width:** 600px (optimal for email clients)
- **Font:** Arial, sans-serif (universal compatibility)
- **Colors:**
  - Primary Blue: `#3b82f6`
  - Success Green: `#10b981`
  - Warning Orange: `#f59e0b`
  - Error Red: `#dc2626`
  - Gray Text: `#64748b`, `#999`

**Template Structure:**
```html
<div style="font-family: Arial, sans-serif; max-width: 600px; margin: 0 auto;">
  <h2>Email Heading</h2>
  <p>Body content with clear messaging.</p>

  <!-- Call-to-Action Button -->
  <a href="URL" style="display: inline-block; background-color: #3b82f6; color: white; padding: 12px 24px; text-decoration: none; border-radius: 6px; margin: 20px 0;">
    Button Text
  </a>

  <!-- Footer Notice -->
  <p style="color: #999; font-size: 12px; margin-top: 30px;">
    Footer or disclaimer text.
  </p>
</div>
```

**Best Practices:**
- **Inline Styles:** All CSS must be inline (no `<style>` tags or external CSS)
- **Plain Text Version:** Every email includes both HTML and plain text
- **Responsive:** Works on mobile and desktop clients
- **Accessibility:** Clear hierarchy, readable font sizes (≥14px body, ≥20px headings)

---

### 5. Error Handling Patterns

#### Non-Critical Emails (Welcome, Low Credit, Receipt, Job Completion)
```typescript
export async function sendWelcomeEmail(email: string, name: string): Promise<boolean> {
  try {
    if (!config.sendgridApiKey) {
      logger.warn('SendGrid not configured, skipping welcome email', { email });
      return false; // Fail silently
    }

    await sgMail.send(msg);
    logger.info('Welcome email sent', { email, name });
    return true;
  } catch (error) {
    logger.error('Failed to send welcome email', { error, email });
    return false; // Don't throw, just log and return false
  }
}
```

**Rationale:** These emails are "nice-to-have" notifications. Failures shouldn't block critical flows.

---

#### Critical Emails (Verification, Password Reset)
```typescript
export async function sendVerificationEmail(email: string, token: string): Promise<boolean> {
  try {
    if (!config.sendgridApiKey) {
      logger.warn('SendGrid not configured, skipping verification email', { email });
      logger.info(`Verification token: ${token}`, { email }); // Log for dev testing
      return false;
    }

    await sgMail.send(msg);
    logger.info('Verification email sent', { email, verificationUrl });
    return true;
  } catch (error) {
    logger.error('Failed to send verification email', { error, email });
    throw new Error('Failed to send verification email'); // Throw to block flow
  }
}
```

**Rationale:** These emails are required for security flows. Failures must surface to the caller for retry logic or user notification.

---

### 6. Development Mode Graceful Degradation

**Behavior When SendGrid Not Configured:**

```typescript
// Development environment without SendGrid
if (!config.sendgridApiKey) {
  logger.warn('SendGrid not configured, skipping verification email', { email });
  logger.info(`Verification token: ${token}`, { email }); // Token logged for manual use
  return false;
}
```

**Benefits:**
- Developers can test flows without SendGrid account
- Tokens/links logged to console for manual testing
- No crashes or exceptions due to missing configuration

**Production Behavior:**
```typescript
// On server startup
if (config.nodeEnv === 'production' && !config.sendgridApiKey) {
  logger.warn('SendGrid API key not configured - emails will fail in production');
  // Warning logged, but server starts (allows troubleshooting)
}
```

---

### 7. Integration Points

#### Called By Routes
```typescript
// api/src/routes/auth.ts
import { sendWelcomeEmail, sendVerificationEmail, sendPasswordResetEmail } from '../services/emailService';

router.post('/register', async (req, res) => {
  const user = await createUser(req.body.email, req.body.password);
  await sendWelcomeEmail(user.email, user.name);
  await sendVerificationEmail(user.email, user.email_verification_token);
  res.json({ success: true });
});
```

#### Called By Services
```typescript
// api/src/services/paypalService.ts
import { sendSubscriptionReceipt } from './emailService';

async function handleSubscriptionActivated(event) {
  await allocateCredits(userId, amount);
  await sendSubscriptionReceipt(user.email, planAmount, planTier);
}
```

#### Called By Workers
```typescript
// api/src/worker.ts
import { sendJobCompletionEmail } from './services/emailService';

translationQueue.process(async (job) => {
  const result = await translateContent(job.data);
  await sendJobCompletionEmail(user.email, job.id, sourceLang, targetLang);
});
```

---

### 8. Testing Strategies

#### Unit Testing with Mocks
```typescript
// __tests__/unit/services/emailService.test.ts
import * as emailService from '../../../services/emailService';
import sgMail from '@sendgrid/mail';

jest.mock('@sendgrid/mail');

describe('emailService', () => {
  beforeEach(() => {
    jest.clearAllMocks();
  });

  it('should send welcome email successfully', async () => {
    (sgMail.send as jest.Mock).mockResolvedValue([{ statusCode: 202 }]);

    const result = await emailService.sendWelcomeEmail('test@example.com', 'John');

    expect(result).toBe(true);
    expect(sgMail.send).toHaveBeenCalledWith(
      expect.objectContaining({
        to: 'test@example.com',
        subject: 'Welcome to TranslatePress.zone!'
      })
    );
  });

  it('should handle SendGrid failures gracefully', async () => {
    (sgMail.send as jest.Mock).mockRejectedValue(new Error('SendGrid API error'));

    const result = await emailService.sendWelcomeEmail('test@example.com', 'John');

    expect(result).toBe(false);
  });
});
```

#### Manual Testing in Development
```bash
# Option 1: Use SendGrid sandbox mode (doesn't actually send)
SENDGRID_API_KEY=<sandbox-key>
SENDGRID_FROM_EMAIL=test@example.com

# Option 2: Skip SendGrid entirely (logs tokens to console)
# Just omit SENDGRID_API_KEY from .env
```

#### Testing Email Rendering
- **Litmus:** Upload HTML to test rendering across clients
- **Email on Acid:** Automated cross-client testing
- **Manual:** Send test emails to Gmail, Outlook, Apple Mail

---

### 9. Production Considerations

#### Rate Limiting
SendGrid enforces rate limits based on plan:
- **Free Tier:** 100 emails/day
- **Essentials:** 40,000 emails/month
- **Pro:** 100,000+ emails/month

**Recommendation:** Monitor `sendgrid_emails_sent` metric, set alerts for 80% of quota.

#### Deliverability
- **SPF/DKIM/DMARC:** Configure DNS records for sender domain
- **Sender Reputation:** Monitor bounce rates (<5%), spam reports (<0.1%)
- **Unsubscribe Links:** Not implemented (transactional emails exempt)

#### Monitoring
```typescript
// Add SendGrid webhook handler (optional)
router.post('/webhooks/sendgrid', async (req, res) => {
  const events = req.body;

  for (const event of events) {
    if (event.event === 'bounce' || event.event === 'dropped') {
      logger.warn('Email delivery failed', { event, email: event.email });
      // Update user email status in database
    }
  }

  res.sendStatus(200);
});
```

---

### 10. Troubleshooting Guide

| Issue | Diagnosis | Solution |
|-------|-----------|----------|
| Emails not sending | Check `SENDGRID_API_KEY` in .env | Verify key is valid, check SendGrid dashboard |
| Wrong "from" address | Check `SENDGRID_FROM_EMAIL` | Must match verified sender in SendGrid |
| Template rendering issues | Inspect HTML in email client | Use inline styles, avoid `<style>` tags |
| Tokens not working (dev) | SendGrid not configured | Check logs for `logger.info(\`Verification token: ...\`)` |
| Rate limit exceeded | Too many emails sent | Upgrade SendGrid plan, implement queue throttling |
| Emails in spam | SPF/DKIM not configured | Configure DNS records, warm up sender reputation |

---

### 11. Future Enhancements

**Planned Improvements:**
1. **Email Templates as Files:** Move HTML templates to separate files (`.ejs`, `.handlebars`)
2. **Template Localization:** Multi-language email support based on user locale
3. **Email Queue:** Retry logic with Bull queue for failed sends
4. **Unsubscribe Management:** Database table + unsubscribe links (for marketing emails)
5. **Email Analytics:** Track open rates, click-through rates (SendGrid webhooks)
6. **Batch Sending:** SendGrid batch API for bulk notifications

**Not Planned:**
- Marketing email campaigns (use dedicated ESP like Mailchimp)
- Newsletter functionality (out of scope for transactional API)

---

## Examples & Patterns

### Complete Registration Flow with Emails
```typescript
// api/src/routes/auth.ts
import { sendWelcomeEmail, sendVerificationEmail } from '../services/emailService';
import { hashPassword } from '../utils/encryption';
import crypto from 'crypto';

router.post('/register', async (req, res) => {
  const { email, password } = req.body;

  // 1. Create user
  const hashedPassword = await hashPassword(password);
  const verificationToken = crypto.randomBytes(32).toString('hex');

  const user = await prisma.user.create({
    data: {
      email,
      password_hash: hashedPassword,
      email_verification_token: verificationToken,
      email_verification_expires: new Date(Date.now() + 86400000), // 24 hours
      status: 'active',
      plan: 'starter'
    }
  });

  // 2. Send emails (non-blocking)
  await Promise.allSettled([
    sendWelcomeEmail(user.email, user.name || 'there'),
    sendVerificationEmail(user.email, verificationToken)
  ]);

  // 3. Return success (even if emails failed)
  res.json({
    success: true,
    message: 'Account created. Please check your email to verify your account.'
  });
});
```

### Async Job Completion with Notification
```typescript
// api/src/worker.ts
import { sendJobCompletionEmail } from './services/emailService';
import { translateContent } from './services/translationService';

translationQueue.process('translation-job', async (job) => {
  const { userId, jobId, content, sourceLang, targetLang } = job.data;

  try {
    // 1. Perform translation
    const result = await translateContent(content, sourceLang, targetLang);

    // 2. Update job in database
    await prisma.translationJob.update({
      where: { id: jobId },
      data: {
        status: 'completed',
        result,
        completed_at: new Date()
      }
    });

    // 3. Get user email
    const user = await prisma.user.findUnique({
      where: { id: userId },
      select: { email: true }
    });

    // 4. Send completion email
    if (user) {
      await sendJobCompletionEmail(user.email, jobId, sourceLang, targetLang);
    }

    return { success: true };
  } catch (error) {
    logger.error('Translation job failed', { jobId, error });
    throw error; // Job will be retried by Bull
  }
});
```

---

This email service provides reliable, well-designed transactional emails with graceful degradation for development environments and comprehensive error handling for production use.

---

---

# Skill 22: ML Accuracy Feedback System

## Identity
- **Skill ID**: `ml-accuracy-tracking`
- **Domain**: Machine Learning, Token Estimation Feedback Loop
- **Technologies**: PostgreSQL, Prisma ORM, Statistical Analysis
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Token estimation accuracy improvements
- ML feedback loop implementation
- Translation accuracy tracking
- Language-specific adjustment factors
- Confidence level calculations
- Statistical analysis of estimation errors
- Continuous learning from production data

**File patterns:**
- `api/src/services/AccuracyTracker.ts`
- `api/src/services/TokenEstimator.ts` (Layer 5 integration)
- `api/prisma/schema.prisma` (accuracy_stats, translation_records)

## Overview

The **ML Accuracy Feedback System** is a self-improving token estimation mechanism that learns from actual translation results. Every completed translation feeds data back into the system, allowing it to calculate language-specific adjustment factors and continuously improve estimation accuracy.

### Key Capabilities

1. **Record Actual Usage** - Store estimated vs. actual token counts
2. **Calculate Accuracy Ratios** - Analyze per-language estimation error
3. **Update ML Adjustments** - Write learned ratios to `accuracy_stats` table
4. **Confidence Levels** - Determine reliability based on sample size
5. **Outlier Detection** - Clamp adjustment ratios to prevent anomalies
6. **Continuous Learning** - Improve with every translation

### Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                    Translation Lifecycle                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 1: Estimation (Before Translation)                        │
├─────────────────────────────────────────────────────────────────┤
│ TokenEstimator.estimateSingle()                                 │
│  ├─ Layer 1: Base tokens (char_count ÷ 4)                       │
│  ├─ Layer 2: Language factor (0.60x - 1.28x)                    │
│  ├─ Layer 3: HTML complexity (0-15%)                            │
│  ├─ Layer 4: Safety buffer (5-8%)                               │
│  └─ Layer 5: ML adjustment ◄─── FROM ACCURACY_STATS TABLE      │
│                                                                  │
│ Output: estimated_tokens = 1,234                                │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 2: Translation (Gemini API)                                │
├─────────────────────────────────────────────────────────────────┤
│ Gemini API processes translation                                │
│ Returns actual_tokens = 1,100                                   │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 3: Feedback (After Translation)                            │
├─────────────────────────────────────────────────────────────────┤
│ AccuracyTracker.recordUsage()                                   │
│  ├─ Insert into translation_records                             │
│  │   (target_lang, estimated_tokens, actual_tokens,             │
│  │    html_complexity, timestamp)                               │
│  │                                                               │
│  └─ Trigger updateMLAdjustment()                                │
│      ├─ Query last 100 translations for this language           │
│      ├─ Calculate avg_ratio = AVG(actual / estimated)           │
│      ├─ Clamp to 0.80 - 1.20 range                              │
│      └─ UPDATE accuracy_stats SET ml_adjustment_ratio           │
│                                                                  │
│ Result: French adjustment updated to 1.05x                      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 4: Next Estimation (Improved)                              │
├─────────────────────────────────────────────────────────────────┤
│ TokenEstimator.getMLAdjustment('fr')                            │
│  └─ Returns 1.05x (learned from 100+ samples)                   │
│                                                                  │
│ Future French translations now 5% more accurate!                │
└─────────────────────────────────────────────────────────────────┘
```

## Database Schema

### Table: `translation_records`

Stores individual translation outcomes for ML training.

```sql
CREATE TABLE translation_records (
  id                BIGSERIAL PRIMARY KEY,
  target_lang       VARCHAR(10) NOT NULL,
  estimated_tokens  INTEGER NOT NULL,
  actual_tokens     INTEGER NOT NULL,
  html_complexity   VARCHAR(10) DEFAULT 'none',  -- none|light|medium|heavy
  created_at        TIMESTAMP DEFAULT NOW(),
  
  INDEX idx_target_lang (target_lang),
  INDEX idx_created_at (created_at)
);
```

**Purpose:**
- Store raw translation data for analysis
- Feed ML adjustment calculations
- Enable historical accuracy audits

**Retention Policy:**
- Keep last 100 records per language for active calculations
- Archive older records for long-term analysis (optional)

### Table: `accuracy_stats`

Stores aggregated ML adjustment ratios per language.

```sql
CREATE TABLE accuracy_stats (
  id                   SERIAL PRIMARY KEY,
  language             VARCHAR(10) UNIQUE NOT NULL,
  sample_count         INTEGER DEFAULT 0,
  avg_error_percent    DECIMAL(5, 2) DEFAULT 0.00,  -- e.g., -8.45 (under-estimated 8.45%)
  ml_adjustment_ratio  DECIMAL(5, 4) DEFAULT 1.0000, -- e.g., 1.0845
  updated_at           TIMESTAMP DEFAULT NOW(),
  
  INDEX idx_language (language),
  INDEX idx_sample_count (sample_count)
);
```

**Purpose:**
- Store learned adjustment factors
- Provide fast lookup for Layer 5 (ML Adjustment)
- Track confidence via sample count

**Key Fields:**
- `language` - ISO language code (e.g., 'en', 'fr', 'es')
- `sample_count` - Number of translations analyzed
- `avg_error_percent` - Average estimation error (positive = over-estimated)
- `ml_adjustment_ratio` - Multiplier to apply (1.0 = no adjustment)

## Core Implementation

### File: `api/src/services/AccuracyTracker.ts`

```typescript
/**
 * Accuracy Tracker Service
 * 
 * Learns from actual translation results to improve estimation accuracy
 */

import { PrismaClient } from '@prisma/client';
import { logger } from '../utils/logger';

const prisma = new PrismaClient();

export class AccuracyTracker {
  
  /**
   * Record a completed translation for ML learning
   * 
   * Called after every successful translation to feed the learning loop
   * 
   * @param targetLang Target language code (e.g., 'fr', 'es')
   * @param estimatedTokens Pre-translation estimate
   * @param actualTokens Actual tokens consumed by Gemini
   * @param htmlComplexity HTML complexity level
   */
  async recordUsage(
    targetLang: string,
    estimatedTokens: number,
    actualTokens: number,
    htmlComplexity: 'none' | 'light' | 'medium' | 'heavy'
  ): Promise<void> {
    try {
      // Step 1: Insert record into translation_records
      await prisma.$executeRaw`
        INSERT INTO translation_records 
        (target_lang, estimated_tokens, actual_tokens, html_complexity)
        VALUES (${targetLang}, ${estimatedTokens}, ${actualTokens}, ${htmlComplexity})
      `;
      
      // Step 2: Trigger ML adjustment update
      await this.updateMLAdjustment(targetLang);
      
      // Step 3: Log the learning event
      const error = actualTokens - estimatedTokens;
      const errorPercent = ((error / estimatedTokens) * 100).toFixed(2);
      
      logger.info('Recorded translation for ML learning', {
        targetLang,
        estimatedTokens,
        actualTokens,
        error,
        errorPercent: `${errorPercent}%`,
        htmlComplexity,
      });
    } catch (error) {
      logger.error('Failed to record translation usage', { error });
      // Non-blocking: Don't fail translation if ML tracking fails
    }
  }
  
  /**
   * Update ML adjustment ratio based on historical data
   * 
   * Analyzes last 100 translations to calculate average error
   * and update the adjustment ratio for future estimates
   * 
   * @param targetLang Language to update adjustment for
   */
  async updateMLAdjustment(targetLang: string): Promise<void> {
    try {
      // Query last 100 records for this language
      const stats = await prisma.$queryRaw<Array<{
        sample_count: number;
        avg_error_percent: number;
        avg_ratio: number;
      }>>`
        SELECT 
          COUNT(*) as sample_count,
          AVG((actual_tokens - estimated_tokens) / estimated_tokens * 100) as avg_error_percent,
          AVG(actual_tokens / estimated_tokens) as avg_ratio
        FROM (
          SELECT estimated_tokens, actual_tokens
          FROM translation_records
          WHERE target_lang = ${targetLang}
          ORDER BY created_at DESC
          LIMIT 100
        ) recent
      `;
      
      if (!stats || stats.length === 0) return;
      
      const { sample_count, avg_error_percent, avg_ratio } = stats[0];
      
      // Calculate ML adjustment ratio
      // If avg_ratio = 1.08, it means we're under-estimating by 8%
      let mlAdjustment = avg_ratio;
      
      // Outlier detection: Clamp to reasonable range (0.80 - 1.20)
      // Prevents anomalies from skewing future estimates
      mlAdjustment = Math.max(0.80, Math.min(1.20, mlAdjustment));
      
      // Upsert accuracy_stats (MySQL syntax)
      await prisma.$executeRaw`
        INSERT INTO accuracy_stats (language, sample_count, avg_error_percent, ml_adjustment_ratio)
        VALUES (${targetLang}, ${sample_count}, ${avg_error_percent}, ${mlAdjustment})
        ON DUPLICATE KEY UPDATE
          sample_count = ${sample_count},
          avg_error_percent = ${avg_error_percent},
          ml_adjustment_ratio = ${mlAdjustment},
          updated_at = CURRENT_TIMESTAMP
      `;
      
      logger.info('Updated ML adjustment', {
        targetLang,
        sample_count,
        avg_error_percent: avg_error_percent.toFixed(2) + '%',
        mlAdjustment,
      });
    } catch (error) {
      logger.error('Failed to update ML adjustment', { targetLang, error });
    }
  }
  
  /**
   * Get accuracy statistics for one or all languages
   * 
   * Returns current state of ML learning with confidence levels
   * 
   * @param lang Optional language filter (returns all if omitted)
   * @returns Array of language stats with status
   */
  async getAccuracyStats(lang?: string): Promise<Array<{
    language: string;
    sample_count: number;
    avg_error_percent: number;
    ml_adjustment_ratio: number;
    status: 'learning' | 'trained' | 'accurate';
  }>> {
    try {
      let stats;
      
      if (lang) {
        // Get stats for specific language
        stats = await prisma.$queryRaw<Array<{
          language: string;
          sample_count: number;
          avg_error_percent: number;
          ml_adjustment_ratio: number;
        }>>`
          SELECT language, sample_count, avg_error_percent, ml_adjustment_ratio
          FROM accuracy_stats
          WHERE language = ${lang}
          ORDER BY sample_count DESC
        `;
      } else {
        // Get all languages with samples
        stats = await prisma.$queryRaw<Array<{
          language: string;
          sample_count: number;
          avg_error_percent: number;
          ml_adjustment_ratio: number;
        }>>`
          SELECT language, sample_count, avg_error_percent, ml_adjustment_ratio
          FROM accuracy_stats
          WHERE sample_count > 0
          ORDER BY sample_count DESC
        `;
      }
      
      // Add training status based on sample count and error
      return stats.map(stat => ({
        ...stat,
        status: this.getStatus(stat.sample_count, stat.avg_error_percent),
      }));
    } catch (error) {
      logger.error('Failed to get accuracy stats', { error });
      return [];
    }
  }
  
  /**
   * Determine training status based on sample size and accuracy
   * 
   * @param sampleCount Number of translations analyzed
   * @param avgError Average estimation error percentage
   * @returns Status level
   */
  private getStatus(
    sampleCount: number, 
    avgError: number
  ): 'learning' | 'trained' | 'accurate' {
    if (sampleCount < 10) {
      return 'learning';   // Low confidence: Not enough data
    }
    
    if (sampleCount < 100 || Math.abs(avgError) > 10) {
      return 'trained';    // Medium confidence: Some data, but high error
    }
    
    return 'accurate';     // High confidence: Lots of data + low error
  }
}

export const accuracyTracker = new AccuracyTracker();
```

## Integration with TokenEstimator (Layer 5)

### File: `api/src/services/TokenEstimator.ts`

```typescript
/**
 * Layer 5: ML adjustment from historical data
 * 
 * Retrieves learned adjustment ratio from accuracy_stats table
 * Only applies adjustment if sample_count >= 10 (medium/high confidence)
 */
private async getMLAdjustment(targetLang: string): Promise<number> {
  try {
    const stats = await prisma.$queryRaw<Array<{ 
      ml_adjustment_ratio: number 
    }>>`
      SELECT ml_adjustment_ratio 
      FROM accuracy_stats 
      WHERE language = ${targetLang} 
      AND sample_count >= 10
      LIMIT 1
    `;
    
    if (stats && stats.length > 0) {
      return stats[0].ml_adjustment_ratio || 1.00;
    }
  } catch (error) {
    logger.warn('Failed to get ML adjustment', { targetLang, error });
  }
  
  return 1.00; // Default: No adjustment (neutral)
}

/**
 * Main estimation function
 * 
 * Applies Layer 5 (ML adjustment) to final token estimate
 */
async estimateSingle(
  content: string,
  _sourceLang: string,
  targetLang: string
): Promise<TokenEstimate> {
  // Layers 1-4: Base calculation
  const baseTokens = this.estimateBaseTokens(content);
  const langFactor = this.getLanguageFactor(targetLang);
  const complexity = this.analyzeHTMLComplexity(content);
  const languageAdjusted = Math.ceil(baseTokens * langFactor);
  const complexityTokens = Math.ceil(languageAdjusted * complexity.overhead);
  const safetyBuffer = this.calculateSafetyBuffer(languageAdjusted, complexity);
  
  const subtotal = languageAdjusted + complexityTokens + safetyBuffer;
  
  // Layer 5: ML adjustment (learned from production data)
  const mlAdjustment = await this.getMLAdjustment(targetLang);
  const finalTokens = Math.ceil(subtotal * mlAdjustment);
  
  // Determine confidence level
  const sampleCount = await this.getSampleCount(targetLang);
  let confidence: 'low' | 'medium' | 'high' = 'low';
  if (sampleCount >= 100) confidence = 'high';
  else if (sampleCount >= 10) confidence = 'medium';
  
  return {
    estimated_tokens: finalTokens,
    confidence,
    breakdown: {
      base: baseTokens,
      languageFactor: languageAdjusted - baseTokens,
      htmlComplexity: complexityTokens,
      safetyBuffer,
      mlAdjustment: finalTokens - subtotal,
    },
  };
}
```

## Confidence Levels

The system provides three confidence levels based on sample size:

| Confidence | Sample Count | Meaning |
|------------|--------------|---------|
| **Low** | 0-9 translations | Insufficient data, using default language factors |
| **Medium** | 10-99 translations | Some learning applied, moderate confidence |
| **High** | 100+ translations | Significant learning applied, high confidence |

### Confidence Level Logic

```typescript
/**
 * Get sample count for a language (for confidence calculation)
 */
private async getSampleCount(language: string): Promise<number> {
  try {
    const result = await prisma.$queryRaw<Array<{ sample_count: number }>>`
      SELECT sample_count 
      FROM accuracy_stats 
      WHERE language = ${language}
      LIMIT 1
    `;
    
    return result && result.length > 0 ? result[0].sample_count : 0;
  } catch {
    return 0;
  }
}
```

### Confidence in API Responses

```typescript
// GET /v1/pricing/estimate response
{
  "target_lang": "fr",
  "estimated_tokens": 1234,
  "confidence": "high",  // Based on 150+ French translations analyzed
  "cost_in_credits": 123,
  "breakdown": {
    "base": 1000,
    "languageFactor": 50,
    "htmlComplexity": 80,
    "safetyBuffer": 50,
    "mlAdjustment": 54  // +4.3% learned adjustment
  }
}
```

## Outlier Detection

### Ratio Clamping

To prevent anomalies (e.g., extremely short/long translations) from skewing future estimates, adjustment ratios are clamped to a safe range:

```typescript
// Clamp to 0.80 - 1.20 range (±20% adjustment maximum)
mlAdjustment = Math.max(0.80, Math.min(1.20, mlAdjustment));
```

**Why clamp?**
- **0.80 lower bound** - Prevents under-estimation by more than 20%
- **1.20 upper bound** - Prevents over-estimation by more than 20%
- **Protects against outliers** - Single anomalous translation won't corrupt future estimates

**Example outlier scenario:**
```typescript
// User translates: "OK" → "D'accord" (very short content)
// Estimated: 5 tokens
// Actual: 15 tokens (Gemini adds context tokens)
// Ratio: 3.0x (outlier!)

// Without clamping: mlAdjustment = 3.0x → Future estimates 3x too high
// With clamping: mlAdjustment = 1.2x → Safe adjustment applied
```

## Learning Workflow

### Complete Flow (From Estimation to Feedback)

```
┌──────────────────────────────────────────────────────────────┐
│ 1. User Submits Translation Request                         │
├──────────────────────────────────────────────────────────────┤
│ POST /v1/translate                                           │
│ {                                                            │
│   "content": "<p>Hello world</p>",                           │
│   "source_lang": "en",                                       │
│   "target_lang": "fr"                                        │
│ }                                                            │
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 2. Token Estimation (Pre-Translation)                       │
├──────────────────────────────────────────────────────────────┤
│ TokenEstimator.estimateSingle('en', 'fr', content)           │
│                                                              │
│ Layer 1: Base tokens = 50                                   │
│ Layer 2: Language factor (French 1.08x) = 54                │
│ Layer 3: HTML complexity (light 5%) = 3                     │
│ Layer 4: Safety buffer (5%) = 3                             │
│ Layer 5: ML adjustment (French 1.05x) = 3                   │
│                                                              │
│ TOTAL ESTIMATE: 63 tokens                                   │
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 3. Credit Check                                             │
├──────────────────────────────────────────────────────────────┤
│ creditService.hasSufficientCredits(userId, 63)               │
│ → TRUE, proceed                                              │
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 4. Translation (Gemini API)                                 │
├──────────────────────────────────────────────────────────────┤
│ geminiClient.translate(content, 'en', 'fr')                  │
│                                                              │
│ Gemini returns:                                              │
│ {                                                            │
│   "translation": "<p>Bonjour le monde</p>",                  │
│   "tokens_used": 58                                          │
│ }                                                            │
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 5. ML Feedback (Post-Translation)                           │
├──────────────────────────────────────────────────────────────┤
│ accuracyTracker.recordUsage('fr', 63, 58, 'light')           │
│                                                              │
│ Step A: Insert into translation_records                     │
│   INSERT INTO translation_records                           │
│   (target_lang, estimated_tokens, actual_tokens,            │
│    html_complexity)                                          │
│   VALUES ('fr', 63, 58, 'light')                             │
│                                                              │
│ Step B: Update ML adjustment                                │
│   SELECT AVG(actual / estimated) FROM last 100 records       │
│   → avg_ratio = 1.03 (we've been over-estimating 3%)        │
│                                                              │
│   UPDATE accuracy_stats                                      │
│   SET ml_adjustment_ratio = 1.03,                            │
│       sample_count = 101,                                    │
│       avg_error_percent = -3.0                               │
│   WHERE language = 'fr'                                      │
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 6. Credit Deduction (Actual Usage)                          │
├──────────────────────────────────────────────────────────────┤
│ creditService.deductCredits(userId, 58, 'Translation', jobId)│
└──────────────────────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ 7. Next French Translation (Improved Estimate)              │
├──────────────────────────────────────────────────────────────┤
│ Layer 5 now uses: ml_adjustment_ratio = 1.03                │
│ Future estimates are 3% more accurate!                      │
└──────────────────────────────────────────────────────────────┘
```

## API Integration

### Recording Translations (Internal)

```typescript
// In api/src/routes/translate.ts or api/src/services/translationService.ts

async function processTranslation(
  content: string,
  sourceLang: string,
  targetLang: string,
  userId: string
): Promise<TranslationResult> {
  
  // Step 1: Estimate tokens
  const estimate = await tokenEstimator.estimateSingle(content, sourceLang, targetLang);
  const estimatedTokens = estimate.estimated_tokens;
  
  // Step 2: Check credits
  const hasFunds = await creditService.hasSufficientCredits(userId, estimatedTokens);
  if (!hasFunds) {
    throw new Error('Insufficient credits');
  }
  
  // Step 3: Translate with Gemini
  const result = await geminiClient.translate(content, sourceLang, targetLang);
  const actualTokens = result.tokens_used;
  
  // Step 4: Record for ML learning
  const htmlComplexity = tokenEstimator.analyzeHTMLComplexity(content).level;
  await accuracyTracker.recordUsage(
    targetLang,
    estimatedTokens,
    actualTokens,
    htmlComplexity
  );
  
  // Step 5: Deduct actual credits used
  await creditService.deductCredits(
    userId,
    actualTokens,
    `Translation: ${sourceLang} → ${targetLang}`,
    jobId
  );
  
  return result;
}
```

### Retrieving Accuracy Stats (Admin)

```typescript
// GET /admin/v1/analytics/accuracy-stats

router.get('/accuracy-stats', adminAuth, async (req, res) => {
  const { lang } = req.query;
  
  const stats = await accuracyTracker.getAccuracyStats(lang as string);
  
  res.json({
    stats,
    summary: {
      total_languages: stats.length,
      high_confidence: stats.filter(s => s.status === 'accurate').length,
      medium_confidence: stats.filter(s => s.status === 'trained').length,
      low_confidence: stats.filter(s => s.status === 'learning').length,
    }
  });
});
```

**Response:**
```json
{
  "stats": [
    {
      "language": "fr",
      "sample_count": 1523,
      "avg_error_percent": -2.34,
      "ml_adjustment_ratio": 1.0234,
      "status": "accurate"
    },
    {
      "language": "es",
      "sample_count": 847,
      "avg_error_percent": 1.12,
      "ml_adjustment_ratio": 0.9888,
      "status": "accurate"
    },
    {
      "language": "de",
      "sample_count": 45,
      "avg_error_percent": -5.67,
      "ml_adjustment_ratio": 1.0567,
      "status": "trained"
    },
    {
      "language": "ja",
      "sample_count": 3,
      "avg_error_percent": 0.00,
      "ml_adjustment_ratio": 1.0000,
      "status": "learning"
    }
  ],
  "summary": {
    "total_languages": 4,
    "high_confidence": 2,
    "medium_confidence": 1,
    "low_confidence": 1
  }
}
```

## Performance Considerations

### 1. Database Indexing

```sql
-- Optimize ML adjustment lookups (Layer 5)
CREATE INDEX idx_language_sample ON accuracy_stats(language, sample_count);

-- Optimize recent records queries
CREATE INDEX idx_target_created ON translation_records(target_lang, created_at DESC);
```

### 2. Query Optimization

```typescript
// Use LIMIT 100 to analyze only recent translations
// Prevents full table scans as translation_records grows
const recentRecords = await prisma.$queryRaw`
  SELECT estimated_tokens, actual_tokens
  FROM translation_records
  WHERE target_lang = ${targetLang}
  ORDER BY created_at DESC
  LIMIT 100  -- Only analyze last 100
`;
```

### 3. Async ML Updates

```typescript
// Option: Make ML tracking non-blocking
async recordUsage(...args): Promise<void> {
  // Fire-and-forget: Don't wait for ML update
  this.updateMLAdjustmentAsync(targetLang).catch(err => {
    logger.error('Async ML update failed', { err });
  });
}

private async updateMLAdjustmentAsync(targetLang: string): Promise<void> {
  // Background task: Won't delay translation response
  await this.updateMLAdjustment(targetLang);
}
```

### 4. Caching ML Adjustments

```typescript
// Optional: Cache adjustment ratios in memory (5-minute TTL)
import NodeCache from 'node-cache';

const mlCache = new NodeCache({ stdTTL: 300 }); // 5 minutes

private async getMLAdjustment(targetLang: string): Promise<number> {
  // Check cache first
  const cached = mlCache.get<number>(`ml_${targetLang}`);
  if (cached !== undefined) {
    return cached;
  }
  
  // Query database
  const ratio = await this.queryMLAdjustment(targetLang);
  
  // Cache result
  mlCache.set(`ml_${targetLang}`, ratio);
  
  return ratio;
}
```

## Error Handling

### Graceful Degradation

```typescript
async recordUsage(...args): Promise<void> {
  try {
    await prisma.$executeRaw`INSERT INTO translation_records ...`;
    await this.updateMLAdjustment(targetLang);
  } catch (error) {
    logger.error('Failed to record translation usage', { error });
    // NON-BLOCKING: Don't fail the translation if ML tracking fails
    // System continues with default estimates
  }
}

private async getMLAdjustment(targetLang: string): Promise<number> {
  try {
    const stats = await prisma.$queryRaw`...`;
    return stats[0]?.ml_adjustment_ratio || 1.00;
  } catch (error) {
    logger.warn('Failed to get ML adjustment', { targetLang, error });
    return 1.00; // SAFE DEFAULT: No adjustment
  }
}
```

### Error Scenarios

| Error | Handling Strategy |
|-------|-------------------|
| `translation_records` insert fails | Log error, continue translation (ML tracking is optional) |
| `accuracy_stats` update fails | Log error, next update will recalculate from raw records |
| `getMLAdjustment()` query fails | Return 1.00 (neutral adjustment), use Layers 1-4 only |
| Database connection lost | ML tracking paused, resumes when connection restored |
| Invalid adjustment ratio (NaN) | Clamp to 1.00, log warning for investigation |

## Monitoring & Metrics

### Prometheus Metrics

```typescript
// In api/src/utils/metrics.ts

import { Counter, Gauge, Histogram } from 'prom-client';

export const mlMetrics = {
  // Accuracy tracking
  mlRecordsInserted: new Counter({
    name: 'ml_records_inserted_total',
    help: 'Total translation records inserted for ML learning',
    labelNames: ['target_lang'],
  }),
  
  mlAdjustmentUpdates: new Counter({
    name: 'ml_adjustment_updates_total',
    help: 'Total ML adjustment ratio updates',
    labelNames: ['target_lang'],
  }),
  
  // Estimation accuracy
  estimationError: new Histogram({
    name: 'estimation_error_percent',
    help: 'Estimation error percentage (actual - estimated) / estimated',
    labelNames: ['target_lang', 'html_complexity'],
    buckets: [-20, -10, -5, 0, 5, 10, 20],
  }),
  
  // Confidence levels
  languageConfidence: new Gauge({
    name: 'language_confidence_level',
    help: 'Confidence level per language (0=low, 1=medium, 2=high)',
    labelNames: ['target_lang'],
  }),
};

// Usage in AccuracyTracker
async recordUsage(...) {
  // ... insert logic ...
  
  mlMetrics.mlRecordsInserted.inc({ target_lang: targetLang });
  
  const errorPercent = ((actualTokens - estimatedTokens) / estimatedTokens) * 100;
  mlMetrics.estimationError.observe(
    { target_lang: targetLang, html_complexity: htmlComplexity },
    errorPercent
  );
}
```

### Logging

```typescript
// Structured logging for ML events
logger.info('Recorded translation for ML learning', {
  targetLang: 'fr',
  estimatedTokens: 1234,
  actualTokens: 1100,
  error: -134,
  errorPercent: '-10.85%',
  htmlComplexity: 'medium',
  timestamp: new Date().toISOString(),
});

logger.info('Updated ML adjustment', {
  targetLang: 'fr',
  sample_count: 105,
  avg_error_percent: '-8.45%',
  mlAdjustment: 1.0845,
  confidence: 'high',
});
```

## Testing Strategy

### 1. Unit Tests

```typescript
// __tests__/unit/services/accuracyTracker.test.ts

describe('AccuracyTracker', () => {
  describe('recordUsage', () => {
    it('should insert translation record', async () => {
      await accuracyTracker.recordUsage('fr', 100, 95, 'light');
      
      const records = await prisma.translationRecord.findMany({
        where: { target_lang: 'fr' },
        orderBy: { created_at: 'desc' },
        take: 1,
      });
      
      expect(records[0]).toMatchObject({
        target_lang: 'fr',
        estimated_tokens: 100,
        actual_tokens: 95,
        html_complexity: 'light',
      });
    });
    
    it('should handle database errors gracefully', async () => {
      jest.spyOn(prisma, '$executeRaw').mockRejectedValue(new Error('DB error'));
      
      // Should not throw
      await expect(
        accuracyTracker.recordUsage('es', 100, 95, 'none')
      ).resolves.not.toThrow();
    });
  });
  
  describe('updateMLAdjustment', () => {
    it('should calculate average ratio correctly', async () => {
      // Insert 10 records with 10% over-estimation
      for (let i = 0; i < 10; i++) {
        await accuracyTracker.recordUsage('de', 100, 90, 'none');
      }
      
      const stats = await accuracyTracker.getAccuracyStats('de');
      
      expect(stats[0]).toMatchObject({
        language: 'de',
        sample_count: 10,
        avg_error_percent: expect.closeTo(-10, 1),
        ml_adjustment_ratio: expect.closeTo(0.90, 0.01),
        status: 'trained',
      });
    });
    
    it('should clamp extreme ratios', async () => {
      // Insert outlier: estimated 10, actual 50 (5x ratio)
      await accuracyTracker.recordUsage('test', 10, 50, 'none');
      
      const stats = await accuracyTracker.getAccuracyStats('test');
      
      // Should be clamped to 1.20 (not 5.0)
      expect(stats[0].ml_adjustment_ratio).toBeLessThanOrEqual(1.20);
    });
  });
  
  describe('getStatus', () => {
    it('should return correct confidence levels', () => {
      const tracker = new AccuracyTracker();
      
      expect(tracker['getStatus'](5, 0)).toBe('learning');
      expect(tracker['getStatus'](50, 8)).toBe('trained');
      expect(tracker['getStatus'](150, 3)).toBe('accurate');
      expect(tracker['getStatus'](150, 15)).toBe('trained'); // High error
    });
  });
});
```

### 2. Integration Tests

```typescript
// __tests__/integration/ml-feedback-loop.test.ts

describe('ML Feedback Loop', () => {
  it('should improve estimates over time', async () => {
    const userId = 'test-user-ml';
    const content = '<p>Test content</p>';
    
    // Mock Gemini to always return 90 tokens (10% less than estimate)
    jest.spyOn(geminiClient, 'translate').mockResolvedValue({
      translation: '<p>Contenu de test</p>',
      tokens_used: 90,
    });
    
    // First translation: No ML adjustment (1.00)
    const estimate1 = await tokenEstimator.estimateSingle(content, 'en', 'fr');
    expect(estimate1.confidence).toBe('low');
    expect(estimate1.breakdown.mlAdjustment).toBe(0);
    
    // Process 15 translations
    for (let i = 0; i < 15; i++) {
      await processTranslation(content, 'en', 'fr', userId);
    }
    
    // After 15 translations: ML adjustment should be ~0.90
    const estimate2 = await tokenEstimator.estimateSingle(content, 'en', 'fr');
    expect(estimate2.confidence).toBe('medium');
    expect(estimate2.estimated_tokens).toBeLessThan(estimate1.estimated_tokens);
  });
});
```

## Admin Panel Integration

### Accuracy Dashboard Component

```tsx
// admin-panel/src/pages/AccuracyStatsPage.tsx

import React from 'react';
import { useQuery } from '@tanstack/react-query';

function AccuracyStatsPage() {
  const { data, isLoading } = useQuery({
    queryKey: ['accuracy-stats'],
    queryFn: () => fetch('/admin/v1/analytics/accuracy-stats').then(r => r.json()),
  });
  
  if (isLoading) return <div>Loading...</div>;
  
  return (
    <div className="accuracy-stats">
      <h1>ML Accuracy Statistics</h1>
      
      <div className="summary">
        <div className="stat-card">
          <h3>Total Languages</h3>
          <p>{data.summary.total_languages}</p>
        </div>
        <div className="stat-card">
          <h3>High Confidence</h3>
          <p>{data.summary.high_confidence}</p>
        </div>
        <div className="stat-card">
          <h3>Medium Confidence</h3>
          <p>{data.summary.medium_confidence}</p>
        </div>
        <div className="stat-card">
          <h3>Learning</h3>
          <p>{data.summary.low_confidence}</p>
        </div>
      </div>
      
      <table className="stats-table">
        <thead>
          <tr>
            <th>Language</th>
            <th>Sample Count</th>
            <th>Avg Error %</th>
            <th>ML Adjustment</th>
            <th>Status</th>
          </tr>
        </thead>
        <tbody>
          {data.stats.map(stat => (
            <tr key={stat.language}>
              <td>{stat.language.toUpperCase()}</td>
              <td>{stat.sample_count}</td>
              <td>{stat.avg_error_percent.toFixed(2)}%</td>
              <td>{stat.ml_adjustment_ratio.toFixed(4)}x</td>
              <td>
                <span className={`badge ${stat.status}`}>
                  {stat.status}
                </span>
              </td>
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
}
```

## Integration with Other Skills

**Often combined with:**
- `token-estimation` (Skill 13) - Provides Layer 5 ML adjustment
- `translation-service-integration` (Skill 18) - Records actual token usage
- `gemini-api-integration` (Skill 19) - Source of actual token counts
- `database-schema-design` - Uses `translation_records` and `accuracy_stats` tables

**Depends on:**
- `database-connection` - PostgreSQL/MySQL access
- `error-handling-logging` - Graceful degradation
- `prometheus-metrics` - ML performance tracking

## Anti-Patterns (Forbidden)

| ❌ Mistake | ✅ Fix |
|-----------|--------|
| Blocking translation on ML tracking failure | Make `recordUsage()` non-blocking (catch errors) |
| No outlier detection (allowing extreme ratios) | Clamp adjustment ratios to 0.80 - 1.20 range |
| Using all historical records (slow queries) | Limit to last 100 records per language |
| Applying ML adjustment with <10 samples | Only use adjustment if `sample_count >= 10` |
| Hardcoding confidence thresholds | Use configurable thresholds (10, 100) |
| No database indexes on `target_lang` | Add `idx_target_lang` on `translation_records` |
| Updating ML adjustment synchronously | Consider async updates to avoid blocking |
| No caching of ML ratios | Cache ratios for 5 minutes to reduce DB queries |
| Throwing errors on ML query failures | Return 1.00 (neutral adjustment) on error |
| No monitoring of estimation accuracy | Track `estimation_error_percent` metric |

## Quick Reference

### Function Signatures

```typescript
// Recording translations
accuracyTracker.recordUsage(
  targetLang: string,
  estimatedTokens: number,
  actualTokens: number,
  htmlComplexity: 'none' | 'light' | 'medium' | 'heavy'
): Promise<void>

// Updating ML adjustments
accuracyTracker.updateMLAdjustment(
  targetLang: string
): Promise<void>

// Retrieving stats
accuracyTracker.getAccuracyStats(
  lang?: string
): Promise<Array<{
  language: string;
  sample_count: number;
  avg_error_percent: number;
  ml_adjustment_ratio: number;
  status: 'learning' | 'trained' | 'accurate';
}>>

// Getting ML adjustment (internal)
tokenEstimator.getMLAdjustment(
  targetLang: string
): Promise<number>
```

### Database Queries

```sql
-- Insert translation record
INSERT INTO translation_records 
(target_lang, estimated_tokens, actual_tokens, html_complexity)
VALUES ('fr', 1234, 1100, 'medium');

-- Calculate ML adjustment (last 100 records)
SELECT 
  COUNT(*) as sample_count,
  AVG((actual_tokens - estimated_tokens) / estimated_tokens * 100) as avg_error_percent,
  AVG(actual_tokens / estimated_tokens) as avg_ratio
FROM (
  SELECT estimated_tokens, actual_tokens
  FROM translation_records
  WHERE target_lang = 'fr'
  ORDER BY created_at DESC
  LIMIT 100
) recent;

-- Update accuracy stats
INSERT INTO accuracy_stats (language, sample_count, avg_error_percent, ml_adjustment_ratio)
VALUES ('fr', 105, -8.45, 1.0845)
ON DUPLICATE KEY UPDATE
  sample_count = 105,
  avg_error_percent = -8.45,
  ml_adjustment_ratio = 1.0845,
  updated_at = CURRENT_TIMESTAMP;

-- Get all stats
SELECT language, sample_count, avg_error_percent, ml_adjustment_ratio
FROM accuracy_stats
WHERE sample_count > 0
ORDER BY sample_count DESC;
```

## Validation Checklist

Before completing ML feedback system implementation:

- [ ] `translation_records` table created with indexes
- [ ] `accuracy_stats` table created with unique constraint on `language`
- [ ] `recordUsage()` inserts records without blocking translations
- [ ] `updateMLAdjustment()` queries last 100 records only
- [ ] Adjustment ratios clamped to 0.80 - 1.20 range
- [ ] Confidence levels calculated correctly (low <10, medium 10-99, high 100+)
- [ ] ML adjustment only applied when `sample_count >= 10`
- [ ] Error handling returns neutral 1.00 on failures
- [ ] Structured logging for all ML operations
- [ ] Prometheus metrics track estimation accuracy
- [ ] Database indexes optimize `target_lang` and `created_at` queries
- [ ] Admin endpoint exposes accuracy stats
- [ ] Unit tests cover outlier detection
- [ ] Integration tests verify learning over time
- [ ] Documentation explains confidence levels
- [ ] Non-blocking async ML updates (optional optimization)
- [ ] Caching reduces DB load (optional optimization)

# Skill: System Settings Management

## Identity
- **Skill ID**: `system-settings-management`
- **Domain**: Configuration Management, Database-Backed Settings, In-Memory Caching
- **Technologies**: Prisma ORM, PostgreSQL (JSONB), In-Memory Cache
- **Source Agent**: `backend-app-agent.md`

## When to Load This Skill

Load this skill when working on:
- System configuration management
- Admin settings panel
- Dynamic configuration updates
- API key management (Gemini, PayPal)
- Pricing and credit allocation configuration
- Rate limit configuration
- Cache invalidation strategies

**File patterns:**
- `api/src/services/settingsService.ts`
- `api/src/routes/admin/settings.ts`
- Database schema: `SystemSetting` model

## Core Patterns

### 1. Architecture Overview

**Purpose:** Centralized system configuration stored in database with automatic in-memory caching.

**Key Features:**
- **Database-Backed:** All settings persisted in `system_setting` table (JSONB values)
- **5-Minute TTL Cache:** Reduces database load for frequently accessed settings
- **Atomic Updates:** Database updated first, then cache (consistency guaranteed)
- **Type-Safe Keys:** `SettingKey` enum prevents typos
- **Convenience Getters:** Bundled config methods (e.g., `getPayPalConfig()`)
- **Singleton Pattern:** Single shared instance across application

---

### 2. Database Schema

```prisma
// prisma/schema.prisma
model SystemSetting {
  key         String    @id @unique
  value       Json      // Flexible JSONB storage
  description String?
  updated_by  String?   // Admin user ID who last updated
  updated_at  DateTime  @default(now()) @updatedAt
}
```

**Field Descriptions:**
- `key` (String): Unique identifier (e.g., `"gemini_api_key"`)
- `value` (Json): JSONB field supporting strings, numbers, objects, arrays
- `description` (String?): Human-readable explanation of the setting
- `updated_by` (String?): UUID of admin user who made the change (audit trail)
- `updated_at` (DateTime): Automatic timestamp on every update

---

### 3. Setting Keys Enum

```typescript
export enum SettingKey {
  // Google Gemini API
  GEMINI_API_KEY = 'gemini_api_key',

  // PayPal Configuration
  PAYPAL_CLIENT_ID = 'paypal_client_id',
  PAYPAL_CLIENT_SECRET = 'paypal_client_secret',
  PAYPAL_MODE = 'paypal_mode', // 'sandbox' | 'live'
  PAYPAL_WEBHOOK_ID = 'paypal_webhook_id',

  // PayPal Plan IDs (6 subscription tiers)
  PAYPAL_PLAN_STARTER_MONTHLY = 'paypal_plan_starter_monthly',
  PAYPAL_PLAN_STARTER_ANNUAL = 'paypal_plan_starter_annual',
  PAYPAL_PLAN_PROFESSIONAL_MONTHLY = 'paypal_plan_professional_monthly',
  PAYPAL_PLAN_PROFESSIONAL_ANNUAL = 'paypal_plan_professional_annual',
  PAYPAL_PLAN_ENTERPRISE_MONTHLY = 'paypal_plan_enterprise_monthly',
  PAYPAL_PLAN_ENTERPRISE_ANNUAL = 'paypal_plan_enterprise_annual',

  // Pricing (USD)
  PRICING_PER_1K_TOKENS = 'pricing_per_1k_tokens',

  // Credit Allocations (monthly tokens)
  CREDITS_STARTER = 'credits_starter',
  CREDITS_PROFESSIONAL = 'credits_professional',
  CREDITS_ENTERPRISE = 'credits_enterprise',

  // Rate Limits (requests per minute)
  RATE_LIMIT_STARTER = 'rate_limit_starter',
  RATE_LIMIT_PROFESSIONAL = 'rate_limit_professional',
  RATE_LIMIT_ENTERPRISE = 'rate_limit_enterprise',
}
```

**Usage:**
```typescript
import { SettingKey, settingsService } from '../services/settingsService';

// Type-safe key usage
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);

// String keys also supported (for dynamic settings)
const customSetting = await settingsService.getSetting('custom_feature_flag');
```

---

### 4. Caching Strategy

**Cache Structure:**
```typescript
interface SettingsCache {
  data: Map<string, any>;      // In-memory key-value store
  lastRefresh: number;          // Unix timestamp of last DB fetch
  refreshInterval: number;      // 5 minutes (300,000 ms)
}

const cache: SettingsCache = {
  data: new Map(),
  lastRefresh: 0,
  refreshInterval: 5 * 60 * 1000,
};
```

**Cache Lifecycle:**
1. **Initial Load:** First `getSetting()` call triggers database fetch
2. **Cached Reads:** Subsequent reads served from memory (< 1ms latency)
3. **Auto-Refresh:** Cache refreshed every 5 minutes on next access
4. **Manual Invalidation:** `forceRefresh()` or `updateSetting()` triggers immediate refresh

**TTL Behavior:**
```typescript
private needsRefresh(): boolean {
  const now = Date.now();
  return now - cache.lastRefresh > cache.refreshInterval;
}
```
- Returns `true` if > 5 minutes since last refresh
- Lazy evaluation (only checked on access, not background timer)

---

### 5. Core Methods

#### 5.1 `getSystemSettings()`

**Signature:**
```typescript
async getSystemSettings(): Promise<Map<string, any>>
```

**Purpose:** Retrieve all settings as a Map.

**Returns:** Fresh copy of the settings Map (prevents external modification).

**Example:**
```typescript
const allSettings = await settingsService.getSystemSettings();

for (const [key, value] of allSettings.entries()) {
  console.log(`${key}: ${JSON.stringify(value)}`);
}
```

**Use Cases:**
- Admin dashboard settings page
- Exporting configuration
- Debugging/logging

---

#### 5.2 `getSetting<T>()`

**Signature:**
```typescript
async getSetting<T = any>(key: string | SettingKey): Promise<T | undefined>
```

**Purpose:** Get a single setting by key (returns `undefined` if not found).

**Type Parameter:** Generic `<T>` for return type casting.

**Example:**
```typescript
// String setting
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);
if (!apiKey) {
  throw new Error('Gemini API key not configured');
}

// Number setting
const pricePer1k = await settingsService.getSetting<number>(SettingKey.PRICING_PER_1K_TOKENS);

// Object setting (custom config)
interface FeatureFlags {
  enableBetaFeatures: boolean;
  maintenanceMode: boolean;
}
const flags = await settingsService.getSetting<FeatureFlags>('feature_flags');
```

---

#### 5.3 `getSettingWithDefault<T>()`

**Signature:**
```typescript
async getSettingWithDefault<T = any>(key: string | SettingKey, defaultValue: T): Promise<T>
```

**Purpose:** Get setting with fallback value (never returns `undefined`).

**Example:**
```typescript
// Pricing with default
const pricePer1k = await settingsService.getSettingWithDefault<number>(
  SettingKey.PRICING_PER_1K_TOKENS,
  0.002 // Default: $0.002 per 1K tokens
);

// Rate limit with default
const starterLimit = await settingsService.getSettingWithDefault<number>(
  SettingKey.RATE_LIMIT_STARTER,
  60 // Default: 60 requests/minute
);
```

**Use Cases:**
- Settings with sensible defaults
- Backward compatibility (new settings not yet in DB)

---

#### 5.4 `updateSetting()`

**Signature:**
```typescript
async updateSetting(
  key: string | SettingKey,
  value: any,
  description?: string,
  updatedBy?: string
): Promise<void>
```

**Purpose:** Create or update a setting (upsert operation).

**Behavior:**
1. Updates database (upsert: create if missing, update if exists)
2. Updates cache immediately (no waiting for TTL)
3. Logs change with admin user ID (audit trail)

**Example:**
```typescript
// Update pricing
await settingsService.updateSetting(
  SettingKey.PRICING_PER_1K_TOKENS,
  0.0025,
  'Increased pricing for new cost structure',
  adminUserId // UUID of admin making change
);

// Add new custom setting
await settingsService.updateSetting(
  'maintenance_mode',
  { enabled: true, message: 'Scheduled maintenance in progress' },
  'Maintenance mode configuration',
  adminUserId
);
```

**Atomicity:** Database and cache updated together (cache always reflects DB state).

---

#### 5.5 `deleteSetting()`

**Signature:**
```typescript
async deleteSetting(key: string | SettingKey): Promise<void>
```

**Purpose:** Remove a setting from database and cache.

**Example:**
```typescript
// Remove deprecated setting
await settingsService.deleteSetting('old_feature_flag');
```

**Throws:** Error if setting doesn't exist (Prisma exception).

---

#### 5.6 `forceRefresh()`

**Signature:**
```typescript
async forceRefresh(): Promise<void>
```

**Purpose:** Manually trigger cache refresh from database (ignores TTL).

**Use Cases:**
- After bulk database updates (manual SQL)
- Testing cache behavior
- Suspected cache staleness

**Example:**
```typescript
// After bulk update via SQL
await prisma.$executeRaw`
  UPDATE system_setting
  SET value = '0.003'
  WHERE key LIKE 'pricing_%'
`;

// Force cache refresh
await settingsService.forceRefresh();
```

---

#### 5.7 `clearCache()`

**Signature:**
```typescript
clearCache(): void
```

**Purpose:** Clear cache (for testing only, not used in production).

**Example:**
```typescript
// In test setup
beforeEach(() => {
  settingsService.clearCache();
});
```

---

### 6. Convenience Getter Methods

#### 6.1 `getGeminiConfig()`

**Signature:**
```typescript
async getGeminiConfig(): Promise<{ apiKey: string } | undefined>
```

**Returns:** Object with `apiKey` or `undefined` if not configured.

**Example:**
```typescript
const geminiConfig = await settingsService.getGeminiConfig();

if (!geminiConfig) {
  throw new Error('Gemini API not configured');
}

// Use apiKey
const response = await fetch('https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent', {
  headers: { 'x-goog-api-key': geminiConfig.apiKey }
});
```

---

#### 6.2 `getPayPalConfig()`

**Signature:**
```typescript
async getPayPalConfig(): Promise<{
  clientId: string;
  clientSecret: string;
  webhookId: string;
  mode: string;
  plans: {
    starterMonthly: string;
    starterAnnual: string;
    professionalMonthly: string;
    professionalAnnual: string;
    enterpriseMonthly: string;
    enterpriseAnnual: string;
  };
} | undefined>
```

**Returns:** Complete PayPal configuration bundle or `undefined` if any required field missing.

**Example:**
```typescript
const paypalConfig = await settingsService.getPayPalConfig();

if (!paypalConfig) {
  throw new Error('PayPal not configured');
}

// OAuth token generation
const authToken = await getPayPalAuthToken(
  paypalConfig.clientId,
  paypalConfig.clientSecret,
  paypalConfig.mode === 'sandbox' // Use sandbox URL
);

// Create subscription
const subscriptionId = await createPayPalSubscription(
  paypalConfig.plans.professionalMonthly,
  authToken
);
```

---

#### 6.3 `getPricingConfig()`

**Signature:**
```typescript
async getPricingConfig(): Promise<{ per1kTokens: number }>
```

**Returns:** Pricing configuration with default fallback (0.002).

**Example:**
```typescript
const { per1kTokens } = await settingsService.getPricingConfig();

// Calculate cost
const tokensUsed = 5000;
const costUSD = (tokensUsed / 1000) * per1kTokens;
console.log(`Cost: $${costUSD.toFixed(4)}`); // $0.0100
```

---

#### 6.4 `getCreditAllocations()`

**Signature:**
```typescript
async getCreditAllocations(): Promise<{
  starter: number;
  professional: number;
  enterprise: number;
}>
```

**Returns:** Monthly credit allocations for each tier (with defaults).

**Defaults:**
- Starter: 100,000 tokens
- Professional: 500,000 tokens
- Enterprise: 2,000,000 tokens

**Example:**
```typescript
const allocations = await settingsService.getCreditAllocations();

// Allocate credits on subscription activation
if (user.plan === 'professional') {
  await creditService.allocateCredits(
    user.id,
    allocations.professional,
    'Monthly subscription allocation'
  );
}
```

---

#### 6.5 `getRateLimits()`

**Signature:**
```typescript
async getRateLimits(): Promise<{
  starter: number;
  professional: number;
  enterprise: number;
}>
```

**Returns:** Rate limits (requests per minute) for each tier.

**Defaults:**
- Starter: 60 req/min
- Professional: 120 req/min
- Enterprise: 0 (unlimited)

**Example:**
```typescript
const limits = await settingsService.getRateLimits();

// Configure rate limiter
const userLimit = limits[user.plan as keyof typeof limits];
if (userLimit > 0) {
  await rateLimiter.set(`user:${user.id}`, userLimit);
}
```

---

### 7. Integration Points

#### Used By Services
```typescript
// api/src/services/geminiClient.ts
import { settingsService } from './settingsService';

const geminiConfig = await settingsService.getGeminiConfig();
const apiKey = geminiConfig?.apiKey;
```

```typescript
// api/src/services/paypalService.ts
import { settingsService } from './settingsService';

const paypalConfig = await settingsService.getPayPalConfig();
const authToken = await getOAuthToken(paypalConfig.clientId, paypalConfig.clientSecret);
```

#### Used By Middleware
```typescript
// api/src/middleware/rateLimiter.ts
import { settingsService } from '../services/settingsService';

async function getRateLimit(user: User): Promise<number> {
  const limits = await settingsService.getRateLimits();
  return limits[user.plan];
}
```

#### Used By Admin Routes
```typescript
// api/src/routes/admin/settings.ts
import { settingsService, SettingKey } from '../services/settingsService';

router.get('/admin/settings', async (req, res) => {
  const settings = await settingsService.getSystemSettings();
  res.json({ settings: Array.from(settings.entries()) });
});

router.patch('/admin/settings', async (req, res) => {
  const { key, value } = req.body;
  await settingsService.updateSetting(key, value, undefined, req.user.id);
  res.json({ success: true });
});
```

---

### 8. Cache Performance Characteristics

**Read Performance:**
- **Cache Hit (95% of reads):** < 1ms (in-memory Map lookup)
- **Cache Miss (first read or after TTL):** ~10-50ms (database query + cache population)
- **Concurrent Reads:** No contention (Map is singleton, Node.js single-threaded)

**Write Performance:**
- **Update:** ~20-80ms (database upsert + cache update)
- **Atomic:** Database and cache updated together (no race conditions)

**Cache Staleness Window:**
- **Max Staleness:** 5 minutes (TTL)
- **Typical Staleness:** < 1 minute (updates trigger immediate cache refresh)
- **Acceptable For:** Configuration settings that change infrequently

---

### 9. Error Handling

#### Cache Refresh Failure
```typescript
try {
  await refreshCache();
} catch (error) {
  logger.error('Failed to refresh settings cache', { error });
  throw new Error('Failed to refresh settings cache');
}
```
- **Behavior:** Throws error to surface failure to caller
- **Recommendation:** Implement retry logic in critical paths

#### Missing Settings
```typescript
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);

if (!apiKey) {
  // Handle missing configuration
  logger.error('Gemini API key not configured');
  throw new Error('Translation service not configured');
}
```

#### Update Failures
```typescript
try {
  await settingsService.updateSetting(key, value);
} catch (error) {
  logger.error('Failed to update setting', { key, error });
  return res.status(500).json({ error: 'Failed to update setting' });
}
```

---

### 10. Security Considerations

**Sensitive Data Storage:**
- API keys (Gemini, PayPal) stored in database (encrypted at rest via PostgreSQL)
- Never log sensitive values (logger filters `password`, `secret`, `apiKey` fields)
- Admin-only access to settings endpoints (authentication required)

**Audit Trail:**
- `updated_by` field tracks which admin made changes
- `updated_at` provides timestamp for change history
- Consider adding `SystemSettingHistory` table for full audit log

**Access Control:**
```typescript
// Middleware for admin-only settings access
import { authenticateAdmin } from '../middleware/auth';

router.patch('/admin/settings', authenticateAdmin, async (req, res) => {
  // Only admins can update settings
  await settingsService.updateSetting(req.body.key, req.body.value, undefined, req.user.id);
  res.json({ success: true });
});
```

---

### 11. Example: Complete Settings Workflow

```typescript
// 1. Admin updates pricing via dashboard
// POST /admin/settings
import { settingsService, SettingKey } from '../services/settingsService';

router.patch('/admin/settings', authenticateAdmin, async (req, res) => {
  const { key, value } = req.body;

  await settingsService.updateSetting(
    key,
    value,
    `Updated by admin: ${req.user.email}`,
    req.user.id
  );

  res.json({ success: true, message: 'Setting updated' });
});

// 2. Translation service reads updated pricing
// (within same 5-minute TTL window, cache refreshes)
import { settingsService } from './services/settingsService';

async function calculateTranslationCost(tokensUsed: number): Promise<number> {
  const { per1kTokens } = await settingsService.getPricingConfig();
  return (tokensUsed / 1000) * per1kTokens;
}

// 3. All subsequent requests use new pricing
const cost = await calculateTranslationCost(5000);
console.log(`Cost: $${cost.toFixed(4)}`); // Uses updated price
```

---

### 12. Testing Strategies

#### Unit Testing
```typescript
// __tests__/unit/services/settingsService.test.ts
import { settingsService } from '../../../services/settingsService';

describe('SettingsService', () => {
  beforeEach(() => {
    settingsService.clearCache(); // Reset cache between tests
  });

  it('should return setting from cache after first load', async () => {
    const value1 = await settingsService.getSetting('test_key');
    const value2 = await settingsService.getSetting('test_key');

    // Second call should be from cache (no DB query)
    expect(value1).toEqual(value2);
  });

  it('should return default value when setting not found', async () => {
    const value = await settingsService.getSettingWithDefault('missing_key', 'default');
    expect(value).toBe('default');
  });
});
```

#### Integration Testing
```typescript
// __tests__/integration/routes/admin/settings.test.ts
import request from 'supertest';
import { app } from '../../../server';

describe('POST /admin/settings', () => {
  it('should update setting and return success', async () => {
    const response = await request(app)
      .patch('/admin/settings')
      .set('Authorization', `Bearer ${adminToken}`)
      .send({ key: 'test_setting', value: 'new_value' })
      .expect(200);

    expect(response.body.success).toBe(true);

    // Verify setting was updated
    const setting = await settingsService.getSetting('test_setting');
    expect(setting).toBe('new_value');
  });
});
```

---

### 13. Troubleshooting Guide

| Issue | Diagnosis | Solution |
|-------|-----------|----------|
| Stale settings (old values) | Cache TTL not expired | Call `forceRefresh()` or wait 5 minutes |
| "Setting not found" errors | Key misspelled or not in DB | Check `SettingKey` enum, verify DB record |
| Slow setting reads | Cache not warming up | First read always hits DB (expected) |
| Settings not persisting | Database connection issue | Check Prisma client, verify DB connectivity |
| Inconsistent values across servers | Multiple API instances, no shared cache | Reduce TTL or use Redis for distributed cache |

---

### 14. Future Enhancements

**Planned:**
- **Distributed Cache:** Redis-backed cache for multi-instance deployments
- **Setting Validation:** JSON schema validation for setting values
- **Change History:** `SystemSettingHistory` table for audit log
- **Real-Time Updates:** WebSocket notifications when settings change
- **Setting Groups:** Organize settings into namespaces (e.g., `paypal.*`, `credits.*`)

**Not Planned:**
- Database encryption for values (rely on PostgreSQL encryption-at-rest)
- Setting versioning (single source of truth, no rollback needed)

---

This settings service provides reliable, cached configuration management with automatic refresh, type safety, and comprehensive admin controls.

---


# Appendix B: Environment Variables Reference

Complete catalog of all environment variables with validation rules, defaults, and per-environment examples.

---

## 1. Overview

### Configuration Sources

The backend uses a **hybrid configuration model**:

1. **Environment Variables (.env)** → Security-critical configs (JWT secrets, database credentials)
2. **Database (SystemSetting)** → Runtime configs (pricing, credits, API keys) - See Skill 16
3. **Code Defaults** → Fallback values defined in Zod schema

**Priority**: Database > .env > Code Defaults

### Security Rules

```bash
# ✅ CORRECT
# .env files are in .gitignore
# Never commit .env to version control
# Use separate .env files per environment

# ❌ FORBIDDEN
git add .env                    # Never commit secrets
echo "JWT_SECRET=123" > README  # Never document real secrets
```

### File Locations

| Environment | File Location | Notes |
|-------------|---------------|-------|
| Development | `api/.env` | Local development |
| Staging | `/var/www/translate-api/.env` | Staging server |
| Production | `/var/www/translate-api/.env` | Production server |

---

## 2. Node Environment

### NODE_ENV

**Description**: Application environment mode.

**Type**: `enum('development', 'production', 'test')`

**Default**: `'development'`

**Validation**: Must be one of: `development`, `production`, `test`

**Examples**:

```bash
# Development
NODE_ENV=development

# Staging
NODE_ENV=production  # Use production mode with staging DB

# Production
NODE_ENV=production
```

**Usage**:

```typescript
import { config, isDevelopment, isProduction } from './config';

if (isDevelopment()) {
  logger.debug('Running in development mode');
}
```

### PORT

**Description**: HTTP server port.

**Type**: `number`

**Default**: `3000`

**Validation**: Must be valid port number (coerced to number)

**Examples**:

```bash
# Development (default)
PORT=3000

# Staging
PORT=3001

# Production (behind Nginx reverse proxy)
PORT=3000
```

---

## 3. Database Configuration

### DATABASE_URL

**Description**: PostgreSQL connection URL.

**Type**: `string (URL format)`

**Default**: None (REQUIRED)

**Validation**: Must be valid URL format

**Format**: `postgresql://[user]:[password]@[host]:[port]/[database]`

**Examples**:

```bash
# Development
DATABASE_URL="postgresql://postgres:devpass123@localhost:5432/translate_presszone_dev"

# Staging
DATABASE_URL="postgresql://translate_user:stagingpass@staging-db.internal:5432/translate_presszone_staging"

# Production
DATABASE_URL="postgresql://translate_user:STRONG_PASSWORD@prod-db.internal:5432/translate_presszone"
```

**Security Notes**:
- Use strong passwords (32+ characters, mixed case, numbers, symbols)
- Restrict database user permissions (no DROP, CREATE USER)
- Use SSL connections in production: `?sslmode=require`

### DATABASE_POOL_SIZE

**Description**: PostgreSQL connection pool size.

**Type**: `number`

**Default**: `20`

**Validation**: Coerced to number

**Examples**:

```bash
# Development (low traffic)
DATABASE_POOL_SIZE=5

# Staging
DATABASE_POOL_SIZE=10

# Production (high traffic)
DATABASE_POOL_SIZE=30
```

**Calculation**: `(number_of_api_instances × expected_concurrent_requests) + buffer`

---

## 4. Redis Configuration

### REDIS_HOST

**Description**: Redis server hostname.

**Type**: `string`

**Default**: `'localhost'`

**Examples**:

```bash
# Development
REDIS_HOST=localhost

# Staging
REDIS_HOST=staging-redis.internal

# Production
REDIS_HOST=prod-redis.internal
```

### REDIS_PORT

**Description**: Redis server port.

**Type**: `number`

**Default**: `6379`

**Examples**:

```bash
# Standard port
REDIS_PORT=6379

# Custom port
REDIS_PORT=6380
```

### REDIS_PASSWORD

**Description**: Redis authentication password (optional).

**Type**: `string (optional)`

**Default**: `undefined`

**Examples**:

```bash
# Development (no auth)
# REDIS_PASSWORD not set

# Production (with auth)
REDIS_PASSWORD=strong_redis_password_here
```

**Security**: Always use password in production.

### REDIS_DB

**Description**: Redis database number (0-15).

**Type**: `number`

**Default**: `0`

**Examples**:

```bash
# Development (separate DB for isolation)
REDIS_DB=0

# Staging
REDIS_DB=1

# Production
REDIS_DB=0
```

---

## 5. JWT Authentication

### JWT_ACCESS_SECRET

**Description**: Secret key for signing JWT access tokens.

**Type**: `string (min 32 characters)`

**Default**: None (REQUIRED)

**Validation**: Minimum 32 characters

**Generation**:

```bash
openssl rand -hex 32
```

**Examples**:

```bash
# Development
JWT_ACCESS_SECRET=d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5

# Staging (DIFFERENT secret)
JWT_ACCESS_SECRET=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

# Production (DIFFERENT secret)
JWT_ACCESS_SECRET=f0e1d2c3b4a5968778695a4b3c2d1e0f9e8d7c6b5a49384756647382910aebfc
```

**Security**:
- NEVER reuse across environments
- Rotate quarterly
- Store in secrets manager (AWS Secrets Manager, HashiCorp Vault)

### JWT_REFRESH_SECRET

**Description**: Secret key for signing JWT refresh tokens.

**Type**: `string (min 32 characters)`

**Default**: None (REQUIRED)

**Validation**: Minimum 32 characters

**Examples**:

```bash
# Development
JWT_REFRESH_SECRET=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b

# Staging
JWT_REFRESH_SECRET=b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4

# Production
JWT_REFRESH_SECRET=a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1
```

**Security**: MUST be different from `JWT_ACCESS_SECRET`.

### JWT_ADMIN_SECRET

**Description**: Secret key for signing admin JWT tokens.

**Type**: `string (min 32 characters)`

**Default**: None (REQUIRED)

**Validation**: Minimum 32 characters

**Examples**:

```bash
# Development
JWT_ADMIN_SECRET=c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6

# Production
JWT_ADMIN_SECRET=e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0
```

**Security**: Admin tokens have elevated privileges. Use strongest secret.

### JWT_ACCESS_EXPIRY

**Description**: Access token expiration time.

**Type**: `string (time format)`

**Default**: `'15m'`

**Format**: `[number][unit]` where unit = `s` (seconds), `m` (minutes), `h` (hours), `d` (days)

**Examples**:

```bash
# Development (longer for convenience)
JWT_ACCESS_EXPIRY=1h

# Staging
JWT_ACCESS_EXPIRY=30m

# Production (short-lived for security)
JWT_ACCESS_EXPIRY=15m
```

### JWT_REFRESH_EXPIRY

**Description**: Refresh token expiration time.

**Type**: `string (time format)`

**Default**: `'7d'`

**Examples**:

```bash
# Development
JWT_REFRESH_EXPIRY=30d

# Production
JWT_REFRESH_EXPIRY=7d
```

---

## 6. Google Gemini API

### GEMINI_API_KEY

**Description**: Google Gemini API key for translation engine.

**Type**: `string (optional - can be from database)`

**Default**: `undefined`

**Source Priority**: Database > .env

**Examples**:

```bash
# Development
GEMINI_API_KEY=AIzaSyDEVELOPMENT_KEY_HERE

# Production (prefer database config)
# GEMINI_API_KEY not set (loaded from database)
```

**Database Override**:

```sql
-- Store in database instead of .env
INSERT INTO "SystemSetting" (key, value, description)
VALUES ('gemini.apiKey', 'AIzaSyPRODUCTION_KEY', 'Gemini API Key');
```

**Security**:
- Prefer database storage for runtime updates
- Use Google Cloud IAM for key management
- Enable API key restrictions (IP allowlist, API restrictions)

### GEMINI_MODEL

**Description**: Gemini model version (hardcoded in service).

**Type**: N/A (not configurable via .env)

**Value**: `'gemini-3-flash-preview'` (hardcoded in geminiClient.ts)

**Note**: Model version is NOT an environment variable. It's defined in code.

---

## 7. PayPal Configuration

All PayPal settings can be loaded from **database** (SystemSetting) or **.env**.

**Database Priority**: Database overrides .env when present.

### PAYPAL_CLIENT_ID

**Description**: PayPal REST API client ID.

**Type**: `string (optional)`

**Examples**:

```bash
# Development (sandbox)
PAYPAL_CLIENT_ID=AeXYZ1234567890_SANDBOX_CLIENT_ID

# Production (live)
PAYPAL_CLIENT_ID=AbCdEf1234567890_LIVE_CLIENT_ID
```

### PAYPAL_CLIENT_SECRET

**Description**: PayPal REST API client secret.

**Type**: `string (optional)`

**Examples**:

```bash
# Development (sandbox)
PAYPAL_CLIENT_SECRET=ELmNoPQrStUvWxYz_SANDBOX_SECRET

# Production (live)
PAYPAL_CLIENT_SECRET=AbCdEfGhIjKlMnOp_LIVE_SECRET
```

**Security**: NEVER log or expose in API responses.

### PAYPAL_WEBHOOK_ID

**Description**: PayPal webhook ID for signature verification.

**Type**: `string (optional)`

**Examples**:

```bash
# Development
PAYPAL_WEBHOOK_ID=1AB23456CD789012E

# Production
PAYPAL_WEBHOOK_ID=9ZY87654XW321098V
```

**Setup**: Create webhook in PayPal Developer Dashboard → copy ID.

### PAYPAL_MODE

**Description**: PayPal environment mode.

**Type**: `enum('sandbox', 'live')`

**Default**: `'sandbox'`

**Examples**:

```bash
# Development
PAYPAL_MODE=sandbox

# Staging
PAYPAL_MODE=sandbox

# Production
PAYPAL_MODE=live
```

**API URLs**:
- Sandbox: `https://api-m.sandbox.paypal.com`
- Live: `https://api-m.paypal.com`

### Subscription Plan IDs

PayPal plan IDs for each tier and billing cycle.

**Format**: `PAYPAL_PLAN_[TIER]_[CYCLE]`

**Variables**:

```bash
PAYPAL_PLAN_STARTER_MONTHLY
PAYPAL_PLAN_STARTER_ANNUAL
PAYPAL_PLAN_PROFESSIONAL_MONTHLY
PAYPAL_PLAN_PROFESSIONAL_ANNUAL
PAYPAL_PLAN_ENTERPRISE_MONTHLY
PAYPAL_PLAN_ENTERPRISE_ANNUAL
```

**Examples**:

```bash
# Development (sandbox plan IDs)
PAYPAL_PLAN_STARTER_MONTHLY=P-1AB23456CD789012
PAYPAL_PLAN_STARTER_ANNUAL=P-2BC34567DE890123
PAYPAL_PLAN_PROFESSIONAL_MONTHLY=P-3CD45678EF901234
PAYPAL_PLAN_PROFESSIONAL_ANNUAL=P-4DE56789FG012345
PAYPAL_PLAN_ENTERPRISE_MONTHLY=P-5EF67890GH123456
PAYPAL_PLAN_ENTERPRISE_ANNUAL=P-6FG78901HI234567

# Production (live plan IDs)
PAYPAL_PLAN_STARTER_MONTHLY=P-9ZY87654XW321098
PAYPAL_PLAN_STARTER_ANNUAL=P-8YX76543WV210987
# ... etc
```

**Setup**: Create billing plans in PayPal Dashboard → copy plan IDs.

---

## 8. SendGrid Email Service

### SENDGRID_API_KEY

**Description**: SendGrid API key for transactional emails.

**Type**: `string (optional)`

**Default**: `undefined`

**Examples**:

```bash
# Development
SENDGRID_API_KEY=SG.development_key_here

# Production
SENDGRID_API_KEY=SG.production_key_here
```

**Permissions**: Mail Send (full access)

### SENDGRID_FROM_EMAIL

**Description**: Default "From" email address.

**Type**: `string (email format)`

**Default**: `'noreply@translate.press.zone'`

**Examples**:

```bash
# Development
SENDGRID_FROM_EMAIL=dev@translate.press.zone

# Production
SENDGRID_FROM_EMAIL=noreply@translate.press.zone
```

**DNS**: Must verify domain in SendGrid (SPF, DKIM records).

### SENDGRID_FROM_NAME

**Description**: Default "From" name.

**Type**: `string`

**Default**: `'translate.press.zone'`

**Examples**:

```bash
# Development
SENDGRID_FROM_NAME=translate.press.zone [DEV]

# Production
SENDGRID_FROM_NAME=translate.press.zone
```

---

## 9. Monitoring & Observability

### LOG_LEVEL

**Description**: Winston logging level.

**Type**: `enum('error', 'warn', 'info', 'debug')`

**Default**: `'info'`

**Examples**:

```bash
# Development (verbose)
LOG_LEVEL=debug

# Staging
LOG_LEVEL=info

# Production
LOG_LEVEL=warn
```

**Levels**:
- `error`: Only critical failures
- `warn`: Warnings + errors
- `info`: General info + warnings + errors
- `debug`: Everything (verbose)

### SENTRY_DSN

**Description**: Sentry error tracking DSN (optional).

**Type**: `string (URL format, optional)`

**Default**: `undefined`

**Examples**:

```bash
# Development (optional)
# SENTRY_DSN not set

# Production
SENTRY_DSN=https://examplePublicKey@o0.ingest.sentry.io/0
```

**Setup**: Create project in Sentry → copy DSN.

### SLACK_WEBHOOK_URL

**Description**: Slack webhook for critical alerts (optional).

**Type**: `string (URL format, optional)`

**Default**: `undefined`

**Examples**:

```bash
# Production
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX
```

**Setup**: Create incoming webhook in Slack workspace.

---

## 10. Application URLs

### FRONTEND_URL

**Description**: Frontend application URL.

**Type**: `string (URL format)`

**Default**: `'https://translate.press.zone'`

**Examples**:

```bash
# Development
FRONTEND_URL=http://localhost:5173

# Staging
FRONTEND_URL=https://staging.translate.press.zone

# Production
FRONTEND_URL=https://translate.press.zone
```

**Usage**: CORS, email links, redirects.

### ADMIN_PANEL_URL

**Description**: Admin panel URL.

**Type**: `string (URL format)`

**Default**: `'https://admin.translate.press.zone'`

**Examples**:

```bash
# Development
ADMIN_PANEL_URL=http://localhost:5174

# Staging
ADMIN_PANEL_URL=https://admin.staging.translate.press.zone

# Production
ADMIN_PANEL_URL=https://admin.translate.press.zone
```

### API_URL

**Description**: API server URL.

**Type**: `string (URL format)`

**Default**: `'https://api.translate.press.zone'`

**Examples**:

```bash
# Development
API_URL=http://localhost:3000

# Staging
API_URL=https://api.staging.translate.press.zone

# Production
API_URL=https://api.translate.press.zone
```

---

## 11. CORS Configuration

### CORS_ALLOWED_ORIGINS

**Description**: Comma-separated list of allowed CORS origins.

**Type**: `string (comma-separated URLs)`

**Default**: None (REQUIRED)

**Validation**: Transformed to array by splitting on comma

**Examples**:

```bash
# Development (allow localhost)
CORS_ALLOWED_ORIGINS=http://localhost:5173,http://localhost:5174

# Staging
CORS_ALLOWED_ORIGINS=https://staging.translate.press.zone,https://admin.staging.translate.press.zone

# Production
CORS_ALLOWED_ORIGINS=https://translate.press.zone,https://admin.translate.press.zone
```

**Transformation**:

```typescript
// "url1,url2,url3" → ['url1', 'url2', 'url3']
corsAllowedOrigins: z.string().transform((val) => val.split(','))
```

---

## 12. Rate Limiting

### RATE_LIMIT_STARTER

**Description**: Requests per minute for Starter tier.

**Type**: `number`

**Default**: `60`

**Examples**:

```bash
# Development (relaxed)
RATE_LIMIT_STARTER=120

# Production
RATE_LIMIT_STARTER=60
```

### RATE_LIMIT_PROFESSIONAL

**Description**: Requests per minute for Professional tier.

**Type**: `number`

**Default**: `120`

**Examples**:

```bash
# Development
RATE_LIMIT_PROFESSIONAL=240

# Production
RATE_LIMIT_PROFESSIONAL=120
```

### RATE_LIMIT_ENTERPRISE

**Description**: Requests per minute for Enterprise tier.

**Type**: `number`

**Default**: `0` (unlimited)

**Examples**:

```bash
# Development
RATE_LIMIT_ENTERPRISE=0

# Production (unlimited)
RATE_LIMIT_ENTERPRISE=0
```

**Note**: `0` means unlimited (no rate limiting).

---

## 13. Content Limits

### MAX_SYNC_CHARS

**Description**: Maximum characters for synchronous translation.

**Type**: `number`

**Default**: `5000`

**Examples**:

```bash
# Development (higher for testing)
MAX_SYNC_CHARS=10000

# Production
MAX_SYNC_CHARS=5000
```

**Behavior**:
- Requests ≤ limit: Synchronous (immediate response)
- Requests > limit: Rejected with error

### MAX_ASYNC_CHARS

**Description**: Maximum characters for asynchronous translation.

**Type**: `number`

**Default**: `50000`

**Examples**:

```bash
# Development
MAX_ASYNC_CHARS=100000

# Production
MAX_ASYNC_CHARS=50000
```

**Behavior**:
- Requests ≤ limit: Accepted (queued)
- Requests > limit: Rejected with error

---

## 14. Webhook Configuration

### WEBHOOK_MAX_RETRIES

**Description**: Maximum webhook delivery retry attempts.

**Type**: `number`

**Default**: `5`

**Examples**:

```bash
# Development (faster failure)
WEBHOOK_MAX_RETRIES=3

# Production
WEBHOOK_MAX_RETRIES=5
```

**Retry Schedule**: Exponential backoff (2s, 4s, 8s, 16s, 32s)

### WEBHOOK_RETRY_DELAY_MS

**Description**: Initial webhook retry delay in milliseconds.

**Type**: `number`

**Default**: `2000` (2 seconds)

**Examples**:

```bash
# Development (faster retries)
WEBHOOK_RETRY_DELAY_MS=1000

# Production
WEBHOOK_RETRY_DELAY_MS=2000
```

**Exponential Backoff**:
- Attempt 1: `WEBHOOK_RETRY_DELAY_MS × 1` (2s)
- Attempt 2: `WEBHOOK_RETRY_DELAY_MS × 2` (4s)
- Attempt 3: `WEBHOOK_RETRY_DELAY_MS × 4` (8s)
- Attempt 4: `WEBHOOK_RETRY_DELAY_MS × 8` (16s)
- Attempt 5: `WEBHOOK_RETRY_DELAY_MS × 16` (32s)

---

## 15. Pricing Configuration

### PRICE_PER_1K_TOKENS

**Description**: Cost per 1,000 tokens in USD (fallback, database overrides).

**Type**: `number`

**Default**: `0.002`

**Examples**:

```bash
# Development (higher for testing)
PRICE_PER_1K_TOKENS=0.005

# Production (database preferred)
PRICE_PER_1K_TOKENS=0.002
```

**Database Override**:

```sql
-- Prefer database config for runtime updates
INSERT INTO "SystemSetting" (key, value, description)
VALUES ('pricing.per1kTokens', '0.002', 'Price per 1K tokens');
```

---

## 16. Credit Allocations

### CREDITS_STARTER

**Description**: Credit allocation for Starter tier (fallback).

**Type**: `number`

**Default**: `100000`

**Examples**:

```bash
# Development
CREDITS_STARTER=100000

# Production (database preferred)
CREDITS_STARTER=100000
```

### CREDITS_PROFESSIONAL

**Description**: Credit allocation for Professional tier (fallback).

**Type**: `number`

**Default**: `500000`

**Examples**:

```bash
# Development
CREDITS_PROFESSIONAL=500000

# Production
CREDITS_PROFESSIONAL=500000
```

### CREDITS_ENTERPRISE

**Description**: Credit allocation for Enterprise tier (fallback).

**Type**: `number`

**Default**: `2000000`

**Examples**:

```bash
# Development
CREDITS_ENTERPRISE=2000000

# Production
CREDITS_ENTERPRISE=2000000
```

**Database Override**:

```sql
-- Prefer database config for runtime updates
INSERT INTO "SystemSetting" (key, value, description)
VALUES 
  ('credits.starter', '100000', 'Starter credits'),
  ('credits.professional', '500000', 'Professional credits'),
  ('credits.enterprise', '2000000', 'Enterprise credits');
```

---

## 17. Admin Authentication (Seed Data)

### ADMIN_EMAIL

**Description**: Admin account email (used by seed script).

**Type**: `string (email format)`

**Default**: None

**Examples**:

```bash
# Development
ADMIN_EMAIL=admin@translate.press.zone

# Production
ADMIN_EMAIL=admin@translate.press.zone
```

**Usage**: Only used by `npm run seed` to create initial admin account.

### ADMIN_PASSWORD

**Description**: Admin account password (used by seed script).

**Type**: `string`

**Default**: None

**Examples**:

```bash
# Development
ADMIN_PASSWORD=admin123

# Production
ADMIN_PASSWORD=STRONG_RANDOM_PASSWORD_HERE
```

**Security**: 
- Change immediately after first login
- Never use default password in production
- Minimum 12 characters, mixed case, numbers, symbols

---

## 18. Complete .env Template

### Development

```bash
# .env.development

# Node Environment
NODE_ENV=development
PORT=3000

# Database
DATABASE_URL="postgresql://postgres:devpass@localhost:5432/translate_presszone_dev"
DATABASE_POOL_SIZE=5

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379
# REDIS_PASSWORD not set (no auth in dev)
REDIS_DB=0

# JWT Secrets (generate with: openssl rand -hex 32)
JWT_ACCESS_SECRET=d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5
JWT_REFRESH_SECRET=1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b
JWT_ADMIN_SECRET=c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
JWT_ACCESS_EXPIRY=1h
JWT_REFRESH_EXPIRY=30d

# Admin Seed Data
ADMIN_EMAIL=admin@translate.press.zone
ADMIN_PASSWORD=admin123

# Google Gemini API
GEMINI_API_KEY=AIzaSyDEVELOPMENT_KEY_HERE

# PayPal (Sandbox)
PAYPAL_CLIENT_ID=AeXYZ1234567890_SANDBOX_CLIENT_ID
PAYPAL_CLIENT_SECRET=ELmNoPQrStUvWxYz_SANDBOX_SECRET
PAYPAL_WEBHOOK_ID=1AB23456CD789012E
PAYPAL_MODE=sandbox
PAYPAL_PLAN_STARTER_MONTHLY=P-1AB23456CD789012
PAYPAL_PLAN_STARTER_ANNUAL=P-2BC34567DE890123
PAYPAL_PLAN_PROFESSIONAL_MONTHLY=P-3CD45678EF901234
PAYPAL_PLAN_PROFESSIONAL_ANNUAL=P-4DE56789FG012345
PAYPAL_PLAN_ENTERPRISE_MONTHLY=P-5EF67890GH123456
PAYPAL_PLAN_ENTERPRISE_ANNUAL=P-6FG78901HI234567

# SendGrid (optional)
SENDGRID_API_KEY=SG.development_key_here
SENDGRID_FROM_EMAIL=dev@translate.press.zone
SENDGRID_FROM_NAME=translate.press.zone [DEV]

# Monitoring
LOG_LEVEL=debug
# SENTRY_DSN not set (optional in dev)
# SLACK_WEBHOOK_URL not set (optional in dev)

# App URLs
FRONTEND_URL=http://localhost:5173
ADMIN_PANEL_URL=http://localhost:5174
API_URL=http://localhost:3000

# CORS
CORS_ALLOWED_ORIGINS=http://localhost:5173,http://localhost:5174

# Rate Limiting (relaxed for dev)
RATE_LIMIT_STARTER=120
RATE_LIMIT_PROFESSIONAL=240
RATE_LIMIT_ENTERPRISE=0

# Content Limits
MAX_SYNC_CHARS=10000
MAX_ASYNC_CHARS=100000

# Webhook Settings
WEBHOOK_MAX_RETRIES=3
WEBHOOK_RETRY_DELAY_MS=1000

# Pricing
PRICE_PER_1K_TOKENS=0.005

# Credit Allocations
CREDITS_STARTER=100000
CREDITS_PROFESSIONAL=500000
CREDITS_ENTERPRISE=2000000
```

### Staging

```bash
# .env.staging

# Node Environment
NODE_ENV=production
PORT=3001

# Database
DATABASE_URL="postgresql://translate_user:stagingpass@staging-db.internal:5432/translate_presszone_staging"
DATABASE_POOL_SIZE=10

# Redis
REDIS_HOST=staging-redis.internal
REDIS_PORT=6379
REDIS_PASSWORD=staging_redis_password
REDIS_DB=1

# JWT Secrets (UNIQUE to staging)
JWT_ACCESS_SECRET=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2
JWT_REFRESH_SECRET=b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4
JWT_ADMIN_SECRET=d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8
JWT_ACCESS_EXPIRY=30m
JWT_REFRESH_EXPIRY=7d

# Admin Seed Data
ADMIN_EMAIL=admin@translate.press.zone
ADMIN_PASSWORD=STAGING_STRONG_PASSWORD

# Google Gemini API (prefer database config)
# GEMINI_API_KEY loaded from database

# PayPal (Sandbox)
PAYPAL_MODE=sandbox
# Other PayPal vars loaded from database

# SendGrid
SENDGRID_API_KEY=SG.staging_key_here
SENDGRID_FROM_EMAIL=noreply@staging.translate.press.zone
SENDGRID_FROM_NAME=translate.press.zone [STAGING]

# Monitoring
LOG_LEVEL=info
SENTRY_DSN=https://stagingKey@o0.ingest.sentry.io/0

# App URLs
FRONTEND_URL=https://staging.translate.press.zone
ADMIN_PANEL_URL=https://admin.staging.translate.press.zone
API_URL=https://api.staging.translate.press.zone

# CORS
CORS_ALLOWED_ORIGINS=https://staging.translate.press.zone,https://admin.staging.translate.press.zone

# Rate Limiting
RATE_LIMIT_STARTER=60
RATE_LIMIT_PROFESSIONAL=120
RATE_LIMIT_ENTERPRISE=0

# Content Limits
MAX_SYNC_CHARS=5000
MAX_ASYNC_CHARS=50000

# Webhook Settings
WEBHOOK_MAX_RETRIES=5
WEBHOOK_RETRY_DELAY_MS=2000

# Pricing (prefer database config)
PRICE_PER_1K_TOKENS=0.002

# Credit Allocations (prefer database config)
CREDITS_STARTER=100000
CREDITS_PROFESSIONAL=500000
CREDITS_ENTERPRISE=2000000
```

### Production

```bash
# .env.production

# Node Environment
NODE_ENV=production
PORT=3000

# Database
DATABASE_URL="postgresql://translate_user:PRODUCTION_STRONG_PASSWORD@prod-db.internal:5432/translate_presszone?sslmode=require"
DATABASE_POOL_SIZE=30

# Redis
REDIS_HOST=prod-redis.internal
REDIS_PORT=6379
REDIS_PASSWORD=PRODUCTION_REDIS_PASSWORD
REDIS_DB=0

# JWT Secrets (UNIQUE to production, rotate quarterly)
JWT_ACCESS_SECRET=f0e1d2c3b4a5968778695a4b3c2d1e0f9e8d7c6b5a49384756647382910aebfc
JWT_REFRESH_SECRET=a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1
JWT_ADMIN_SECRET=e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0
JWT_ACCESS_EXPIRY=15m
JWT_REFRESH_EXPIRY=7d

# Admin Seed Data (only for initial setup)
ADMIN_EMAIL=admin@translate.press.zone
ADMIN_PASSWORD=PRODUCTION_INITIAL_PASSWORD_CHANGE_IMMEDIATELY

# Google Gemini API (prefer database config)
# GEMINI_API_KEY loaded from database

# PayPal (Live)
PAYPAL_MODE=live
# All PayPal configs loaded from database for security

# SendGrid
SENDGRID_API_KEY=SG.production_key_here
SENDGRID_FROM_EMAIL=noreply@translate.press.zone
SENDGRID_FROM_NAME=translate.press.zone

# Monitoring
LOG_LEVEL=warn
SENTRY_DSN=https://productionKey@o0.ingest.sentry.io/0
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX

# App URLs
FRONTEND_URL=https://translate.press.zone
ADMIN_PANEL_URL=https://admin.translate.press.zone
API_URL=https://api.translate.press.zone

# CORS
CORS_ALLOWED_ORIGINS=https://translate.press.zone,https://admin.translate.press.zone

# Rate Limiting
RATE_LIMIT_STARTER=60
RATE_LIMIT_PROFESSIONAL=120
RATE_LIMIT_ENTERPRISE=0

# Content Limits
MAX_SYNC_CHARS=5000
MAX_ASYNC_CHARS=50000

# Webhook Settings
WEBHOOK_MAX_RETRIES=5
WEBHOOK_RETRY_DELAY_MS=2000

# Pricing (loaded from database)
PRICE_PER_1K_TOKENS=0.002

# Credit Allocations (loaded from database)
CREDITS_STARTER=100000
CREDITS_PROFESSIONAL=500000
CREDITS_ENTERPRISE=2000000
```

---

## 19. Validation Reference

All variables are validated using Zod schema in `api/src/config/index.ts`.

### Validation Rules

```typescript
const ConfigSchema = z.object({
  // Node environment
  nodeEnv: z.enum(['development', 'production', 'test']).default('development'),
  port: z.coerce.number().default(3000),

  // Database
  databaseUrl: z.string().url(),
  databasePoolSize: z.coerce.number().default(20),

  // Redis
  redisHost: z.string().default('localhost'),
  redisPort: z.coerce.number().default(6379),
  redisPassword: z.string().optional(),
  redisDb: z.coerce.number().default(0),

  // JWT secrets (always from .env, never from database)
  jwtAccessSecret: z.string().min(32),
  jwtRefreshSecret: z.string().min(32),
  jwtAdminSecret: z.string().min(32),
  jwtAccessExpiry: z.string().default('15m'),
  jwtRefreshExpiry: z.string().default('7d'),

  // Google Gemini API (optional - can be from database)
  geminiApiKey: z.string().optional(),

  // PayPal (optional - can be from database)
  paypalClientId: z.string().optional(),
  paypalClientSecret: z.string().optional(),
  paypalWebhookId: z.string().optional(),
  paypalMode: z.enum(['sandbox', 'live']).default('sandbox'),
  paypalPlanStarterMonthly: z.string().optional(),
  paypalPlanStarterAnnual: z.string().optional(),
  paypalPlanProfessionalMonthly: z.string().optional(),
  paypalPlanProfessionalAnnual: z.string().optional(),
  paypalPlanEnterpriseMonthly: z.string().optional(),
  paypalPlanEnterpriseAnnual: z.string().optional(),

  // Email (SendGrid)
  sendgridApiKey: z.string().optional(),
  sendgridFromEmail: z.string().email().default('noreply@translate.press.zone'),
  sendgridFromName: z.string().default('translate.press.zone'),

  // Monitoring
  sentryDsn: z.string().url().optional(),
  logLevel: z.enum(['error', 'warn', 'info', 'debug']).default('info'),

  // Alerting
  slackWebhookUrl: z.string().url().optional(),

  // App URLs
  frontendUrl: z.string().url().default('https://translate.press.zone'),
  adminPanelUrl: z.string().url().default('https://admin.translate.press.zone'),
  apiUrl: z.string().url().default('https://api.translate.press.zone'),

  // CORS
  corsAllowedOrigins: z.string().transform((val) => val.split(',')),

  // Rate Limiting
  rateLimitStarter: z.coerce.number().default(60),
  rateLimitProfessional: z.coerce.number().default(120),
  rateLimitEnterprise: z.coerce.number().default(0), // 0 = unlimited

  // Content Limits
  maxSyncChars: z.coerce.number().default(5000),
  maxAsyncChars: z.coerce.number().default(50000),

  // Webhook Settings
  webhookMaxRetries: z.coerce.number().default(5),
  webhookRetryDelayMs: z.coerce.number().default(2000),

  // Pricing (cost per 1K tokens in USD)
  pricePerThousandTokens: z.coerce.number().default(0.002),

  // Credit Allocations
  creditsStarter: z.coerce.number().default(100000),
  creditsProfessional: z.coerce.number().default(500000),
  creditsEnterprise: z.coerce.number().default(2000000),
});
```

### Validation Errors

**Example**: Missing required variable

```bash
# Error output
Configuration validation failed: databaseUrl: Required
```

**Example**: Invalid format

```bash
# Error output
Configuration validation failed: jwtAccessSecret: String must contain at least 32 character(s)
```

**Fix**: Ensure all required variables are set with correct format/length.

---

## 20. Security Checklist

### Environment Variables

- [ ] All `.env` files in `.gitignore`
- [ ] No `.env` files committed to version control
- [ ] Unique secrets per environment (dev, staging, prod)
- [ ] JWT secrets ≥ 32 characters
- [ ] JWT secrets rotated quarterly
- [ ] Strong database passwords (32+ chars)
- [ ] Redis password enabled in production
- [ ] Database SSL enabled in production (`?sslmode=require`)

### Secret Management

- [ ] Use secrets manager (AWS Secrets Manager, HashiCorp Vault) in production
- [ ] Restrict .env file permissions: `chmod 600 .env`
- [ ] Never log environment variables
- [ ] Never expose secrets in API responses
- [ ] Audit access to .env files (who can read/modify)

### Access Control

- [ ] Database user has minimal permissions (no DROP, CREATE USER)
- [ ] Redis password enabled
- [ ] PayPal webhook signature verification enabled
- [ ] CORS origins restricted to known domains
- [ ] Rate limiting enabled for all tiers

---

## 21. Troubleshooting

### Configuration Validation Failed

**Error**: `Configuration validation failed: jwtAccessSecret: Required`

**Cause**: Missing required environment variable

**Fix**:
```bash
# Check .env file exists
ls -la api/.env

# Verify variable is set
grep JWT_ACCESS_SECRET api/.env

# Generate if missing
openssl rand -hex 32
```

### Database Connection Failed

**Error**: `ECONNREFUSED` or `Connection timeout`

**Cause**: Invalid `DATABASE_URL` or database not running

**Fix**:
```bash
# Test database connectivity
psql "$DATABASE_URL"

# Check database service
systemctl status postgresql

# Verify connection string format
echo $DATABASE_URL
# Should be: postgresql://user:pass@host:port/database
```

### Redis Connection Failed

**Error**: `ECONNREFUSED` or `NOAUTH Authentication required`

**Cause**: Invalid Redis config or missing password

**Fix**:
```bash
# Test Redis connectivity
redis-cli -h $REDIS_HOST -p $REDIS_PORT

# If password protected
redis-cli -h $REDIS_HOST -p $REDIS_PORT -a $REDIS_PASSWORD

# Check Redis service
systemctl status redis
```

### JWT Token Invalid

**Error**: `invalid signature` or `jwt malformed`

**Cause**: Mismatched JWT secrets between environments

**Fix**:
```bash
# Ensure JWT_ACCESS_SECRET matches token-signing environment
# Never copy JWT secrets between environments

# Verify secret length
echo -n "$JWT_ACCESS_SECRET" | wc -c
# Should be ≥ 64 (32 bytes in hex)
```

### PayPal Webhook Verification Failed

**Error**: `Webhook signature verification failed`

**Cause**: Invalid `PAYPAL_WEBHOOK_ID` or wrong environment mode

**Fix**:
```bash
# Verify webhook ID matches PayPal dashboard
echo $PAYPAL_WEBHOOK_ID

# Ensure PAYPAL_MODE matches webhook environment
# Sandbox webhook ID won't work with live mode and vice versa
echo $PAYPAL_MODE
```

### CORS Errors

**Error**: `Access-Control-Allow-Origin header missing`

**Cause**: Frontend origin not in `CORS_ALLOWED_ORIGINS`

**Fix**:
```bash
# Add frontend URL to CORS list
CORS_ALLOWED_ORIGINS=https://translate.press.zone,https://admin.translate.press.zone,https://new-frontend.com

# Ensure no trailing slashes in URLs
# ✅ https://example.com
# ❌ https://example.com/
```

---

This comprehensive environment variables reference provides complete documentation of all configuration options with validation rules, security best practices, and per-environment examples.

---

## Appendix C: WordPress Plugin Integration Guide

**Complete integration flow for WordPress plugins consuming the Translation API.**

---

### 1. Integration Overview

The Press.Zone Backend provides a REST API designed specifically for WordPress plugins. This appendix documents the complete integration flow, from plugin installation to webhook processing.

#### Integration Phases

1. **Plugin Installation**: User installs `translate-press-zone` plugin
2. **API Key Generation**: User creates API key at translate.press.zone
3. **Site Registration**: Plugin registers WordPress site with backend
4. **Translation Requests**: Plugin submits content for translation
5. **Webhook Callbacks**: Backend notifies plugin when translation completes

---

### 2. WordPress to API Data Transformation

The API automatically transforms WordPress snake_case payloads to internal camelCase format.

#### 2.1 Payload Transformation Flow

```
WordPress Plugin (snake_case)
         ↓
   transformWordPressJobPayload middleware
         ↓
   API Internal Processing (camelCase)
         ↓
   transformResponseForWordPress middleware
         ↓
WordPress Plugin (snake_case)
```

#### 2.2 Request Transformation Mapping

```php
// WordPress Plugin Sends (snake_case):
$payload = [
    'source_lang' => 'en',
    'target_lang' => 'es',
    'content' => 'Hello world',
    'model' => 'MODEL_4B',
    'tone' => 'formal',
    'preserve_tags' => true,
    'callback_url' => 'https://example.com/webhook',
    'callback_secret' => 'secret123',
    'client_job_id' => 'wp_post_123'
];

// API Receives (transformed to camelCase):
{
    sourceLang: 'en',
    targetLang: 'es',
    content: 'Hello world',
    model: 'MODEL_4B',
    tone: 'formal',
    preserveTags: true,
    callbackUrl: 'https://example.com/webhook',
    callbackSecret: 'secret123',
    clientJobId: 'wp_post_123'
}
```

#### 2.3 Response Transformation

```php
// API Returns (automatically transformed to snake_case):
{
    "success": true,
    "job_id": "uuid-v4",
    "status": "completed",
    "translation": "Hola mundo",
    "tokens_used": 10,
    "cost_usd": 0.001,
    "processing_time_ms": 1500,
    "credit_balance": 9990,
    "timestamp": "2026-01-27T00:00:00Z"
}
```

#### 2.4 WordPress-Specific Headers

To enable automatic transformation, plugins should set these headers:

```php
$headers = [
    'Content-Type' => 'application/json',
    'Authorization' => 'Bearer ' . $api_key,
    'User-Agent' => 'WordPress/' . $wp_version,  // Triggers transformation
    'X-WP-Source' => 'true',                      // Alternative flag
];
```

---

### 3. Authentication

#### 3.1 API Key Format

API keys follow this format:
- **Live Mode**: `sk_live_<32-character-base64>`
- **Test Mode**: `sk_test_<32-character-base64>`

```php
// Example API keys
$live_key = 'sk_live_5b7c2f8e9a1d4e6f3b0c8a7d4e2f1b9c';
$test_key = 'sk_test_9c1b2f4e7d3a6f8e0b5c7a9d2e4f1b3c';
```

#### 3.2 API Key Storage (WordPress)

**Security Requirements:**
- Store API keys in `wp_options` table
- Encrypt keys using WordPress salts
- Never log or display full keys in UI
- Use masked display (e.g., `sk_live_****1b9c`)

```php
// Secure storage implementation
class TPZ_API_Key_Manager {
    /**
     * Store API key securely
     */
    public function store_api_key( $api_key ) {
        // Validate format
        if ( ! preg_match( '/^sk_(live|test)_[a-zA-Z0-9]{32}$/', $api_key ) ) {
            return new WP_Error( 'invalid_key', 'Invalid API key format' );
        }
        
        // Encrypt using WordPress salts
        $encrypted = $this->encrypt( $api_key );
        
        // Store encrypted key
        update_option( 'tpz_api_key_encrypted', $encrypted, false ); // autoload=false
        
        return true;
    }
    
    /**
     * Retrieve decrypted API key
     */
    public function get_api_key() {
        $encrypted = get_option( 'tpz_api_key_encrypted' );
        
        if ( ! $encrypted ) {
            return null;
        }
        
        return $this->decrypt( $encrypted );
    }
    
    /**
     * Get masked version for display
     */
    public function get_masked_key() {
        $key = $this->get_api_key();
        
        if ( ! $key ) {
            return null;
        }
        
        // Show first 12 chars and last 4 chars
        return substr( $key, 0, 12 ) . '****' . substr( $key, -4 );
    }
    
    /**
     * Simple encryption using WordPress salts
     */
    private function encrypt( $data ) {
        $salt = wp_salt( 'auth' );
        $iv = substr( hash( 'sha256', $salt ), 0, 16 );
        
        return base64_encode( openssl_encrypt(
            $data,
            'AES-256-CBC',
            $salt,
            0,
            $iv
        ) );
    }
    
    /**
     * Decrypt data
     */
    private function decrypt( $data ) {
        $salt = wp_salt( 'auth' );
        $iv = substr( hash( 'sha256', $salt ), 0, 16 );
        
        return openssl_decrypt(
            base64_decode( $data ),
            'AES-256-CBC',
            $salt,
            0,
            $iv
        );
    }
}
```

#### 3.3 Making Authenticated Requests

```php
/**
 * WordPress HTTP Client for Translation API
 */
class TPZ_API_Client {
    private $api_key;
    private $base_url = 'https://api.press.zone';
    
    public function __construct( $api_key ) {
        $this->api_key = $api_key;
    }
    
    /**
     * Make authenticated request to API
     */
    public function request( $endpoint, $method = 'GET', $data = null ) {
        $url = $this->base_url . $endpoint;
        
        $args = [
            'method' => $method,
            'headers' => [
                'Authorization' => 'Bearer ' . $this->api_key,
                'Content-Type' => 'application/json',
                'User-Agent' => 'WordPress/' . get_bloginfo( 'version' ),
                'X-WP-Source' => 'true',
            ],
            'timeout' => 30,
            'sslverify' => true, // ALWAYS verify SSL in production
        ];
        
        if ( $data && in_array( $method, [ 'POST', 'PUT', 'PATCH' ] ) ) {
            $args['body'] = wp_json_encode( $data );
        }
        
        // Make request
        $response = wp_remote_request( $url, $args );
        
        // Handle errors
        if ( is_wp_error( $response ) ) {
            return $response;
        }
        
        $status_code = wp_remote_retrieve_response_code( $response );
        $body = wp_remote_retrieve_body( $response );
        
        // Decode JSON
        $data = json_decode( $body, true );
        
        // Handle API errors
        if ( $status_code >= 400 ) {
            return new WP_Error(
                $data['code'] ?? 'api_error',
                $data['message'] ?? 'Unknown API error',
                [ 'status' => $status_code, 'data' => $data ]
            );
        }
        
        return $data;
    }
}
```

---

### 4. Site Registration

Before making translation requests, the plugin must register the WordPress site with the backend.

#### 4.1 Registration Flow

```php
/**
 * Register WordPress site with backend
 * 
 * Endpoint: POST /v1/sites/register
 */
class TPZ_Site_Registration {
    private $api_client;
    
    public function __construct( TPZ_API_Client $api_client ) {
        $this->api_client = $api_client;
    }
    
    /**
     * Register or update site registration
     */
    public function register_site() {
        // Gather site information
        $site_data = $this->gather_site_data();
        
        // Send registration request
        $response = $this->api_client->request(
            '/v1/sites/register',
            'POST',
            $site_data
        );
        
        if ( is_wp_error( $response ) ) {
            error_log( 'TPZ Site Registration Failed: ' . $response->get_error_message() );
            return $response;
        }
        
        // Store site ID
        update_option( 'tpz_site_id', $response['siteId'] );
        update_option( 'tpz_site_registered_at', current_time( 'mysql' ) );
        
        return $response;
    }
    
    /**
     * Gather site information for registration
     */
    private function gather_site_data() {
        global $wp_version;
        
        $theme = wp_get_theme();
        $plugins = get_plugins();
        $active_plugins = get_option( 'active_plugins', [] );
        
        $installed_plugins = [];
        foreach ( $plugins as $plugin_path => $plugin_data ) {
            $installed_plugins[] = [
                'name' => $plugin_data['Name'],
                'version' => $plugin_data['Version'],
                'active' => in_array( $plugin_path, $active_plugins, true ),
            ];
        }
        
        return [
            'site_url' => get_site_url(),
            'site_name' => get_bloginfo( 'name' ),
            'wp_version' => $wp_version,
            'plugin_version' => TPZ_VERSION,
            'php_version' => PHP_VERSION,
            'active_theme' => $theme->get( 'Name' ),
            'locale' => get_locale(),
            'timezone' => get_option( 'timezone_string' ),
            'installed_plugins' => $installed_plugins,
        ];
    }
    
    /**
     * Send heartbeat to keep site registration active
     * 
     * Run this daily via wp_cron
     */
    public function send_heartbeat() {
        $site_id = get_option( 'tpz_site_id' );
        
        if ( ! $site_id ) {
            // Not registered yet, register now
            return $this->register_site();
        }
        
        // Update site data
        $response = $this->api_client->request(
            '/v1/sites/' . $site_id,
            'PATCH',
            $this->gather_site_data()
        );
        
        if ( is_wp_error( $response ) ) {
            error_log( 'TPZ Site Heartbeat Failed: ' . $response->get_error_message() );
        }
        
        return $response;
    }
}
```

#### 4.2 Registration WP-Cron Job

```php
/**
 * Schedule daily heartbeat
 */
add_action( 'init', function() {
    if ( ! wp_next_scheduled( 'tpz_site_heartbeat' ) ) {
        wp_schedule_event( time(), 'daily', 'tpz_site_heartbeat' );
    }
} );

add_action( 'tpz_site_heartbeat', function() {
    $api_key_manager = new TPZ_API_Key_Manager();
    $api_key = $api_key_manager->get_api_key();
    
    if ( ! $api_key ) {
        return; // No API key configured
    }
    
    $api_client = new TPZ_API_Client( $api_key );
    $registration = new TPZ_Site_Registration( $api_client );
    $registration->send_heartbeat();
} );
```

---

### 5. Translation Requests

#### 5.1 Synchronous Translation (up to 5000 characters)

```php
/**
 * Translate content synchronously
 * 
 * Endpoint: POST /v1/translate
 * Use for: Short content (posts, pages, product descriptions)
 */
class TPZ_Sync_Translator {
    private $api_client;
    
    public function __construct( TPZ_API_Client $api_client ) {
        $this->api_client = $api_client;
    }
    
    /**
     * Translate content and return immediately
     */
    public function translate( $content, $source_lang, $target_lang, $options = [] ) {
        // Validate content length
        if ( strlen( $content ) > 5000 ) {
            return new WP_Error(
                'content_too_long',
                'Content exceeds 5000 characters. Use async translation instead.'
            );
        }
        
        // Sanitize content (preserve HTML)
        $content = wp_kses_post( $content );
        
        // Prepare request
        $payload = [
            'source_lang' => $source_lang,
            'target_lang' => $target_lang,
            'content' => $content,
            'model' => $options['model'] ?? 'MODEL_4B',
            'tone' => $options['tone'] ?? null,
            'client_job_id' => $options['client_job_id'] ?? null,
        ];
        
        // Make request
        $response = $this->api_client->request( '/v1/translate', 'POST', $payload );
        
        if ( is_wp_error( $response ) ) {
            return $response;
        }
        
        // Extract translation
        return [
            'translation' => $response['translation'],
            'job_id' => $response['job_id'],
            'tokens_used' => $response['tokens_used'],
            'cost' => $response['cost_usd'],
            'credit_balance' => $response['credit_balance'],
        ];
    }
}
```

#### 5.2 Asynchronous Translation (up to 50000 characters)

```php
/**
 * Submit async translation job
 * 
 * Endpoint: POST /v1/jobs
 * Use for: Large content (documentation, eBooks, courses)
 */
class TPZ_Async_Translator {
    private $api_client;
    
    public function __construct( TPZ_API_Client $api_client ) {
        $this->api_client = $api_client;
    }
    
    /**
     * Submit translation job
     */
    public function submit_job( $content, $source_lang, $target_lang, $options = [] ) {
        // Validate content length
        if ( strlen( $content ) > 50000 ) {
            return new WP_Error(
                'content_too_long',
                'Content exceeds 50000 characters limit.'
            );
        }
        
        // Sanitize content
        $content = wp_kses_post( $content );
        
        // Prepare webhook URL
        $callback_url = add_query_arg(
            'tpz_webhook',
            '1',
            get_site_url( null, '/wp-json/tpz/v1/webhook' )
        );
        
        // Generate callback secret
        $callback_secret = wp_generate_password( 32, false );
        
        // Store secret for verification
        set_transient(
            'tpz_webhook_secret_' . $options['post_id'],
            $callback_secret,
            DAY_IN_SECONDS * 7
        );
        
        // Prepare request
        $payload = [
            'source_lang' => $source_lang,
            'target_lang' => $target_lang,
            'content' => $content,
            'model' => $options['model'] ?? 'MODEL_4B',
            'tone' => $options['tone'] ?? null,
            'callback_url' => $callback_url,
            'callback_secret' => $callback_secret,
            'client_job_id' => 'wp_post_' . $options['post_id'],
        ];
        
        // Make request
        $response = $this->api_client->request( '/v1/jobs', 'POST', $payload );
        
        if ( is_wp_error( $response ) ) {
            return $response;
        }
        
        // Store job ID in post meta
        update_post_meta(
            $options['post_id'],
            '_tpz_translation_job_id',
            $response['data']['jobId']
        );
        
        update_post_meta(
            $options['post_id'],
            '_tpz_translation_status',
            $response['data']['status']
        );
        
        return $response['data'];
    }
    
    /**
     * Check job status
     */
    public function check_status( $job_id ) {
        $response = $this->api_client->request( '/v1/jobs/' . $job_id, 'GET' );
        
        if ( is_wp_error( $response ) ) {
            return $response;
        }
        
        return $response['data'];
    }
    
    /**
     * Cancel pending job
     */
    public function cancel_job( $job_id ) {
        $response = $this->api_client->request(
            '/v1/jobs/' . $job_id . '/cancel',
            'POST'
        );
        
        return $response;
    }
}
```

---

### 6. Webhook Processing

When an async translation job completes, the backend sends a webhook callback to the WordPress plugin.

#### 6.1 Webhook Payload Structure

```json
{
  "event": "translation.completed",
  "jobId": "uuid-v4",
  "clientJobId": "wp_post_123",
  "status": "completed",
  "translation": "Translated content here...",
  "tokensUsed": 5000,
  "cost": 0.005,
  "processingTimeMs": 3500,
  "timestamp": "2026-01-27T00:00:00Z"
}
```

For failed jobs:
```json
{
  "event": "translation.failed",
  "jobId": "uuid-v4",
  "clientJobId": "wp_post_123",
  "status": "failed",
  "errorMessage": "Gemini API rate limit exceeded",
  "timestamp": "2026-01-27T00:00:00Z"
}
```

#### 6.2 Webhook Signature Verification

**CRITICAL: Always verify webhook signatures to prevent unauthorized callbacks.**

```php
/**
 * Webhook handler with signature verification
 */
class TPZ_Webhook_Handler {
    /**
     * Register webhook endpoint
     */
    public function register_endpoint() {
        register_rest_route( 'tpz/v1', '/webhook', [
            'methods' => 'POST',
            'callback' => [ $this, 'handle_webhook' ],
            'permission_callback' => '__return_true', // Verify via signature
        ] );
    }
    
    /**
     * Handle incoming webhook
     */
    public function handle_webhook( WP_REST_Request $request ) {
        // Get payload
        $payload = $request->get_json_params();
        
        if ( ! $payload ) {
            return new WP_Error( 'invalid_payload', 'Invalid JSON payload', [ 'status' => 400 ] );
        }
        
        // Verify signature
        $signature = $request->get_header( 'x-tpz-signature' );
        $timestamp = $request->get_header( 'x-tpz-timestamp' );
        
        if ( ! $signature || ! $timestamp ) {
            return new WP_Error( 'missing_signature', 'Missing signature headers', [ 'status' => 401 ] );
        }
        
        // Verify signature
        if ( ! $this->verify_signature( $payload, $signature ) ) {
            error_log( 'TPZ Webhook: Invalid signature' );
            return new WP_Error( 'invalid_signature', 'Signature verification failed', [ 'status' => 401 ] );
        }
        
        // Verify timestamp (prevent replay attacks)
        $current_time = time();
        $webhook_time = intval( $timestamp ) / 1000; // Convert ms to seconds
        
        if ( abs( $current_time - $webhook_time ) > 300 ) { // 5 minute tolerance
            return new WP_Error( 'timestamp_expired', 'Webhook timestamp too old', [ 'status' => 401 ] );
        }
        
        // Process webhook based on event type
        switch ( $payload['event'] ) {
            case 'translation.completed':
                return $this->handle_completed( $payload );
                
            case 'translation.failed':
                return $this->handle_failed( $payload );
                
            default:
                return new WP_Error( 'unknown_event', 'Unknown webhook event', [ 'status' => 400 ] );
        }
    }
    
    /**
     * Verify webhook signature
     */
    private function verify_signature( $payload, $received_signature ) {
        // Extract post ID from clientJobId
        $client_job_id = $payload['clientJobId'] ?? null;
        
        if ( ! $client_job_id || ! preg_match( '/^wp_post_(\d+)$/', $client_job_id, $matches ) ) {
            error_log( 'TPZ Webhook: Invalid clientJobId format' );
            return false;
        }
        
        $post_id = intval( $matches[1] );
        
        // Retrieve stored secret
        $secret = get_transient( 'tpz_webhook_secret_' . $post_id );
        
        if ( ! $secret ) {
            error_log( 'TPZ Webhook: Secret not found for post ' . $post_id );
            return false;
        }
        
        // Calculate expected signature
        $payload_json = wp_json_encode( $payload );
        $expected_signature = hash_hmac( 'sha256', $payload_json, $secret );
        
        // Compare signatures (timing-safe comparison)
        return hash_equals( $expected_signature, $received_signature );
    }
    
    /**
     * Handle successful translation
     */
    private function handle_completed( $payload ) {
        // Extract post ID from clientJobId
        preg_match( '/^wp_post_(\d+)$/', $payload['clientJobId'], $matches );
        $post_id = intval( $matches[1] );
        
        // Sanitize translation
        $translation = wp_kses_post( $payload['translation'] );
        
        // Update post content
        wp_update_post( [
            'ID' => $post_id,
            'post_content' => $translation,
        ] );
        
        // Update post meta
        update_post_meta( $post_id, '_tpz_translation_status', 'completed' );
        update_post_meta( $post_id, '_tpz_translation_job_id', $payload['jobId'] );
        update_post_meta( $post_id, '_tpz_tokens_used', $payload['tokensUsed'] );
        update_post_meta( $post_id, '_tpz_translation_cost', $payload['cost'] );
        update_post_meta( $post_id, '_tpz_completed_at', current_time( 'mysql' ) );
        
        // Trigger action for custom handling
        do_action( 'tpz_translation_completed', $post_id, $payload );
        
        // Clean up transient
        delete_transient( 'tpz_webhook_secret_' . $post_id );
        
        return [ 'success' => true, 'message' => 'Translation processed' ];
    }
    
    /**
     * Handle failed translation
     */
    private function handle_failed( $payload ) {
        // Extract post ID
        preg_match( '/^wp_post_(\d+)$/', $payload['clientJobId'], $matches );
        $post_id = intval( $matches[1] );
        
        // Update post meta
        update_post_meta( $post_id, '_tpz_translation_status', 'failed' );
        update_post_meta( $post_id, '_tpz_translation_job_id', $payload['jobId'] );
        update_post_meta( $post_id, '_tpz_error_message', sanitize_text_field( $payload['errorMessage'] ) );
        
        // Trigger action for error handling
        do_action( 'tpz_translation_failed', $post_id, $payload );
        
        // Clean up transient
        delete_transient( 'tpz_webhook_secret_' . $post_id );
        
        return [ 'success' => true, 'message' => 'Failure recorded' ];
    }
}

// Register webhook endpoint
add_action( 'rest_api_init', function() {
    $handler = new TPZ_Webhook_Handler();
    $handler->register_endpoint();
} );
```

---

### 7. Error Handling

#### 7.1 Common HTTP Status Codes

| Status | Code | Meaning | Plugin Action |
|--------|------|---------|---------------|
| 401 | `MISSING_API_KEY` | No Authorization header | Prompt user to configure API key |
| 401 | `INVALID_API_KEY` | API key not found in database | Display error, prompt re-entry |
| 401 | `API_KEY_INACTIVE` | API key has been deactivated | Notify user, link to dashboard |
| 402 | `INSUFFICIENT_CREDITS` | User has insufficient credits | Display credit balance, link to purchase |
| 403 | `SUBSCRIPTION_REQUIRED` | No active subscription | Display upgrade prompt |
| 413 | `CONTENT_TOO_LONG` | Content exceeds character limit | Split content or show error |
| 429 | `RATE_LIMIT_EXCEEDED` | Too many requests | Implement exponential backoff |
| 500 | `TRANSLATION_FAILED` | Translation API error | Log error, show retry button |
| 503 | `SERVICE_UNAVAILABLE` | Backend is down | Queue for retry, show maintenance message |

#### 7.2 Error Handling Implementation

```php
/**
 * Centralized error handler for API responses
 */
class TPZ_Error_Handler {
    /**
     * Handle API error response
     */
    public function handle_error( WP_Error $error ) {
        $error_code = $error->get_error_code();
        $error_message = $error->get_error_message();
        $error_data = $error->get_error_data();
        
        // Log error
        error_log( sprintf(
            'TPZ API Error: [%s] %s (HTTP %d)',
            $error_code,
            $error_message,
            $error_data['status'] ?? 0
        ) );
        
        // Handle specific errors
        switch ( $error_code ) {
            case 'INSUFFICIENT_CREDITS':
                return $this->handle_insufficient_credits( $error_data );
                
            case 'RATE_LIMIT_EXCEEDED':
                return $this->handle_rate_limit( $error_data );
                
            case 'CONTENT_TOO_LONG':
                return $this->handle_content_too_long( $error_data );
                
            case 'SUBSCRIPTION_REQUIRED':
                return $this->handle_subscription_required( $error_data );
                
            default:
                return $this->handle_generic_error( $error_code, $error_message );
        }
    }
    
    /**
     * Handle insufficient credits (402)
     */
    private function handle_insufficient_credits( $data ) {
        $balance = $data['data']['creditBalance'] ?? 0;
        
        return new WP_Error(
            'tpz_insufficient_credits',
            sprintf(
                'Insufficient credits. Current balance: %d credits. <a href="%s" target="_blank">Purchase more credits</a>',
                $balance,
                'https://translate.press.zone/credits'
            ),
            [ 'recoverable' => true ]
        );
    }
    
    /**
     * Handle rate limit (429)
     */
    private function handle_rate_limit( $data ) {
        // Store rate limit timestamp
        set_transient( 'tpz_rate_limited', time(), MINUTE_IN_SECONDS * 5 );
        
        return new WP_Error(
            'tpz_rate_limited',
            'API rate limit exceeded. Please wait a few minutes before trying again.',
            [ 'recoverable' => true, 'retry_after' => 300 ]
        );
    }
    
    /**
     * Handle content too long (413)
     */
    private function handle_content_too_long( $data ) {
        return new WP_Error(
            'tpz_content_too_long',
            'Content is too long for translation. Maximum: 50,000 characters.',
            [ 'recoverable' => false ]
        );
    }
    
    /**
     * Handle subscription required (403)
     */
    private function handle_subscription_required( $data ) {
        return new WP_Error(
            'tpz_subscription_required',
            sprintf(
                'Active subscription required. <a href="%s" target="_blank">View pricing plans</a>',
                'https://translate.press.zone/pricing'
            ),
            [ 'recoverable' => false ]
        );
    }
    
    /**
     * Handle generic errors
     */
    private function handle_generic_error( $code, $message ) {
        return new WP_Error(
            'tpz_api_error',
            sprintf( 'Translation failed: %s', esc_html( $message ) ),
            [ 'recoverable' => true ]
        );
    }
}
```

---

### 8. WordPress-Specific Considerations

#### 8.1 Content Sanitization

**Always sanitize content before sending to API and after receiving translations.**

```php
/**
 * Content sanitization helper
 */
class TPZ_Content_Sanitizer {
    /**
     * Sanitize content before sending to API
     */
    public function sanitize_for_api( $content ) {
        // Preserve HTML structure while removing unsafe tags
        $content = wp_kses_post( $content );
        
        // Normalize whitespace
        $content = normalize_whitespace( $content );
        
        return $content;
    }
    
    /**
     * Sanitize translation received from API
     */
    public function sanitize_from_api( $translation ) {
        // Apply same rules as editor content
        $translation = wp_kses_post( $translation );
        
        // Balance tags (fix unclosed tags)
        $translation = balanceTags( $translation, true );
        
        return $translation;
    }
}
```

#### 8.2 HTML Preservation

The API preserves HTML structure during translation. However, WordPress may apply additional filtering.

```php
/**
 * Disable WordPress auto-formatting during translation
 */
add_filter( 'the_content', function( $content ) {
    // Check if this is a translated post
    $is_translated = get_post_meta( get_the_ID(), '_tpz_translation_status', true ) === 'completed';
    
    if ( $is_translated ) {
        // Disable wpautop for translated content
        remove_filter( 'the_content', 'wpautop' );
    }
    
    return $content;
}, 5 ); // Run early, before wpautop
```

#### 8.3 Post Meta Storage

Store translation metadata for tracking and debugging.

```php
/**
 * Translation metadata schema
 */
$translation_meta = [
    '_tpz_translation_job_id' => 'uuid-v4',           // Backend job ID
    '_tpz_translation_status' => 'completed',         // pending|processing|completed|failed
    '_tpz_source_lang' => 'en',                       // Source language code
    '_tpz_target_lang' => 'es',                       // Target language code
    '_tpz_model' => 'MODEL_4B',                       // Model used
    '_tpz_tokens_used' => 5000,                       // Tokens consumed
    '_tpz_translation_cost' => 0.005,                 // Cost in USD
    '_tpz_completed_at' => '2026-01-27 00:00:00',     // Completion timestamp
    '_tpz_error_message' => 'Error details',          // Error message (if failed)
];

/**
 * Helper to get translation status
 */
function tpz_get_translation_status( $post_id ) {
    return get_post_meta( $post_id, '_tpz_translation_status', true );
}

/**
 * Helper to check if translation is complete
 */
function tpz_is_translated( $post_id ) {
    return tpz_get_translation_status( $post_id ) === 'completed';
}
```

---

### 9. Complete Integration Example

Here's a complete end-to-end example of translating a WordPress post:

```php
<?php
/**
 * Complete Translation Flow Example
 * 
 * This example demonstrates:
 * 1. API key configuration
 * 2. Site registration
 * 3. Sync translation (short content)
 * 4. Async translation (long content)
 * 5. Webhook processing
 * 6. Error handling
 */

class TPZ_Translation_Example {
    private $api_key_manager;
    private $api_client;
    private $registration;
    private $sync_translator;
    private $async_translator;
    private $error_handler;
    
    public function __construct() {
        $this->api_key_manager = new TPZ_API_Key_Manager();
        
        $api_key = $this->api_key_manager->get_api_key();
        
        if ( ! $api_key ) {
            return; // API key not configured
        }
        
        $this->api_client = new TPZ_API_Client( $api_key );
        $this->registration = new TPZ_Site_Registration( $this->api_client );
        $this->sync_translator = new TPZ_Sync_Translator( $this->api_client );
        $this->async_translator = new TPZ_Async_Translator( $this->api_client );
        $this->error_handler = new TPZ_Error_Handler();
    }
    
    /**
     * Example 1: Register site on plugin activation
     */
    public function example_register_site() {
        $result = $this->registration->register_site();
        
        if ( is_wp_error( $result ) ) {
            wp_die( 'Site registration failed: ' . $result->get_error_message() );
        }
        
        add_option( 'tpz_site_registered', true );
    }
    
    /**
     * Example 2: Translate short post content (sync)
     */
    public function example_translate_short_post( $post_id ) {
        $post = get_post( $post_id );
        
        if ( ! $post ) {
            return new WP_Error( 'invalid_post', 'Post not found' );
        }
        
        // Check content length
        if ( strlen( $post->post_content ) > 5000 ) {
            return new WP_Error( 'content_too_long', 'Use async translation for long content' );
        }
        
        // Translate
        $result = $this->sync_translator->translate(
            $post->post_content,
            'en',
            'es',
            [
                'model' => 'MODEL_4B',
                'tone' => 'professional',
                'client_job_id' => 'wp_post_' . $post_id,
            ]
        );
        
        // Handle errors
        if ( is_wp_error( $result ) ) {
            return $this->error_handler->handle_error( $result );
        }
        
        // Create translated post
        $translated_post_id = wp_insert_post( [
            'post_title' => $post->post_title . ' (Spanish)',
            'post_content' => $result['translation'],
            'post_status' => 'draft',
            'post_type' => $post->post_type,
        ] );
        
        // Store metadata
        update_post_meta( $translated_post_id, '_tpz_translation_job_id', $result['job_id'] );
        update_post_meta( $translated_post_id, '_tpz_translation_status', 'completed' );
        update_post_meta( $translated_post_id, '_tpz_source_post_id', $post_id );
        update_post_meta( $translated_post_id, '_tpz_source_lang', 'en' );
        update_post_meta( $translated_post_id, '_tpz_target_lang', 'es' );
        update_post_meta( $translated_post_id, '_tpz_tokens_used', $result['tokens_used'] );
        update_post_meta( $translated_post_id, '_tpz_translation_cost', $result['cost'] );
        
        return [
            'success' => true,
            'translated_post_id' => $translated_post_id,
            'credit_balance' => $result['credit_balance'],
        ];
    }
    
    /**
     * Example 3: Translate long post content (async)
     */
    public function example_translate_long_post( $post_id ) {
        $post = get_post( $post_id );
        
        if ( ! $post ) {
            return new WP_Error( 'invalid_post', 'Post not found' );
        }
        
        // Submit async job
        $result = $this->async_translator->submit_job(
            $post->post_content,
            'en',
            'es',
            [
                'model' => 'MODEL_4B',
                'tone' => 'professional',
                'post_id' => $post_id,
            ]
        );
        
        // Handle errors
        if ( is_wp_error( $result ) ) {
            return $this->error_handler->handle_error( $result );
        }
        
        // Job submitted successfully
        // Translation will arrive via webhook
        
        return [
            'success' => true,
            'job_id' => $result['jobId'],
            'status' => $result['status'],
            'message' => 'Translation job submitted. You will be notified when complete.',
        ];
    }
    
    /**
     * Example 4: Check async job status
     */
    public function example_check_job_status( $post_id ) {
        $job_id = get_post_meta( $post_id, '_tpz_translation_job_id', true );
        
        if ( ! $job_id ) {
            return new WP_Error( 'no_job', 'No translation job found for this post' );
        }
        
        $result = $this->async_translator->check_status( $job_id );
        
        if ( is_wp_error( $result ) ) {
            return $this->error_handler->handle_error( $result );
        }
        
        // Update post meta with current status
        update_post_meta( $post_id, '_tpz_translation_status', $result['status'] );
        
        return $result;
    }
    
    /**
     * Example 5: Cancel pending job
     */
    public function example_cancel_job( $post_id ) {
        $job_id = get_post_meta( $post_id, '_tpz_translation_job_id', true );
        
        if ( ! $job_id ) {
            return new WP_Error( 'no_job', 'No translation job found' );
        }
        
        $result = $this->async_translator->cancel_job( $job_id );
        
        if ( is_wp_error( $result ) ) {
            return $this->error_handler->handle_error( $result );
        }
        
        // Update status
        update_post_meta( $post_id, '_tpz_translation_status', 'cancelled' );
        
        return [ 'success' => true, 'message' => 'Job cancelled successfully' ];
    }
}

// Usage examples
add_action( 'init', function() {
    $example = new TPZ_Translation_Example();
    
    // Example: Translate a post when clicking "Translate" button
    if ( isset( $_GET['tpz_translate_post'] ) && current_user_can( 'edit_posts' ) ) {
        $post_id = intval( $_GET['tpz_translate_post'] );
        
        $result = $example->example_translate_short_post( $post_id );
        
        if ( is_wp_error( $result ) ) {
            wp_die( $result->get_error_message() );
        }
        
        wp_safe_redirect( admin_url( 'post.php?action=edit&post=' . $result['translated_post_id'] ) );
        exit;
    }
} );
```

---

### 10. Testing Checklist

Use this checklist to verify proper integration:

#### Setup Tests
- [ ] API key stored securely (encrypted in wp_options)
- [ ] API key validates successfully (GET /v1/account)
- [ ] Site registered successfully (POST /v1/sites/register)
- [ ] Daily heartbeat scheduled (wp_cron job)

#### Translation Tests
- [ ] Sync translation works for short content (<5000 chars)
- [ ] Async translation works for long content (<50000 chars)
- [ ] HTML structure preserved in translations
- [ ] Special characters (quotes, accents) handled correctly
- [ ] Whitespace normalized properly

#### Webhook Tests
- [ ] Webhook endpoint registered (/wp-json/tpz/v1/webhook)
- [ ] Signature verification works correctly
- [ ] Completed translations processed successfully
- [ ] Failed translations handled gracefully
- [ ] Replay attacks prevented (timestamp check)

#### Error Handling Tests
- [ ] 401 errors display API key prompt
- [ ] 402 errors show credit balance and purchase link
- [ ] 429 errors implement exponential backoff
- [ ] 500 errors logged and show retry button
- [ ] Network timeouts handled gracefully

#### Security Tests
- [ ] API key never logged in plaintext
- [ ] Webhook signatures always verified
- [ ] Content sanitized before API submission
- [ ] Translations sanitized after reception
- [ ] SQL injection prevented (use $wpdb->prepare)
- [ ] XSS prevented (use esc_html, esc_attr)

---

### 11. Performance Optimization

#### 11.1 Caching Strategy

```php
/**
 * Cache translation results
 */
class TPZ_Translation_Cache {
    /**
     * Get cached translation
     */
    public function get( $content_hash, $source_lang, $target_lang ) {
        $cache_key = $this->get_cache_key( $content_hash, $source_lang, $target_lang );
        
        return get_transient( $cache_key );
    }
    
    /**
     * Store translation in cache
     */
    public function set( $content_hash, $source_lang, $target_lang, $translation ) {
        $cache_key = $this->get_cache_key( $content_hash, $source_lang, $target_lang );
        
        // Cache for 30 days
        set_transient( $cache_key, $translation, DAY_IN_SECONDS * 30 );
    }
    
    /**
     * Generate cache key
     */
    private function get_cache_key( $content_hash, $source_lang, $target_lang ) {
        return sprintf(
            'tpz_translation_%s_%s_%s',
            $content_hash,
            $source_lang,
            $target_lang
        );
    }
}
```

#### 11.2 Batch Translation

```php
/**
 * Batch translate multiple posts
 */
class TPZ_Batch_Translator {
    /**
     * Translate multiple posts in queue
     */
    public function batch_translate( $post_ids, $target_lang ) {
        $results = [];
        
        foreach ( $post_ids as $post_id ) {
            // Check rate limit
            if ( get_transient( 'tpz_rate_limited' ) ) {
                // Wait and retry later
                wp_schedule_single_event(
                    time() + 300,
                    'tpz_resume_batch_translation',
                    [ $post_ids, $target_lang, $results ]
                );
                break;
            }
            
            // Translate post
            $result = $this->translate_post( $post_id, $target_lang );
            
            $results[ $post_id ] = $result;
            
            // Small delay to avoid rate limits
            sleep( 1 );
        }
        
        return $results;
    }
}
```

---

### 12. Troubleshooting Guide

| Issue | Diagnosis | Solution |
|-------|-----------|----------|
| Webhook not received | Check callback URL reachability | Verify site is public, not localhost |
| Invalid signature error | Callback secret mismatch | Verify secret stored correctly |
| Translation incomplete | Content length exceeded | Split content or use async endpoint |
| Rate limit errors | Too many requests | Implement exponential backoff |
| Incorrect translation | Wrong language code | Verify ISO 639-1 language codes |
| HTML broken | WordPress filters applied | Disable wpautop for translated content |
| Credits not deducted | API error during translation | Check logs for API errors |
| SSL verification fails | Outdated CA bundle | Update server certificates |

---

### 13. API Endpoint Quick Reference

| Endpoint | Method | Purpose | Auth |
|----------|--------|---------|------|
| `/v1/sites/register` | POST | Register WordPress site | API Key |
| `/v1/translate` | POST | Sync translation (<5000 chars) | API Key |
| `/v1/jobs` | POST | Async translation (<50000 chars) | API Key |
| `/v1/jobs/:jobId` | GET | Check job status | API Key |
| `/v1/jobs/:jobId/cancel` | POST | Cancel pending job | API Key |
| `/v1/account` | GET | Get account info | API Key |
| `/v1/account/balance` | GET | Get credit balance | API Key |
| `/v1/estimate` | POST | Estimate translation cost | API Key |

---

### 14. Language Code Reference

The API supports 131 languages. Use ISO 639-1 two-letter codes:

**Common Languages:**
- `en` - English
- `es` - Spanish
- `fr` - French
- `de` - German
- `it` - Italian
- `pt` - Portuguese
- `ru` - Russian
- `zh` - Chinese
- `ja` - Japanese
- `ko` - Korean
- `ar` - Arabic
- `hi` - Hindi

**Full list:** See Skill 13: Token Estimation & Pricing for complete 131-language catalog.

---

This completes Appendix C: WordPress Plugin Integration Guide. Plugins now have a complete reference for integrating with the Press.Zone Backend API.


---

## Appendix A: Operations Runbook - Zero-Downtime Deployment

### Overview

This runbook provides step-by-step procedures for deploying updates to the Press.Zone Backend API with zero downtime, managing database migrations, operating systemd services, and performing emergency rollbacks.

**Deployment Architecture:**
- **Blue-Green Strategy**: Run new version alongside old, switch traffic when ready
- **Native Processes**: Podman containers managed by systemd (no Kubernetes)
- **Database**: PostgreSQL with Prisma migrations
- **Queue**: Redis-backed BullMQ for async translation jobs
- **Reverse Proxy**: Nginx routes traffic to API containers

---

### 1. Pre-Deployment Checklist

**Before deploying ANY update, verify:**

```bash
# 1. Check current system health
systemctl status presszone-backend.service
podman ps --filter name=presszone
curl -f https://api.press.zone/health || echo "HEALTH CHECK FAILED"

# 2. Verify no pending migrations
cd ~/press-zone-backend/api
npx prisma migrate status

# 3. Check Redis queue depth (avoid deploying with large backlog)
redis-cli LLEN bullmq:translation:waiting

# 4. Confirm backup exists (daily automated)
ls -lh ~/press-zone-backend/backup/postgres_*.dump.gz | tail -1

# 5. Notify users (if breaking changes)
# Post maintenance notice to status page
```

**Critical Questions:**
- Is this a **breaking API change**? (Update major version, coordinate with WordPress plugin)
- Does this include **database migrations**? (Follow migration procedure below)
- Is the **migration reversible**? (If not, take manual backup first)

---

### 2. Blue-Green Deployment Procedure

**Goal:** Deploy new version without service interruption.

**Strategy:**
1. Pull new container images (or rebuild from updated code)
2. Run database migrations (if any)
3. Start new containers with different names/ports
4. Verify health checks pass
5. Switch Nginx upstream to new containers
6. Gracefully shutdown old containers after drain period

#### Step-by-Step Commands

```bash
# Navigate to backend directory
cd ~/press-zone-backend

# 1. PULL LATEST CODE (if deploying from Git)
git fetch origin
git checkout v1.2.0  # Replace with target version tag

# 2. BUILD NEW IMAGES (tagged with version)
cd api
podman build -t presszone-api:v1.2.0 -f Dockerfile --target production .

# 3. RUN DATABASE MIGRATIONS (if needed - see Section 3)
# CRITICAL: Migrations run BEFORE switching traffic
npx prisma migrate deploy

# 4. START NEW CONTAINERS (blue-green approach)
# Note: Use different container names and ports
podman run -d \
  --name presszone-api-new \
  --network host \
  -e PORT=3001 \
  -e NODE_ENV=production \
  --env-file ~/press-zone-backend/api/.env \
  -v ~/press-zone-backend/api/logs:/app/logs:Z \
  -v ~/press-zone-backend/api/prisma:/app/prisma:Z \
  presszone-api:v1.2.0 \
  npm start

podman run -d \
  --name presszone-worker-new \
  --network host \
  -e NODE_ENV=production \
  --env-file ~/press-zone-backend/api/.env \
  -v ~/press-zone-backend/api/logs:/app/logs:Z \
  presszone-api:v1.2.0 \
  node dist/worker.js

# 5. VERIFY NEW CONTAINERS (wait 30s for startup)
sleep 30
curl -f http://localhost:3001/health || echo "NEW API HEALTH CHECK FAILED"
podman logs presszone-api-new --tail 20
podman logs presszone-worker-new --tail 20

# 6. CHECK FOR ERRORS (look for crashes, missing env vars)
podman ps --filter name=presszone-*-new
# Both containers should show "Up X seconds"

# 7. SWITCH NGINX UPSTREAM (atomic traffic switch)
sudo tee /etc/nginx/conf.d/api.press.zone.conf > /dev/null <<'NGINX'
upstream presszone_api {
    server 127.0.0.1:3001 fail_timeout=10s max_fails=3;  # NEW PORT
    keepalive 32;
}
# ... (rest of nginx config unchanged)
NGINX

# 8. RELOAD NGINX (zero downtime)
sudo nginx -t && sudo systemctl reload nginx

# 9. VERIFY PRODUCTION TRAFFIC (test live endpoint)
curl -f https://api.press.zone/health
curl -H "X-API-Key: $TEST_API_KEY" https://api.press.zone/v1/pricing

# 10. MONITOR NEW CONTAINERS (5 minutes)
watch -n 5 'podman stats --no-stream presszone-api-new presszone-worker-new'
# Watch for memory leaks, CPU spikes, or crashes

# 11. DRAIN OLD CONTAINERS (wait for active requests to finish)
# Check Nginx access log for last request to old port
sudo tail -f /var/log/nginx/api.press.zone.access.log | grep ":3000"
# Wait 60 seconds after last request

# 12. STOP OLD CONTAINERS (graceful shutdown)
podman stop presszone-api presszone-worker
podman rm presszone-api presszone-worker

# 13. RENAME NEW CONTAINERS (make them the primary)
podman rename presszone-api-new presszone-api
podman rename presszone-worker-new presszone-worker

# 14. UPDATE SYSTEMD SERVICE (if needed)
# Edit ExecStart to use new image tag
sudo systemctl daemon-reload
sudo systemctl restart presszone-backend.service

# 15. TAG DEPLOYMENT (Git tag for rollback reference)
git tag -a deploy-$(date +%Y%m%d-%H%M%S) -m "Deployed v1.2.0 to production"
git push origin --tags
```

**Validation:**
- All health checks pass (`/health` returns 200)
- No errors in logs (`podman logs`)
- Redis queue processing continues (`LLEN bullmq:translation:active`)
- New translation jobs complete successfully

---

### 3. Database Migration Best Practices

**Prisma Migrations with Zero Downtime**

#### 3.1 Backward-Compatible Migrations (Safe)

**Safe Operations:**
- Adding nullable columns: `ALTER TABLE users ADD COLUMN phone TEXT;`
- Adding new tables: `CREATE TABLE new_feature (...);`
- Adding indexes: `CREATE INDEX idx_user_email ON users(email);`
- Adding check constraints (non-blocking)

**Deployment Flow:**
```bash
# 1. Create migration locally
npx prisma migrate dev --name add_phone_column

# 2. Review migration SQL
cat api/prisma/migrations/20260127_add_phone_column/migration.sql

# 3. Test on staging database
DATABASE_URL=$STAGING_DB_URL npx prisma migrate deploy

# 4. Deploy to production (runs before code deployment)
cd ~/press-zone-backend/api
npx prisma migrate deploy

# 5. Deploy application code (sees new column, handles null gracefully)
# Follow blue-green deployment above
```

#### 3.2 Breaking Migrations (Multi-Step Required)

**Breaking Operations:**
- Renaming columns: `ALTER TABLE users RENAME COLUMN name TO full_name;`
- Dropping columns: `ALTER TABLE users DROP COLUMN deprecated_field;`
- Changing column types: `ALTER TABLE jobs ALTER COLUMN status TYPE TEXT;`
- Adding NOT NULL constraints to existing columns

**Multi-Step Strategy:**

**Example: Rename `users.name` to `users.full_name`**

**Step 1 (Additive Migration):**
```sql
-- Migration: 20260127_add_full_name_column
ALTER TABLE users ADD COLUMN full_name TEXT;
UPDATE users SET full_name = name WHERE full_name IS NULL;
```

Deploy code that writes to BOTH columns:
```typescript
// v1.1.0 code
await prisma.user.create({
  data: {
    name: fullName,      // OLD column (keep for backward compat)
    full_name: fullName  // NEW column
  }
});
```

**Step 2 (Backfill Data - Run After Deployment):**
```sql
-- Run as separate script (not in migration)
UPDATE users SET full_name = name WHERE full_name IS NULL;
```

**Step 3 (Remove Old Column - Next Release):**
```sql
-- Migration: 20260128_drop_name_column
ALTER TABLE users DROP COLUMN name;
```

Deploy code that uses only new column:
```typescript
// v1.2.0 code
await prisma.user.create({
  data: {
    full_name: fullName  // Only NEW column
  }
});
```

#### 3.3 Migration Rollback Procedure

**If Migration Fails:**
```bash
# 1. STOP DEPLOYMENT (do not deploy application code)
echo "Migration failed - aborting deployment"

# 2. CHECK MIGRATION STATUS
npx prisma migrate status
# Output shows which migration failed

# 3. ROLLBACK MIGRATION (manual SQL required)
# Prisma doesn't support automatic rollback - write inverse SQL

# Example: If add_phone_column failed:
psql -U translate_user -d translate_db -c "ALTER TABLE users DROP COLUMN phone;"

# 4. MARK MIGRATION AS ROLLED BACK
npx prisma migrate resolve --rolled-back 20260127_add_phone_column

# 5. FIX MIGRATION SQL (locally)
# Edit migration file, test on staging

# 6. RETRY DEPLOYMENT
npx prisma migrate deploy
```

#### 3.4 Long-Running Migrations

**Problem:** Adding indexes on large tables locks rows.

**Solution:** Use `CREATE INDEX CONCURRENTLY` (PostgreSQL-specific).

```sql
-- In migration file:
-- CreateIndex (non-blocking)
CREATE INDEX CONCURRENTLY "idx_jobs_user_created" ON "translation_jobs"("user_id", "created_at");
```

**Deployment Notes:**
- Cannot run in transaction (Prisma wraps migrations in transactions by default)
- Must use raw SQL migration: `npx prisma migrate dev --create-only`
- Edit generated SQL to add `CONCURRENTLY`
- Apply manually: `psql -U translate_user -d translate_db -f migration.sql`

---

### 4. Systemd Service Management

**Service Unit:** `/etc/systemd/user/presszone-backend.service`

#### 4.1 Reload vs Restart

| Operation | Command | Downtime | Use Case |
|-----------|---------|----------|----------|
| **Reload** | `systemctl reload` | None | Config changes (nginx, env vars) |
| **Restart** | `systemctl restart` | 2-5s | Code updates, dependency changes |
| **Stop/Start** | `systemctl stop && systemctl start` | Manual control | Emergency maintenance |

#### 4.2 Common Commands

```bash
# View service status
systemctl --user status presszone-backend.service

# View logs (last 100 lines)
journalctl --user -u presszone-backend.service -n 100 --no-pager

# Follow logs (real-time)
journalctl --user -u presszone-backend.service -f

# Restart service (graceful shutdown)
systemctl --user restart presszone-backend.service

# Reload systemd config (after editing .service file)
systemctl --user daemon-reload

# Enable on boot
systemctl --user enable presszone-backend.service

# Disable on boot
systemctl --user disable presszone-backend.service

# Check if service is active
systemctl --user is-active presszone-backend.service || echo "SERVICE DOWN"
```

#### 4.3 Graceful Shutdown

**SIGTERM Handling (Node.js):**

The API server listens for SIGTERM and drains connections:

```typescript
// src/server.ts
process.on('SIGTERM', async () => {
  logger.info('SIGTERM received, starting graceful shutdown...');
  
  // 1. Stop accepting new requests
  server.close(() => {
    logger.info('HTTP server closed');
  });
  
  // 2. Stop worker queue processing
  await translationQueue.close();
  
  // 3. Wait for active jobs to finish (timeout 30s)
  setTimeout(() => {
    logger.warn('Shutdown timeout - forcing exit');
    process.exit(1);
  }, 30000);
  
  // 4. Close database connections
  await prisma.$disconnect();
  
  logger.info('Graceful shutdown complete');
  process.exit(0);
});
```

**Systemd Configuration:**
```ini
[Service]
Type=forking
TimeoutStopSec=30s  # Wait 30s for graceful shutdown before SIGKILL
Restart=on-failure
RestartSec=10s      # Wait 10s before restart on crash
```

#### 4.4 Service Dependencies

**Dependency Chain:**
```
presszone-backend.service
  ↳ Requires: network-online.target
  ↳ After: postgresql.service
  ↳ After: redis.service
```

**Why This Matters:**
- Systemd starts dependencies first
- If PostgreSQL crashes, systemd does NOT restart API (set `BindsTo=` to link lifecycles)
- If Redis crashes, API continues but translation jobs fail (monitor Redis separately)

---

### 5. Rolling Update for Worker Processes

**Use Case:** Deploy new worker code without losing queued jobs.

**Strategy:**
1. Start new worker containers
2. Let old workers finish active jobs (no new jobs assigned)
3. Stop old workers after queue drains

#### Commands

```bash
# 1. CHECK QUEUE DEPTH
redis-cli LLEN bullmq:translation:waiting
redis-cli LLEN bullmq:translation:active
# Output: 50 waiting, 3 active

# 2. START NEW WORKER (alongside old worker)
podman run -d \
  --name presszone-worker-v2 \
  --network host \
  -e NODE_ENV=production \
  --env-file ~/press-zone-backend/api/.env \
  presszone-api:v1.2.0 \
  node dist/worker.js

# 3. VERIFY NEW WORKER PROCESSING JOBS
podman logs presszone-worker-v2 | grep "Job completed"
# Should see new jobs being processed

# 4. PAUSE OLD WORKER (stop accepting new jobs)
# BullMQ doesn't support pausing workers - instead:
# Send SIGTERM to trigger graceful shutdown
podman kill --signal SIGTERM presszone-worker

# 5. WAIT FOR ACTIVE JOBS TO FINISH (monitor logs)
podman logs -f presszone-worker
# Wait for "Worker stopped" message

# 6. STOP OLD WORKER
podman stop presszone-worker
podman rm presszone-worker

# 7. RENAME NEW WORKER
podman rename presszone-worker-v2 presszone-worker

# 8. VERIFY QUEUE PROCESSING (no backlog)
redis-cli LLEN bullmq:translation:waiting
# Should decrease over time
```

**Monitoring During Rolling Update:**
```bash
# Watch queue metrics
watch -n 2 'redis-cli LLEN bullmq:translation:waiting && redis-cli LLEN bullmq:translation:active'

# Check worker logs for errors
podman logs presszone-worker-v2 --tail 50 | grep -E "ERROR|WARN"
```

---

### 6. Rollback Procedures

#### 6.1 Application Rollback (Emergency)

**Scenario:** New code has critical bug, production is broken.

**Fast Rollback (5 minutes):**

```bash
# 1. IDENTIFY PREVIOUS VERSION
git tag -l "deploy-*" | tail -2
# Output: deploy-20260127-100000 (current) and deploy-20260126-140000 (previous)

# 2. STOP BROKEN CONTAINERS
podman stop presszone-api presszone-worker

# 3. CHECKOUT PREVIOUS VERSION
cd ~/press-zone-backend
git checkout deploy-20260126-140000

# 4. REBUILD CONTAINERS (from previous code)
cd api
podman build -t presszone-api:rollback -f Dockerfile --target production .

# 5. START ROLLBACK CONTAINERS
podman run -d \
  --name presszone-api \
  --network host \
  -e PORT=3000 \
  -e NODE_ENV=production \
  --env-file ~/press-zone-backend/api/.env \
  -v ~/press-zone-backend/api/logs:/app/logs:Z \
  presszone-api:rollback \
  npm start

podman run -d \
  --name presszone-worker \
  --network host \
  -e NODE_ENV=production \
  --env-file ~/press-zone-backend/api/.env \
  presszone-api:rollback \
  node dist/worker.js

# 6. VERIFY HEALTH
curl -f http://localhost:3000/health

# 7. RELOAD NGINX (if port changed)
sudo systemctl reload nginx

# 8. NOTIFY TEAM (post-mortem)
echo "Rollback completed - investigate broken version in non-prod"
```

#### 6.2 Database Rollback (Dangerous)

**Scenario:** Migration broke production, need to revert schema.

**WARNING:** Database rollback is DESTRUCTIVE. Data loss is possible.

**Rollback Strategies:**

**Strategy 1: Restore from Backup (Safest)**
```bash
# 1. STOP API (prevent writes during restore)
systemctl --user stop presszone-backend.service

# 2. IDENTIFY BACKUP (pre-migration backup)
ls -lh ~/press-zone-backend/backup/postgres_*.dump.gz | tail -5

# 3. RESTORE DATABASE
cd ~/press-zone-backend/backup
gunzip postgres_20260127_095000.dump.gz
pg_restore -U translate_user -d translate_db -c postgres_20260127_095000.dump

# 4. VERIFY DATA INTEGRITY
psql -U translate_user -d translate_db -c "SELECT COUNT(*) FROM users;"

# 5. ROLLBACK CODE (to match schema)
cd ~/press-zone-backend
git checkout deploy-20260126-140000

# 6. RESTART API
systemctl --user start presszone-backend.service

# 7. VERIFY SERVICE
curl -f https://api.press.zone/health
```

**Strategy 2: Reverse Migration (Manual SQL)**
```bash
# 1. WRITE REVERSE MIGRATION (inverse of broken migration)
# Example: If migration added column, drop it
cat > /tmp/rollback.sql <<'SQL'
ALTER TABLE users DROP COLUMN IF EXISTS phone;
SQL

# 2. APPLY REVERSE MIGRATION
psql -U translate_user -d translate_db -f /tmp/rollback.sql

# 3. MARK MIGRATION AS ROLLED BACK (Prisma)
cd ~/press-zone-backend/api
npx prisma migrate resolve --rolled-back 20260127_add_phone_column

# 4. VERIFY SCHEMA
npx prisma db pull  # Sync Prisma schema with DB state
```

**Strategy 3: Point-in-Time Recovery (PostgreSQL PITR)**
```bash
# Requires WAL archiving configured (see Section 9)
# 1. Stop PostgreSQL
sudo systemctl stop postgresql

# 2. Restore base backup
sudo -u postgres pg_basebackup -D /var/lib/postgresql/data-restore

# 3. Create recovery.conf
cat > /var/lib/postgresql/data-restore/recovery.conf <<'CONF'
restore_command = 'cp /var/lib/postgresql/wal_archive/%f %p'
recovery_target_time = '2026-01-27 09:50:00'  # Before migration
CONF

# 4. Start PostgreSQL in recovery mode
sudo systemctl start postgresql
```

#### 6.3 Rollback Decision Matrix

| Issue Severity | Response Time | Action |
|----------------|---------------|--------|
| **Critical Outage** (API returns 500, all requests fail) | < 5 minutes | Application rollback (Section 6.1) |
| **Data Corruption** (migration broke FK constraints) | < 15 minutes | Database restore from backup (6.2 Strategy 1) |
| **Degraded Performance** (new code slower, no errors) | < 30 minutes | Investigate first, rollback if no quick fix |
| **Minor Bug** (edge case fails, most requests work) | Next deployment | Fix forward, don't rollback |

---

### 7. Health Check Monitoring During Deployment

**Automated Health Checks:**

```bash
#!/bin/bash
# healthcheck.sh - Run during deployment to catch issues early

set -e

API_URL="https://api.press.zone"

# 1. Basic Health Check
echo "Testing /health endpoint..."
curl -f "$API_URL/health" || exit 1

# 2. Authenticated Request (test API key auth)
echo "Testing authenticated endpoint..."
curl -f -H "X-API-Key: $TEST_API_KEY" "$API_URL/v1/account" || exit 1

# 3. Database Connectivity (check query works)
echo "Testing /v1/pricing (DB query)..."
curl -f "$API_URL/v1/pricing" | jq .data || exit 1

# 4. Redis Queue (check worker processing)
echo "Testing queue depth..."
QUEUE_DEPTH=$(redis-cli LLEN bullmq:translation:waiting)
if [ "$QUEUE_DEPTH" -gt 1000 ]; then
  echo "WARNING: Queue backlog is $QUEUE_DEPTH"
fi

# 5. Check Error Rate (Prometheus metrics)
echo "Checking error rate..."
ERROR_RATE=$(curl -s http://localhost:9090/metrics | grep 'api_errors_total' | awk '{print $2}')
echo "Current error count: $ERROR_RATE"

echo "All health checks passed!"
```

**Continuous Monitoring During Deployment:**
```bash
# Run in separate terminal
watch -n 5 './healthcheck.sh'
```

---

### 8. Nginx Traffic Switching

**Configuration File:** `/etc/nginx/conf.d/api.press.zone.conf`

#### 8.1 Blue-Green Upstream Switch

**Before Deployment (Old Version on Port 3000):**
```nginx
upstream presszone_api {
    server 127.0.0.1:3000 fail_timeout=10s max_fails=3;
    keepalive 32;
}
```

**During Deployment (New Version on Port 3001):**
```nginx
upstream presszone_api {
    server 127.0.0.1:3001 fail_timeout=10s max_fails=3;  # CHANGED
    keepalive 32;
}
```

**Apply Change (Zero Downtime):**
```bash
# 1. Test config syntax
sudo nginx -t

# 2. Reload (no connection drops)
sudo systemctl reload nginx

# 3. Verify active connections moved
sudo ss -tunap | grep nginx | grep ESTABLISHED
```

#### 8.2 Canary Deployment (Advanced)

**Gradual Traffic Shift (10% → 50% → 100%):**

```nginx
upstream presszone_api {
    server 127.0.0.1:3000 weight=9;  # Old version (90%)
    server 127.0.0.1:3001 weight=1;  # New version (10%)
    keepalive 32;
}
```

**Increase weight over time:**
```bash
# After 5 minutes, no errors:
# Change to weight=5 / weight=5 (50/50 split)

# After 10 minutes, no errors:
# Change to weight=1 / weight=9 (90% new)

# After 15 minutes, no errors:
# Remove old upstream entirely
```

#### 8.3 Emergency Traffic Drain

**If new version has issues, redirect all traffic to old version:**

```bash
# 1. Update upstream (revert to old port)
sudo tee /etc/nginx/conf.d/api.press.zone.conf > /dev/null <<'NGINX'
upstream presszone_api {
    server 127.0.0.1:3000 fail_timeout=10s max_fails=3;  # OLD PORT
    keepalive 32;
}
NGINX

# 2. Reload nginx
sudo nginx -t && sudo systemctl reload nginx

# 3. Verify traffic routing
curl -f https://api.press.zone/health
```

---

### 9. Backup Verification Before Deployment

**Never deploy without a recent backup.**

#### 9.1 Automated Daily Backups

**Cron Job (runs at 2 AM daily):**
```bash
# crontab -e (as user 'press')
0 2 * * * /home/press/press-zone-backend/backup/backup.sh >> /home/press/press-zone-backend/backup/backup.log 2>&1
```

#### 9.2 Pre-Deployment Manual Backup

```bash
# Run before risky deployments (schema changes, major refactors)
cd ~/press-zone-backend/backup
./backup.sh

# Verify backup size (should be >10MB for production)
ls -lh postgres_*.dump.gz | tail -1

# Test restore on staging (optional)
gunzip < postgres_20260127_130000.dump.gz | pg_restore -U staging_user -d staging_db
```

#### 9.3 Backup Retention Policy

| Backup Type | Retention | Location | Purpose |
|-------------|-----------|----------|---------|
| **Daily** | 30 days | `~/press-zone-backend/backup/` | Routine recovery |
| **Pre-Deployment** | 90 days | Same directory | Rollback safety net |
| **Monthly** | 1 year | S3/external storage | Compliance, long-term recovery |

---

### 10. Post-Deployment Validation

**After deployment completes, verify:**

#### 10.1 Functional Tests

```bash
# 1. Health Check
curl -f https://api.press.zone/health

# 2. Authentication (API Key)
curl -H "X-API-Key: $PROD_API_KEY" https://api.press.zone/v1/account

# 3. Translation Job (End-to-End Test)
curl -X POST https://api.press.zone/v1/translate \
  -H "X-API-Key: $PROD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_lang": "en",
    "target_lang": "es",
    "content": "Hello, world!"
  }'

# 4. Check Job Completed
JOB_ID="<job_id_from_response>"
curl https://api.press.zone/v1/jobs/$JOB_ID \
  -H "X-API-Key: $PROD_API_KEY"
```

#### 10.2 Performance Metrics

```bash
# 1. Response Time (should be <200ms for /health)
time curl -f https://api.press.zone/health

# 2. Queue Processing Rate (jobs/minute)
watch -n 10 'redis-cli LLEN bullmq:translation:waiting'

# 3. Memory Usage (should be stable)
podman stats --no-stream presszone-api presszone-worker

# 4. Error Rate (check logs)
podman logs presszone-api --since 5m | grep -i error | wc -l
```

#### 10.3 Database Integrity

```bash
# 1. Run Prisma validation
cd ~/press-zone-backend/api
npx prisma validate

# 2. Check foreign key constraints
psql -U translate_user -d translate_db -c "
  SELECT conname, conrelid::regclass 
  FROM pg_constraint 
  WHERE contype = 'f';
"

# 3. Verify row counts (no data loss)
psql -U translate_user -d translate_db -c "
  SELECT 'users' AS table, COUNT(*) FROM users
  UNION ALL
  SELECT 'translation_jobs', COUNT(*) FROM translation_jobs;
"
```

---

### 11. Common Deployment Issues & Solutions

| Issue | Symptom | Root Cause | Solution |
|-------|---------|------------|----------|
| **500 Errors on New Version** | All requests fail | Missing environment variable | Check `podman logs` for "undefined", add to `.env` |
| **Database Connection Errors** | `Can't connect to PostgreSQL` | PostgreSQL not running or wrong credentials | Verify `systemctl status postgresql` and `DATABASE_URL` |
| **Migration Stuck** | `npx prisma migrate deploy` hangs | Migration locked by active transaction | Kill blocking queries: `SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active';` |
| **Nginx 502 Bad Gateway** | Frontend can't reach API | Container not listening on expected port | Check `PORT` env var matches Nginx upstream |
| **Queue Jobs Not Processing** | `bullmq:translation:waiting` grows | Worker crashed or not started | Restart worker: `podman restart presszone-worker` |
| **Old Container Still Running** | Traffic goes to wrong version | Forgot to stop old container | `podman stop presszone-api-old` |
| **Health Check Fails** | `/health` returns 503 | Database connection issue or migration failed | Check logs, verify migrations applied |

---

### 12. Emergency Procedures

#### 12.1 Complete Service Outage

**Scenario:** API is down, users can't access service.

**Response (P0 - Critical):**

```bash
# 1. CHECK SYSTEMD STATUS
systemctl --user status presszone-backend.service
# If "failed", proceed to restart

# 2. CHECK CONTAINER STATUS
podman ps --all | grep presszone
# If containers "Exited", check logs

# 3. VIEW LOGS (last 50 lines)
podman logs presszone-api --tail 50
podman logs presszone-worker --tail 50

# 4. RESTART CONTAINERS
podman restart presszone-api presszone-worker

# 5. VERIFY HEALTH
curl -f http://localhost:3000/health

# 6. CHECK DEPENDENCIES (PostgreSQL, Redis)
systemctl status postgresql
systemctl status redis

# 7. IF DATABASE ISSUE, RESTORE BACKUP
cd ~/press-zone-backend/backup
./restore.sh postgres_20260127_095000.dump.gz

# 8. NOTIFY USERS (update status page)
# Post: "Service restored - investigating root cause"
```

#### 12.2 Data Corruption Detected

**Scenario:** Reports of incorrect credit balances or missing jobs.

**Response (P1 - High):**

```bash
# 1. STOP API IMMEDIATELY (prevent further corruption)
systemctl --user stop presszone-backend.service

# 2. IDENTIFY EXTENT OF CORRUPTION
psql -U translate_user -d translate_db -c "
  SELECT user_id, COUNT(*) AS tx_count, MAX(balance_after) AS balance
  FROM credit_transactions
  GROUP BY user_id
  HAVING MAX(balance_after) < 0;  -- Negative balances (corrupted)
"

# 3. RESTORE DATABASE FROM LAST KNOWN GOOD BACKUP
cd ~/press-zone-backend/backup
./restore.sh postgres_20260127_020000.dump.gz  # 2 AM backup (before corruption)

# 4. VERIFY DATA INTEGRITY
psql -U translate_user -d translate_db -c "SELECT COUNT(*) FROM users;"

# 5. RESTART SERVICE WITH ROLLBACK CODE
git checkout deploy-20260126-140000  # Before corruption
systemctl --user start presszone-backend.service

# 6. INVESTIGATE ROOT CAUSE (offline analysis)
# - Check application logs for errors
# - Review recent code changes
# - Test on staging with same data

# 7. COMMUNICATE WITH USERS (affected accounts)
# - Send email: "We restored from backup, some recent data may be lost"
```

---

### 13. Deployment Schedule Best Practices

**Recommended Deployment Windows:**

| Day | Time (UTC) | Reason |
|-----|------------|--------|
| **Tuesday-Thursday** | 10:00-14:00 | Midweek, team available for rollback |
| **Avoid Monday** | Any | Weekend issues may not be discovered yet |
| **Avoid Friday** | Any | No weekend support for rollback |
| **Avoid Nights/Weekends** | Any | Reduced monitoring, slower response |

**Deployment Frequency:**
- **Hotfixes (P0):** Deploy immediately (any time)
- **Security Patches:** Deploy within 24 hours
- **Feature Releases:** Weekly cadence (Tuesdays)
- **Major Versions:** Monthly, with 2-week notice to WordPress plugin users

---

### 14. Deployment Automation (Future)

**Current:** Manual deployment with shell scripts
**Goal:** CI/CD pipeline with automated testing and rollback

**Proposed GitHub Actions Workflow:**

```yaml
# .github/workflows/deploy-production.yml
name: Deploy to Production

on:
  push:
    tags:
      - 'v*.*.*'  # Trigger on version tags

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Run tests
        run: |
          cd api
          npm ci
          npm test

      - name: Build container image
        run: |
          cd api
          podman build -t presszone-api:${{ github.ref_name }} -f Dockerfile .

      - name: Push to registry (optional)
        run: |
          podman push presszone-api:${{ github.ref_name }} docker.io/presszone/api:${{ github.ref_name }}

      - name: Deploy to production (SSH)
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.PROD_HOST }}
          username: press
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          script: |
            cd ~/press-zone-backend
            git fetch --tags
            git checkout ${{ github.ref_name }}
            ./scripts/deploy-blue-green.sh ${{ github.ref_name }}

      - name: Run health checks
        run: |
          sleep 30
          curl -f https://api.press.zone/health || exit 1

      - name: Rollback on failure
        if: failure()
        run: |
          ssh press@${{ secrets.PROD_HOST }} './scripts/rollback.sh'
```

---

### 15. Validation Checklist

**Before considering deployment complete:**

- [ ] All health checks passing (`/health` returns 200)
- [ ] No errors in container logs (last 100 lines)
- [ ] Queue processing normally (backlog decreasing)
- [ ] Memory/CPU usage within normal range (< 500MB per container)
- [ ] Database migrations applied successfully (`prisma migrate status`)
- [ ] Old containers stopped and removed
- [ ] Nginx routing to correct port
- [ ] End-to-end test passed (translation job completed)
- [ ] Error rate < 0.1% (Prometheus metrics)
- [ ] Monitoring alerts not firing
- [ ] Rollback plan documented (git tag, backup timestamp)
- [ ] Team notified in Slack/Discord
- [ ] Deployment logged in audit log (`CHANGELOG.md` updated)

---

### 16. Rollback Speed Targets

**Service Level Objectives (SLOs):**

| Severity | Detection Time | Rollback Time | Total Time to Recovery |
|----------|----------------|---------------|------------------------|
| **P0 - Critical Outage** | < 2 minutes | < 5 minutes | < 7 minutes |
| **P1 - Major Degradation** | < 5 minutes | < 15 minutes | < 20 minutes |
| **P2 - Minor Issues** | < 15 minutes | < 30 minutes | < 45 minutes |

**Rollback Drill (Monthly):**
- Practice rollback on staging environment
- Measure time from "decision to rollback" to "service restored"
- Document friction points and improve scripts

---

## Summary

This runbook provides battle-tested procedures for:
1. **Blue-Green Deployment**: Zero-downtime updates with traffic switching
2. **Database Migrations**: Backward-compatible and multi-step breaking changes
3. **Systemd Management**: Graceful restarts and dependency handling
4. **Rolling Updates**: Worker process updates without losing queue jobs
5. **Emergency Rollback**: Fast recovery from failed deployments (< 7 minutes)

**Key Principles:**
- Always have a recent backup before deployment
- Test migrations on staging first
- Monitor health checks during and after deployment
- Practice rollback procedures regularly
- Document every deployment in git tags

**Next Steps:**
- Automate deployments with CI/CD pipeline (Section 14)
- Set up Prometheus alerting for failed deployments
- Create runbook for disaster recovery (full data center loss)

---

# Skill: System Settings Management

## Identity
- **Skill ID**: `system-settings-management`
- **Domain**: Configuration Management, Database-Backed Settings, In-Memory Caching
- **Technologies**: Prisma ORM, PostgreSQL (JSONB), In-Memory Cache
- **Source Agent**: `backend-app-agent.md`

## When to Load This Skill

Load this skill when working on:
- System configuration management
- Admin settings panel
- Dynamic configuration updates
- API key management (Gemini, PayPal)
- Pricing and credit allocation configuration
- Rate limit configuration
- Cache invalidation strategies

**File patterns:**
- `api/src/services/settingsService.ts`
- `api/src/routes/admin/settings.ts`
- Database schema: `SystemSetting` model

## Core Patterns

### 1. Architecture Overview

**Purpose:** Centralized system configuration stored in database with automatic in-memory caching.

**Key Features:**
- **Database-Backed:** All settings persisted in `system_setting` table (JSONB values)
- **5-Minute TTL Cache:** Reduces database load for frequently accessed settings
- **Atomic Updates:** Database updated first, then cache (consistency guaranteed)
- **Type-Safe Keys:** `SettingKey` enum prevents typos
- **Convenience Getters:** Bundled config methods (e.g., `getPayPalConfig()`)
- **Singleton Pattern:** Single shared instance across application

---

### 2. Database Schema

```prisma
// prisma/schema.prisma
model SystemSetting {
  key         String    @id @unique
  value       Json      // Flexible JSONB storage
  description String?
  updated_by  String?   // Admin user ID who last updated
  updated_at  DateTime  @default(now()) @updatedAt
}
```

**Field Descriptions:**
- `key` (String): Unique identifier (e.g., `"gemini_api_key"`)
- `value` (Json): JSONB field supporting strings, numbers, objects, arrays
- `description` (String?): Human-readable explanation of the setting
- `updated_by` (String?): UUID of admin user who made the change (audit trail)
- `updated_at` (DateTime): Automatic timestamp on every update

---

### 3. Setting Keys Enum

```typescript
export enum SettingKey {
  // Google Gemini API
  GEMINI_API_KEY = 'gemini_api_key',

  // PayPal Configuration
  PAYPAL_CLIENT_ID = 'paypal_client_id',
  PAYPAL_CLIENT_SECRET = 'paypal_client_secret',
  PAYPAL_MODE = 'paypal_mode', // 'sandbox' | 'live'
  PAYPAL_WEBHOOK_ID = 'paypal_webhook_id',

  // PayPal Plan IDs (6 subscription tiers)
  PAYPAL_PLAN_STARTER_MONTHLY = 'paypal_plan_starter_monthly',
  PAYPAL_PLAN_STARTER_ANNUAL = 'paypal_plan_starter_annual',
  PAYPAL_PLAN_PROFESSIONAL_MONTHLY = 'paypal_plan_professional_monthly',
  PAYPAL_PLAN_PROFESSIONAL_ANNUAL = 'paypal_plan_professional_annual',
  PAYPAL_PLAN_ENTERPRISE_MONTHLY = 'paypal_plan_enterprise_monthly',
  PAYPAL_PLAN_ENTERPRISE_ANNUAL = 'paypal_plan_enterprise_annual',

  // Pricing (USD)
  PRICING_PER_1K_TOKENS = 'pricing_per_1k_tokens',

  // Credit Allocations (monthly tokens)
  CREDITS_STARTER = 'credits_starter',
  CREDITS_PROFESSIONAL = 'credits_professional',
  CREDITS_ENTERPRISE = 'credits_enterprise',

  // Rate Limits (requests per minute)
  RATE_LIMIT_STARTER = 'rate_limit_starter',
  RATE_LIMIT_PROFESSIONAL = 'rate_limit_professional',
  RATE_LIMIT_ENTERPRISE = 'rate_limit_enterprise',
}
```

**Usage:**
```typescript
import { SettingKey, settingsService } from '../services/settingsService';

// Type-safe key usage
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);

// String keys also supported (for dynamic settings)
const customSetting = await settingsService.getSetting('custom_feature_flag');
```

---

### 4. Caching Strategy

**Cache Structure:**
```typescript
interface SettingsCache {
  data: Map<string, any>;      // In-memory key-value store
  lastRefresh: number;          // Unix timestamp of last DB fetch
  refreshInterval: number;      // 5 minutes (300,000 ms)
}

const cache: SettingsCache = {
  data: new Map(),
  lastRefresh: 0,
  refreshInterval: 5 * 60 * 1000,
};
```

**Cache Lifecycle:**
1. **Initial Load:** First `getSetting()` call triggers database fetch
2. **Cached Reads:** Subsequent reads served from memory (< 1ms latency)
3. **Auto-Refresh:** Cache refreshed every 5 minutes on next access
4. **Manual Invalidation:** `forceRefresh()` or `updateSetting()` triggers immediate refresh

**TTL Behavior:**
```typescript
private needsRefresh(): boolean {
  const now = Date.now();
  return now - cache.lastRefresh > cache.refreshInterval;
}
```
- Returns `true` if > 5 minutes since last refresh
- Lazy evaluation (only checked on access, not background timer)

---

### 5. Core Methods

#### 5.1 `getSystemSettings()`

**Signature:**
```typescript
async getSystemSettings(): Promise<Map<string, any>>
```

**Purpose:** Retrieve all settings as a Map.

**Returns:** Fresh copy of the settings Map (prevents external modification).

**Example:**
```typescript
const allSettings = await settingsService.getSystemSettings();

for (const [key, value] of allSettings.entries()) {
  console.log(`${key}: ${JSON.stringify(value)}`);
}
```

**Use Cases:**
- Admin dashboard settings page
- Exporting configuration
- Debugging/logging

---

#### 5.2 `getSetting<T>()`

**Signature:**
```typescript
async getSetting<T = any>(key: string | SettingKey): Promise<T | undefined>
```

**Purpose:** Get a single setting by key (returns `undefined` if not found).

**Type Parameter:** Generic `<T>` for return type casting.

**Example:**
```typescript
// String setting
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);
if (!apiKey) {
  throw new Error('Gemini API key not configured');
}

// Number setting
const pricePer1k = await settingsService.getSetting<number>(SettingKey.PRICING_PER_1K_TOKENS);

// Object setting (custom config)
interface FeatureFlags {
  enableBetaFeatures: boolean;
  maintenanceMode: boolean;
}
const flags = await settingsService.getSetting<FeatureFlags>('feature_flags');
```

---

#### 5.3 `getSettingWithDefault<T>()`

**Signature:**
```typescript
async getSettingWithDefault<T = any>(key: string | SettingKey, defaultValue: T): Promise<T>
```

**Purpose:** Get setting with fallback value (never returns `undefined`).

**Example:**
```typescript
// Pricing with default
const pricePer1k = await settingsService.getSettingWithDefault<number>(
  SettingKey.PRICING_PER_1K_TOKENS,
  0.002 // Default: $0.002 per 1K tokens
);

// Rate limit with default
const starterLimit = await settingsService.getSettingWithDefault<number>(
  SettingKey.RATE_LIMIT_STARTER,
  60 // Default: 60 requests/minute
);
```

**Use Cases:**
- Settings with sensible defaults
- Backward compatibility (new settings not yet in DB)

---

#### 5.4 `updateSetting()`

**Signature:**
```typescript
async updateSetting(
  key: string | SettingKey,
  value: any,
  description?: string,
  updatedBy?: string
): Promise<void>
```

**Purpose:** Create or update a setting (upsert operation).

**Behavior:**
1. Updates database (upsert: create if missing, update if exists)
2. Updates cache immediately (no waiting for TTL)
3. Logs change with admin user ID (audit trail)

**Example:**
```typescript
// Update pricing
await settingsService.updateSetting(
  SettingKey.PRICING_PER_1K_TOKENS,
  0.0025,
  'Increased pricing for new cost structure',
  adminUserId // UUID of admin making change
);

// Add new custom setting
await settingsService.updateSetting(
  'maintenance_mode',
  { enabled: true, message: 'Scheduled maintenance in progress' },
  'Maintenance mode configuration',
  adminUserId
);
```

**Atomicity:** Database and cache updated together (cache always reflects DB state).

---

#### 5.5 `deleteSetting()`

**Signature:**
```typescript
async deleteSetting(key: string | SettingKey): Promise<void>
```

**Purpose:** Remove a setting from database and cache.

**Example:**
```typescript
// Remove deprecated setting
await settingsService.deleteSetting('old_feature_flag');
```

**Throws:** Error if setting doesn't exist (Prisma exception).

---

#### 5.6 `forceRefresh()`

**Signature:**
```typescript
async forceRefresh(): Promise<void>
```

**Purpose:** Manually trigger cache refresh from database (ignores TTL).

**Use Cases:**
- After bulk database updates (manual SQL)
- Testing cache behavior
- Suspected cache staleness

**Example:**
```typescript
// After bulk update via SQL
await prisma.$executeRaw`
  UPDATE system_setting
  SET value = '0.003'
  WHERE key LIKE 'pricing_%'
`;

// Force cache refresh
await settingsService.forceRefresh();
```

---

#### 5.7 `clearCache()`

**Signature:**
```typescript
clearCache(): void
```

**Purpose:** Clear cache (for testing only, not used in production).

**Example:**
```typescript
// In test setup
beforeEach(() => {
  settingsService.clearCache();
});
```

---

### 6. Convenience Getter Methods

#### 6.1 `getGeminiConfig()`

**Signature:**
```typescript
async getGeminiConfig(): Promise<{ apiKey: string } | undefined>
```

**Returns:** Object with `apiKey` or `undefined` if not configured.

**Example:**
```typescript
const geminiConfig = await settingsService.getGeminiConfig();

if (!geminiConfig) {
  throw new Error('Gemini API not configured');
}

// Use apiKey
const response = await fetch('https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent', {
  headers: { 'x-goog-api-key': geminiConfig.apiKey }
});
```

---

#### 6.2 `getPayPalConfig()`

**Signature:**
```typescript
async getPayPalConfig(): Promise<{
  clientId: string;
  clientSecret: string;
  webhookId: string;
  mode: string;
  plans: {
    starterMonthly: string;
    starterAnnual: string;
    professionalMonthly: string;
    professionalAnnual: string;
    enterpriseMonthly: string;
    enterpriseAnnual: string;
  };
} | undefined>
```

**Returns:** Complete PayPal configuration bundle or `undefined` if any required field missing.

**Example:**
```typescript
const paypalConfig = await settingsService.getPayPalConfig();

if (!paypalConfig) {
  throw new Error('PayPal not configured');
}

// OAuth token generation
const authToken = await getPayPalAuthToken(
  paypalConfig.clientId,
  paypalConfig.clientSecret,
  paypalConfig.mode === 'sandbox' // Use sandbox URL
);

// Create subscription
const subscriptionId = await createPayPalSubscription(
  paypalConfig.plans.professionalMonthly,
  authToken
);
```

---

#### 6.3 `getPricingConfig()`

**Signature:**
```typescript
async getPricingConfig(): Promise<{ per1kTokens: number }>
```

**Returns:** Pricing configuration with default fallback (0.002).

**Example:**
```typescript
const { per1kTokens } = await settingsService.getPricingConfig();

// Calculate cost
const tokensUsed = 5000;
const costUSD = (tokensUsed / 1000) * per1kTokens;
console.log(`Cost: $${costUSD.toFixed(4)}`); // $0.0100
```

---

#### 6.4 `getCreditAllocations()`

**Signature:**
```typescript
async getCreditAllocations(): Promise<{
  starter: number;
  professional: number;
  enterprise: number;
}>
```

**Returns:** Monthly credit allocations for each tier (with defaults).

**Defaults:**
- Starter: 100,000 tokens
- Professional: 500,000 tokens
- Enterprise: 2,000,000 tokens

**Example:**
```typescript
const allocations = await settingsService.getCreditAllocations();

// Allocate credits on subscription activation
if (user.plan === 'professional') {
  await creditService.allocateCredits(
    user.id,
    allocations.professional,
    'Monthly subscription allocation'
  );
}
```

---

#### 6.5 `getRateLimits()`

**Signature:**
```typescript
async getRateLimits(): Promise<{
  starter: number;
  professional: number;
  enterprise: number;
}>
```

**Returns:** Rate limits (requests per minute) for each tier.

**Defaults:**
- Starter: 60 req/min
- Professional: 120 req/min
- Enterprise: 0 (unlimited)

**Example:**
```typescript
const limits = await settingsService.getRateLimits();

// Configure rate limiter
const userLimit = limits[user.plan as keyof typeof limits];
if (userLimit > 0) {
  await rateLimiter.set(`user:${user.id}`, userLimit);
}
```

---

### 7. Integration Points

#### Used By Services
```typescript
// api/src/services/geminiClient.ts
import { settingsService } from './settingsService';

const geminiConfig = await settingsService.getGeminiConfig();
const apiKey = geminiConfig?.apiKey;
```

```typescript
// api/src/services/paypalService.ts
import { settingsService } from './settingsService';

const paypalConfig = await settingsService.getPayPalConfig();
const authToken = await getOAuthToken(paypalConfig.clientId, paypalConfig.clientSecret);
```

#### Used By Middleware
```typescript
// api/src/middleware/rateLimiter.ts
import { settingsService } from '../services/settingsService';

async function getRateLimit(user: User): Promise<number> {
  const limits = await settingsService.getRateLimits();
  return limits[user.plan];
}
```

#### Used By Admin Routes
```typescript
// api/src/routes/admin/settings.ts
import { settingsService, SettingKey } from '../services/settingsService';

router.get('/admin/settings', async (req, res) => {
  const settings = await settingsService.getSystemSettings();
  res.json({ settings: Array.from(settings.entries()) });
});

router.patch('/admin/settings', async (req, res) => {
  const { key, value } = req.body;
  await settingsService.updateSetting(key, value, undefined, req.user.id);
  res.json({ success: true });
});
```

---

### 8. Cache Performance Characteristics

**Read Performance:**
- **Cache Hit (95% of reads):** < 1ms (in-memory Map lookup)
- **Cache Miss (first read or after TTL):** ~10-50ms (database query + cache population)
- **Concurrent Reads:** No contention (Map is singleton, Node.js single-threaded)

**Write Performance:**
- **Update:** ~20-80ms (database upsert + cache update)
- **Atomic:** Database and cache updated together (no race conditions)

**Cache Staleness Window:**
- **Max Staleness:** 5 minutes (TTL)
- **Typical Staleness:** < 1 minute (updates trigger immediate cache refresh)
- **Acceptable For:** Configuration settings that change infrequently

---

### 9. Error Handling

#### Cache Refresh Failure
```typescript
try {
  await refreshCache();
} catch (error) {
  logger.error('Failed to refresh settings cache', { error });
  throw new Error('Failed to refresh settings cache');
}
```
- **Behavior:** Throws error to surface failure to caller
- **Recommendation:** Implement retry logic in critical paths

#### Missing Settings
```typescript
const apiKey = await settingsService.getSetting<string>(SettingKey.GEMINI_API_KEY);

if (!apiKey) {
  // Handle missing configuration
  logger.error('Gemini API key not configured');
  throw new Error('Translation service not configured');
}
```

#### Update Failures
```typescript
try {
  await settingsService.updateSetting(key, value);
} catch (error) {
  logger.error('Failed to update setting', { key, error });
  return res.status(500).json({ error: 'Failed to update setting' });
}
```

---

### 10. Security Considerations

**Sensitive Data Storage:**
- API keys (Gemini, PayPal) stored in database (encrypted at rest via PostgreSQL)
- Never log sensitive values (logger filters `password`, `secret`, `apiKey` fields)
- Admin-only access to settings endpoints (authentication required)

**Audit Trail:**
- `updated_by` field tracks which admin made changes
- `updated_at` provides timestamp for change history
- Consider adding `SystemSettingHistory` table for full audit log

**Access Control:**
```typescript
// Middleware for admin-only settings access
import { authenticateAdmin } from '../middleware/auth';

router.patch('/admin/settings', authenticateAdmin, async (req, res) => {
  // Only admins can update settings
  await settingsService.updateSetting(req.body.key, req.body.value, undefined, req.user.id);
  res.json({ success: true });
});
```

---

### 11. Example: Complete Settings Workflow

```typescript
// 1. Admin updates pricing via dashboard
// POST /admin/settings
import { settingsService, SettingKey } from '../services/settingsService';

router.patch('/admin/settings', authenticateAdmin, async (req, res) => {
  const { key, value } = req.body;

  await settingsService.updateSetting(
    key,
    value,
    `Updated by admin: ${req.user.email}`,
    req.user.id
  );

  res.json({ success: true, message: 'Setting updated' });
});

// 2. Translation service reads updated pricing
// (within same 5-minute TTL window, cache refreshes)
import { settingsService } from './services/settingsService';

async function calculateTranslationCost(tokensUsed: number): Promise<number> {
  const { per1kTokens } = await settingsService.getPricingConfig();
  return (tokensUsed / 1000) * per1kTokens;
}

// 3. All subsequent requests use new pricing
const cost = await calculateTranslationCost(5000);
console.log(`Cost: $${cost.toFixed(4)}`); // Uses updated price
```

---

### 12. Testing Strategies

#### Unit Testing
```typescript
// __tests__/unit/services/settingsService.test.ts
import { settingsService } from '../../../services/settingsService';

describe('SettingsService', () => {
  beforeEach(() => {
    settingsService.clearCache(); // Reset cache between tests
  });

  it('should return setting from cache after first load', async () => {
    const value1 = await settingsService.getSetting('test_key');
    const value2 = await settingsService.getSetting('test_key');

    // Second call should be from cache (no DB query)
    expect(value1).toEqual(value2);
  });

  it('should return default value when setting not found', async () => {
    const value = await settingsService.getSettingWithDefault('missing_key', 'default');
    expect(value).toBe('default');
  });
});
```

#### Integration Testing
```typescript
// __tests__/integration/routes/admin/settings.test.ts
import request from 'supertest';
import { app } from '../../../server';

describe('POST /admin/settings', () => {
  it('should update setting and return success', async () => {
    const response = await request(app)
      .patch('/admin/settings')
      .set('Authorization', `Bearer ${adminToken}`)
      .send({ key: 'test_setting', value: 'new_value' })
      .expect(200);

    expect(response.body.success).toBe(true);

    // Verify setting was updated
    const setting = await settingsService.getSetting('test_setting');
    expect(setting).toBe('new_value');
  });
});
```

---

### 13. Troubleshooting Guide

| Issue | Diagnosis | Solution |
|-------|-----------|----------|
| Stale settings (old values) | Cache TTL not expired | Call `forceRefresh()` or wait 5 minutes |
| "Setting not found" errors | Key misspelled or not in DB | Check `SettingKey` enum, verify DB record |
| Slow setting reads | Cache not warming up | First read always hits DB (expected) |
| Settings not persisting | Database connection issue | Check Prisma client, verify DB connectivity |
| Inconsistent values across servers | Multiple API instances, no shared cache | Reduce TTL or use Redis for distributed cache |

---

### 14. Future Enhancements

**Planned:**
- **Distributed Cache:** Redis-backed cache for multi-instance deployments
- **Setting Validation:** JSON schema validation for setting values
- **Change History:** `SystemSettingHistory` table for audit log
- **Real-Time Updates:** WebSocket notifications when settings change
- **Setting Groups:** Organize settings into namespaces (e.g., `paypal.*`, `credits.*`)

**Not Planned:**
- Database encryption for values (rely on PostgreSQL encryption-at-rest)
- Setting versioning (single source of truth, no rollback needed)

---

This settings service provides reliable, cached configuration management with automatic refresh, type safety, and comprehensive admin controls.
# Skill 17: Testing & Quality Assurance

## Identity
- **Skill ID**: `testing-quality-assurance`
- **Domain**: Test Infrastructure, Jest Configuration, Unit/Integration Testing
- **Technologies**: Jest, TypeScript, Supertest, Prisma Mock, jest-mock-extended
- **Source Agent**: `backend-app-expert.md`

## When to Load This Skill

Load this skill when working on:
- Writing unit tests for services, utilities, and business logic
- Writing integration tests for API routes
- Configuring Jest test environment
- Mocking Prisma database calls
- Mocking Redis client
- Setting up test fixtures and helpers
- Configuring test coverage thresholds
- CI/CD test automation
- Test-driven development (TDD)

**File patterns:**
- `api/__tests__/**/*.test.ts`
- `api/__tests__/setup.ts`
- `api/jest.config.js`
- `api/src/**/*.test.ts`

---

## Core Testing Architecture

### 1. Test Directory Structure

```
api/__tests__/
├── setup.ts              # Global test configuration, mocks, utilities
├── unit/                 # Fast, isolated tests for business logic
│   ├── auth/
│   │   └── jwtService.test.ts
│   ├── services/
│   │   ├── creditService.test.ts
│   │   ├── tokenEstimator.test.ts
│   │   └── webhookService.test.ts
│   └── utils/
│       ├── encryption.test.ts
│       └── tokenCalculation.test.ts
├── integration/          # API route tests with mocked DB/Redis
│   └── routes/
│       ├── auth.test.ts
│       ├── health.test.ts
│       ├── translate.test.ts
│       └── webhooks.test.ts
└── e2e/                  # Full application tests (optional)
    └── translationFlow.test.ts
```

**Test Categories:**

| Type | Purpose | Speed | External Deps | Coverage Target |
|------|---------|-------|---------------|-----------------|
| **Unit** | Test individual functions/classes in isolation | < 50ms per test | None (all mocked) | 80%+ |
| **Integration** | Test API routes with mocked DB/Redis | < 500ms per test | Mocked Prisma/Redis | 70%+ |
| **E2E** | Test full workflows with real DB/Redis | < 5s per test | Real database | 50%+ |

---

## Jest Configuration

### 1. Jest Config File (`jest.config.js`)

```javascript
/**
 * Jest Configuration for translate-press-zone API
 *
 * Test Infrastructure:
 * - Unit tests: Fast, isolated tests for utility functions and business logic
 * - Integration tests: Test API routes with mocked external dependencies
 * - E2E tests: Full application tests with real database/Redis
 *
 * Run tests:
 *   npm test              # Run all tests
 *   npm run test:unit     # Run only unit tests
 *   npm run test:integration  # Run only integration tests
 *   npm run test:e2e      # Run only e2e tests
 *   npm run test:coverage # Run tests with coverage report
 *
 * Coverage reports are generated in ./coverage/
 */

module.exports = {
  // Use ts-jest preset for TypeScript support
  preset: 'ts-jest',

  // Node environment for backend testing
  testEnvironment: 'node',

  // Root directories for tests and source
  roots: ['<rootDir>/src', '<rootDir>/__tests__'],

  // Test match patterns
  testMatch: [
    '**/__tests__/**/*.test.ts',
    '**/?(*.)+(spec|test).ts',
  ],

  // Transform TypeScript files
  transform: {
    '^.+\\.ts$': ['ts-jest', {
      tsconfig: {
        // Use ES2022 target for tests
        target: 'ES2022',
        module: 'commonjs',
        esModuleInterop: true,
        skipLibCheck: true,
        forceConsistentCasingInFileNames: true,
      },
    }],
  },

  // Module name mapper for path aliases (if using)
  moduleNameMapper: {
    '^@/(.*)$': '<rootDir>/src/$1',
  },

  // Setup files to run before tests
  setupFilesAfterEnv: ['<rootDir>/__tests__/setup.ts'],

  // Coverage configuration
  collectCoverageFrom: [
    'src/**/*.ts',
    '!src/**/*.d.ts',
    '!src/**/*.test.ts',
    '!src/**/*.spec.ts',
    '!src/index.ts',       // Entry point
    '!src/server.ts',      // Server initialization
    '!src/worker.ts',      // Worker process
    '!src/types/**',       // Type definitions
  ],

  // Coverage thresholds (enforce minimum coverage)
  coverageThresholds: {
    global: {
      branches: 70,
      functions: 70,
      lines: 70,
      statements: 70,
    },
  },

  // Coverage reporters
  coverageReporters: [
    'text',           // Terminal output
    'text-summary',   // Summary in terminal
    'html',           // HTML report in coverage/
    'lcov',           // For CI/CD integration
  ],

  // Coverage directory
  coverageDirectory: '<rootDir>/coverage',

  // Ignore patterns
  testPathIgnorePatterns: [
    '/node_modules/',
    '/dist/',
    '/.next/',
  ],

  // Module file extensions
  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json', 'node'],

  // Global setup/teardown (optional)
  // globalSetup: '<rootDir>/__tests__/globalSetup.ts',
  // globalTeardown: '<rootDir>/__tests__/globalTeardown.ts',

  // Verbose output
  verbose: true,

  // Bail after first test failure (useful for CI)
  bail: false,

  // Timeout for tests (10 seconds default)
  testTimeout: 10000,

  // Clear mocks between tests
  clearMocks: true,

  // Restore mocks between tests
  restoreMocks: true,

  // Reset mocks between tests
  resetMocks: true,
};
```

**Key Configuration Options:**

| Option | Value | Purpose |
|--------|-------|---------|
| `preset` | `'ts-jest'` | TypeScript support |
| `testEnvironment` | `'node'` | Node.js environment (vs browser) |
| `setupFilesAfterEnv` | `['setup.ts']` | Run global setup before each test file |
| `coverageThresholds` | 70% | Fail build if coverage drops below threshold |
| `clearMocks` | `true` | Reset mock call history between tests |
| `testTimeout` | `10000` | 10-second timeout per test |

---

### 2. Test Setup File (`__tests__/setup.ts`)

```typescript
/**
 * Test Setup File
 *
 * This file runs before all tests to configure the test environment:
 * - Mock external dependencies (Prisma, Redis)
 * - Set up test environment variables
 * - Configure global test utilities
 * - Setup/teardown hooks
 *
 * Jest automatically loads this file via setupFilesAfterEnv in jest.config.js
 */

import { mockDeep, mockReset, DeepMockProxy } from 'jest-mock-extended';
import { PrismaClient } from '@prisma/client';
import Redis from 'ioredis';

/**
 * Mock Prisma Client
 *
 * Creates a deep mock of the Prisma client to avoid database calls in tests.
 * The mock is automatically injected into the application via jest.mock().
 */
export let prismaMock: DeepMockProxy<PrismaClient>;

jest.mock('@prisma/client', () => ({
  PrismaClient: jest.fn(() => {
    prismaMock = mockDeep<PrismaClient>();
    return prismaMock;
  }),
}));

/**
 * Mock Redis Client
 *
 * Creates a mock Redis client to avoid Redis calls in tests.
 * Common Redis methods are mocked with sensible defaults.
 */
export const mockRedisClient = {
  // Basic operations
  get: jest.fn(),
  set: jest.fn(),
  setex: jest.fn(),
  del: jest.fn(),
  expire: jest.fn(),
  ttl: jest.fn(),
  incr: jest.fn(),
  decr: jest.fn(),
  exists: jest.fn(),
  keys: jest.fn(),
  scan: jest.fn(),

  // Hash operations
  hget: jest.fn(),
  hset: jest.fn(),
  hdel: jest.fn(),
  hgetall: jest.fn(),

  // List operations
  lpush: jest.fn(),
  rpush: jest.fn(),
  lpop: jest.fn(),
  rpop: jest.fn(),
  lrange: jest.fn(),

  // Sorted set operations
  zadd: jest.fn(),
  zrange: jest.fn(),
  zrem: jest.fn(),

  // Connection management
  ping: jest.fn().mockResolvedValue('PONG'),
  quit: jest.fn().mockResolvedValue('OK'),
  disconnect: jest.fn(),

  // Event emitter methods
  on: jest.fn(),
  once: jest.fn(),
  off: jest.fn(),
  emit: jest.fn(),
};

jest.mock('ioredis', () => {
  return jest.fn().mockImplementation(() => mockRedisClient);
});

/**
 * Set up test environment variables
 *
 * Override environment variables for testing to avoid using production values.
 * These are minimal values needed for tests to run.
 */
process.env.NODE_ENV = 'test';
process.env.PORT = '3001';
process.env.DATABASE_URL = 'postgresql://test:test@localhost:5432/test';
process.env.REDIS_HOST = 'localhost';
process.env.REDIS_PORT = '6379';
process.env.REDIS_DB = '1'; // Use different DB for tests
process.env.JWT_ACCESS_SECRET = 'test-access-secret-32-chars-long-for-testing';
process.env.JWT_REFRESH_SECRET = 'test-refresh-secret-32-chars-long-for-testing';
process.env.JWT_ACCESS_EXPIRY = '15m';
process.env.JWT_REFRESH_EXPIRY = '7d';
process.env.GEMINI_API_KEY = 'test-gemini-api-key';
process.env.GEMINI_MODEL = 'gemini-3-flash-preview';
process.env.PAYPAL_CLIENT_ID = 'test-paypal-client-id';
process.env.PAYPAL_CLIENT_SECRET = 'test-paypal-client-secret';
process.env.PAYPAL_WEBHOOK_ID = 'test-paypal-webhook-id';
process.env.PAYPAL_MODE = 'sandbox';
process.env.PAYPAL_PLAN_STARTER_MONTHLY = 'P-STARTER-MONTHLY-TEST';
process.env.PAYPAL_PLAN_STARTER_ANNUAL = 'P-STARTER-ANNUAL-TEST';
process.env.PAYPAL_PLAN_PROFESSIONAL_MONTHLY = 'P-PRO-MONTHLY-TEST';
process.env.PAYPAL_PLAN_PROFESSIONAL_ANNUAL = 'P-PRO-ANNUAL-TEST';
process.env.PAYPAL_PLAN_ENTERPRISE_MONTHLY = 'P-ENT-MONTHLY-TEST';
process.env.PAYPAL_PLAN_ENTERPRISE_ANNUAL = 'P-ENT-ANNUAL-TEST';
process.env.FRONTEND_URL = 'https://test.translate.press.zone';
process.env.ADMIN_PANEL_URL = 'https://test-admin.translate.press.zone';
process.env.API_URL = 'https://test-api.translate.press.zone';
process.env.CORS_ALLOWED_ORIGINS = 'https://test.translate.press.zone,https://test-admin.translate.press.zone';
process.env.LOG_LEVEL = 'error'; // Reduce log noise in tests
process.env.PRICE_4B_MODEL = '0.001';
process.env.PRICE_27B_MODEL = '0.005';
process.env.CREDITS_STARTER = '100000';
process.env.CREDITS_PROFESSIONAL = '500000';
process.env.CREDITS_ENTERPRISE = '2000000';
process.env.RATE_LIMIT_STARTER = '60';
process.env.RATE_LIMIT_PROFESSIONAL = '120';
process.env.RATE_LIMIT_ENTERPRISE = '0';
process.env.MAX_SYNC_CHARS = '5000';
process.env.MAX_ASYNC_CHARS = '50000';

/**
 * Global test hooks
 *
 * Run before/after each test and before/after all tests
 */

// Before each test
beforeEach(() => {
  // Reset all mocks to clean state
  if (prismaMock) {
    mockReset(prismaMock);
  }

  // Reset Redis mock calls
  Object.values(mockRedisClient).forEach((fn) => {
    if (typeof fn === 'function' && fn.mockReset) {
      fn.mockReset();
    }
  });

  // Re-apply default mock implementations
  mockRedisClient.ping.mockResolvedValue('PONG');
  mockRedisClient.quit.mockResolvedValue('OK');
});

// After each test
afterEach(() => {
  // Clean up any timers, intervals, etc.
  jest.clearAllTimers();
});

// Before all tests
beforeAll(() => {
  // Set up global test utilities if needed
  // For example, increase timeout for all tests
  jest.setTimeout(10000);
});

// After all tests
afterAll(() => {
  // Clean up global resources
  // For example, close database connections (if using real DB in e2e tests)
});

/**
 * Global test utilities
 *
 * Helper functions available in all test files
 */

/**
 * Wait for a specific amount of time
 * Useful for testing async operations with delays
 */
export const wait = (ms: number): Promise<void> => {
  return new Promise((resolve) => setTimeout(resolve, ms));
};

/**
 * Mock timestamp for consistent test results
 * Use in tests where Date.now() or new Date() is called
 */
export const mockTimestamp = (timestamp: string | number): void => {
  const date = typeof timestamp === 'string' ? new Date(timestamp) : new Date(timestamp);
  jest.spyOn(global, 'Date').mockImplementation(() => date as any);
};

/**
 * Restore Date mock
 */
export const restoreDateMock = (): void => {
  jest.spyOn(global, 'Date').mockRestore();
};

/**
 * Generate a mock UUID (deterministic for tests)
 */
export const mockUUID = (seed: number = 1): string => {
  return `00000000-0000-4000-8000-${seed.toString().padStart(12, '0')}`;
};

/**
 * Generate a mock API key
 */
export const mockApiKey = (prefix: string = 'sk_test'): string => {
  return `${prefix}_${Math.random().toString(36).substring(2, 15)}`;
};

/**
 * Generate a mock JWT token
 */
export const mockJWT = (payload: Record<string, any>): string => {
  const header = Buffer.from(JSON.stringify({ alg: 'HS256', typ: 'JWT' })).toString('base64');
  const body = Buffer.from(JSON.stringify(payload)).toString('base64');
  const signature = 'mock-signature';
  return `${header}.${body}.${signature}`;
};

/**
 * Console suppression for cleaner test output
 *
 * Suppress console.log, console.warn, console.error during tests
 * to reduce noise. Comment out if you need to debug tests.
 */

// Uncomment to suppress console output in tests
// global.console = {
//   ...console,
//   log: jest.fn(),
//   warn: jest.fn(),
//   error: jest.fn(),
//   info: jest.fn(),
//   debug: jest.fn(),
// };
```

**Key Setup Features:**

1. **Prisma Mock**: Deep mock of Prisma client using `jest-mock-extended`
2. **Redis Mock**: Mock of all common Redis operations
3. **Environment Variables**: Test-specific values to avoid production conflicts
4. **Test Hooks**: `beforeEach`, `afterEach`, `beforeAll`, `afterAll`
5. **Test Utilities**: Helpers for waiting, mocking timestamps, generating fixtures

---

## Unit Testing Patterns

### 1. Example: JWT Service Unit Tests

**File:** `api/__tests__/unit/auth/jwtService.test.ts`

```typescript
/**
 * Unit Tests for JWT Service
 *
 * Tests JWT token generation, verification, and extraction
 */

import {
  generateAccessToken,
  generateRefreshToken,
  verifyAccessToken,
  verifyRefreshToken,
  extractTokenFromHeader,
} from '../../../auth/jwtService';
import jwt from 'jsonwebtoken';
import { config } from '../../../config';

describe('JWT Service', () => {
  const mockUserId = 'user_123';
  const mockEmail = 'test@example.com';
  const mockPlan = 'professional';
  const mockSubscriptionStatus = 'active';

  describe('generateAccessToken', () => {
    it('should generate valid access token', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);

      expect(token).toBeDefined();
      expect(typeof token).toBe('string');
      expect(token.split('.')).toHaveLength(3); // JWT has 3 parts
    });

    it('should include user data in token payload', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const decoded = jwt.decode(token) as any;

      expect(decoded.userId).toBe(mockUserId);
      expect(decoded.email).toBe(mockEmail);
      expect(decoded.plan).toBe(mockPlan);
      expect(decoded.subscriptionStatus).toBe(mockSubscriptionStatus);
      expect(decoded.type).toBe('access');
    });

    it('should include issuer and audience claims', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const decoded = jwt.decode(token) as any;

      expect(decoded.iss).toBe('translate.press.zone');
      expect(decoded.aud).toBe('api');
    });

    it('should include expiration time', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const decoded = jwt.decode(token) as any;

      expect(decoded.exp).toBeDefined();
      expect(decoded.exp).toBeGreaterThan(Date.now() / 1000);
    });

    it('should generate different tokens for different users', () => {
      const token1 = generateAccessToken('user_1', 'user1@test.com', mockPlan, mockSubscriptionStatus);
      const token2 = generateAccessToken('user_2', 'user2@test.com', mockPlan, mockSubscriptionStatus);

      expect(token1).not.toBe(token2);
    });
  });

  describe('verifyAccessToken', () => {
    it('should verify valid access token', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const payload = verifyAccessToken(token);

      expect(payload.userId).toBe(mockUserId);
      expect(payload.email).toBe(mockEmail);
      expect(payload.plan).toBe(mockPlan);
      expect(payload.subscriptionStatus).toBe(mockSubscriptionStatus);
      expect(payload.type).toBe('access');
    });

    it('should reject expired token', () => {
      // Create token with immediate expiration
      const expiredToken = jwt.sign(
        { userId: mockUserId, email: mockEmail, type: 'access' },
        config.jwtAccessSecret,
        { expiresIn: '-1s', issuer: 'translate.press.zone', audience: 'api' }
      );

      expect(() => verifyAccessToken(expiredToken)).toThrow('Access token expired');
    });

    it('should reject token with invalid signature', () => {
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const tamperedToken = token.slice(0, -10) + 'tampered123';

      expect(() => verifyAccessToken(tamperedToken)).toThrow('Invalid access token');
    });

    it('should reject token with wrong type', () => {
      const refreshToken = generateRefreshToken(mockUserId);

      expect(() => verifyAccessToken(refreshToken)).toThrow('Invalid token type');
    });

    it('should reject malformed token', () => {
      expect(() => verifyAccessToken('not.a.valid.jwt')).toThrow();
    });

    it('should reject empty token', () => {
      expect(() => verifyAccessToken('')).toThrow();
    });
  });

  describe('extractTokenFromHeader', () => {
    it('should extract token from valid Authorization header', () => {
      const token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...';
      const authHeader = `Bearer ${token}`;

      const extracted = extractTokenFromHeader(authHeader);

      expect(extracted).toBe(token);
    });

    it('should return null for missing header', () => {
      const extracted = extractTokenFromHeader(undefined);

      expect(extracted).toBeNull();
    });

    it('should return null for header without Bearer prefix', () => {
      const token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...';
      const authHeader = token;

      const extracted = extractTokenFromHeader(authHeader);

      expect(extracted).toBeNull();
    });

    it('should return null for wrong authentication scheme', () => {
      const token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...';
      const authHeader = `Basic ${token}`;

      const extracted = extractTokenFromHeader(authHeader);

      expect(extracted).toBeNull();
    });
  });

  describe('Token integration', () => {
    it('should support full access token flow', () => {
      // Generate token
      const token = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);

      // Extract from header
      const authHeader = `Bearer ${token}`;
      const extracted = extractTokenFromHeader(authHeader);

      // Verify token
      const payload = verifyAccessToken(extracted!);

      expect(payload.userId).toBe(mockUserId);
      expect(payload.email).toBe(mockEmail);
    });

    it('should maintain user identity through token renewal', () => {
      // Generate initial access token
      const initialToken = generateAccessToken(mockUserId, mockEmail, mockPlan, mockSubscriptionStatus);
      const initialPayload = verifyAccessToken(initialToken);

      // Generate new token for same user (simulating refresh)
      const renewedToken = generateAccessToken(
        initialPayload.userId,
        initialPayload.email,
        initialPayload.plan,
        initialPayload.subscriptionStatus
      );
      const renewedPayload = verifyAccessToken(renewedToken);

      expect(renewedPayload.userId).toBe(initialPayload.userId);
      expect(renewedPayload.email).toBe(initialPayload.email);
    });
  });
});
```

**Unit Test Patterns:**

1. **Test Success Paths**: Valid inputs return expected outputs
2. **Test Failure Paths**: Invalid inputs throw expected errors
3. **Test Edge Cases**: Empty strings, null values, boundary conditions
4. **Test Integration**: Multiple functions working together
5. **Test Security**: Token tampering, expiration, type validation

---

### 2. Example: Credit Service Unit Tests

**File:** `api/__tests__/unit/services/creditService.test.ts`

```typescript
/**
 * Unit Tests for Credit Service
 *
 * Tests credit allocation, deduction, refund, and balance tracking
 */

import { prismaMock } from '../../../__tests__/setup';
import * as creditService from '../../../services/creditService';
import { CreditTransactionType } from '@prisma/client';

describe('Credit Service', () => {
  beforeEach(() => {
    jest.clearAllMocks();
  });

  describe('getCurrentBalance', () => {
    it('should return latest balance from transaction history', async () => {
      const userId = 'user_123';
      const mockTransaction = {
        balance_after: 5000,
        created_at: new Date(),
      };

      prismaMock.creditTransaction.findFirst.mockResolvedValue(mockTransaction as any);

      const balance = await creditService.getCurrentBalance(userId);

      expect(balance).toBe(5000);
      expect(prismaMock.creditTransaction.findFirst).toHaveBeenCalledWith({
        where: { user_id: userId },
        orderBy: { created_at: 'desc' },
        select: { balance_after: true },
      });
    });

    it('should return 0 if no transactions exist', async () => {
      const userId = 'user_456';

      prismaMock.creditTransaction.findFirst.mockResolvedValue(null);

      const balance = await creditService.getCurrentBalance(userId);

      expect(balance).toBe(0);
    });

    it('should handle database errors', async () => {
      const userId = 'user_789';
      const error = new Error('Database connection failed');

      prismaMock.creditTransaction.findFirst.mockRejectedValue(error);

      await expect(creditService.getCurrentBalance(userId)).rejects.toThrow(error);
    });
  });

  describe('allocateCredits', () => {
    it('should allocate credits successfully', async () => {
      const userId = 'user_123';
      const amount = 100000;
      const description = 'Monthly subscription allocation';
      const currentBalance = 5000;
      const newBalance = currentBalance + amount;

      const mockTransaction = {
        id: 'txn_123',
        user_id: userId,
        type: CreditTransactionType.allocation,
        amount,
        balance_after: newBalance,
        description,
        related_payment_id: null,
        related_job_id: null,
        created_at: new Date(),
      };

      // Mock getCurrentBalance
      prismaMock.creditTransaction.findFirst.mockResolvedValue({ balance_after: currentBalance } as any);

      // Mock transaction
      prismaMock.$transaction.mockImplementation(async (callback: any) => {
        return callback({
          creditTransaction: {
            create: jest.fn().mockResolvedValue(mockTransaction),
          },
        });
      });

      const result = await creditService.allocateCredits(userId, amount, description);

      expect(result.amount).toBe(amount);
      expect(result.balance_after).toBe(newBalance);
      expect(result.type).toBe(CreditTransactionType.allocation);
    });

    it('should reject negative allocation amount', async () => {
      const userId = 'user_123';
      const amount = -1000;
      const description = 'Invalid allocation';

      await expect(
        creditService.allocateCredits(userId, amount, description)
      ).rejects.toThrow('Allocation amount must be positive');
    });

    it('should reject zero allocation amount', async () => {
      const userId = 'user_123';
      const amount = 0;
      const description = 'Invalid allocation';

      await expect(
        creditService.allocateCredits(userId, amount, description)
      ).rejects.toThrow('Allocation amount must be positive');
    });
  });

  describe('deductCredits', () => {
    it('should deduct credits successfully', async () => {
      const userId = 'user_123';
      const amount = 1000;
      const description = 'Translation job';
      const currentBalance = 5000;
      const newBalance = currentBalance - amount;

      const mockTransaction = {
        id: 'txn_123',
        user_id: userId,
        type: CreditTransactionType.deduction,
        amount,
        balance_after: newBalance,
        description,
        created_at: new Date(),
      };

      prismaMock.creditTransaction.findFirst.mockResolvedValue({ balance_after: currentBalance } as any);

      prismaMock.$transaction.mockImplementation(async (callback: any) => {
        return callback({
          creditTransaction: {
            create: jest.fn().mockResolvedValue(mockTransaction),
          },
        });
      });

      const result = await creditService.deductCredits(userId, amount, description);

      expect(result.amount).toBe(amount);
      expect(result.balance_after).toBe(newBalance);
    });

    it('should reject deduction with insufficient balance', async () => {
      const userId = 'user_123';
      const amount = 10000;
      const description = 'Large translation job';
      const currentBalance = 500;

      prismaMock.creditTransaction.findFirst.mockResolvedValue({ balance_after: currentBalance } as any);

      await expect(
        creditService.deductCredits(userId, amount, description)
      ).rejects.toThrow('Insufficient credits');
    });
  });
});
```

**Credit Service Test Patterns:**

1. **Mock Prisma Calls**: Use `prismaMock` to simulate database operations
2. **Test Transactions**: Mock `$transaction` for atomic operations
3. **Test Balance Calculations**: Verify arithmetic correctness
4. **Test Error Conditions**: Insufficient balance, negative amounts
5. **Test Call Parameters**: Verify correct queries are executed

---

## Integration Testing Patterns

### 1. Example: Auth Route Integration Tests

**File:** `api/__tests__/integration/routes/auth.test.ts`

```typescript
/**
 * Integration Tests for Authentication Routes
 *
 * Tests user registration, login, token refresh, email verification, and password reset
 */

import request from 'supertest';
import { createServer } from '../../../server';
import { prismaMock } from '../../setup';
import { generateAccessToken, generateRefreshToken } from '../../../auth/jwtService';
import { hashPassword } from '../../../utils/encryption';
import { Application } from 'express';

describe('Auth Routes', () => {
  let app: Application;

  beforeAll(() => {
    app = createServer();
  });

  describe('POST /v1/auth/register', () => {
    it('should register a new user successfully', async () => {
      const userData = {
        email: 'newuser@example.com',
        password: 'SecurePassword123!',
      };

      // Mock: User doesn't exist
      prismaMock.user.findUnique.mockResolvedValue(null);

      // Mock: User creation
      prismaMock.user.create.mockResolvedValue({
        id: 'user_123',
        email: userData.email,
        passwordHash: 'hashed_password',
        emailVerificationToken: 'verification_token',
        status: 'active',
        emailVerified: false,
        createdAt: new Date(),
        updatedAt: new Date(),
      } as any);

      const response = await request(app)
        .post('/v1/auth/register')
        .send(userData)
        .expect(201);

      expect(response.body).toMatchObject({
        message: 'User registered successfully. Please check your email to verify your account.',
        userId: 'user_123',
      });
    });

    it('should reject registration with existing email', async () => {
      const userData = {
        email: 'existing@example.com',
        password: 'SecurePassword123!',
      };

      // Mock: User already exists
      prismaMock.user.findUnique.mockResolvedValue({
        id: 'user_456',
        email: userData.email,
      } as any);

      const response = await request(app)
        .post('/v1/auth/register')
        .send(userData)
        .expect(400);

      expect(response.body.error.code).toBe('USER_EXISTS');
      expect(response.body.error.message).toContain('already exists');
    });

    it('should validate email format', async () => {
      const response = await request(app)
        .post('/v1/auth/register')
        .send({
          email: 'invalid-email',
          password: 'SecurePassword123!',
        })
        .expect(400);

      expect(response.body.error).toBeDefined();
    });

    it('should validate password length', async () => {
      const response = await request(app)
        .post('/v1/auth/register')
        .send({
          email: 'test@example.com',
          password: 'short',
        })
        .expect(400);

      expect(response.body.error).toBeDefined();
    });
  });

  describe('POST /v1/auth/login', () => {
    it('should login successfully with valid credentials', async () => {
      const credentials = {
        email: 'user@example.com',
        password: 'SecurePassword123!',
      };

      const passwordHash = await hashPassword(credentials.password);

      // Mock: User exists
      prismaMock.user.findUnique.mockResolvedValue({
        id: 'user_123',
        email: credentials.email,
        passwordHash,
        status: 'active',
        emailVerified: true,
        subscription: {
          planTier: 'professional',
          status: 'active',
        },
      } as any);

      const response = await request(app)
        .post('/v1/auth/login')
        .send(credentials)
        .expect(200);

      expect(response.body).toHaveProperty('accessToken');
      expect(response.body).toHaveProperty('refreshToken');
      expect(response.body.user).toMatchObject({
        id: 'user_123',
        email: credentials.email,
        plan: 'professional',
        subscriptionStatus: 'active',
      });
    });

    it('should reject login with invalid email', async () => {
      const credentials = {
        email: 'nonexistent@example.com',
        password: 'SecurePassword123!',
      };

      // Mock: User doesn't exist
      prismaMock.user.findUnique.mockResolvedValue(null);

      const response = await request(app)
        .post('/v1/auth/login')
        .send(credentials)
        .expect(401);

      expect(response.body.error.code).toBe('INVALID_CREDENTIALS');
    });

    it('should reject login for suspended account', async () => {
      const credentials = {
        email: 'suspended@example.com',
        password: 'SecurePassword123!',
      };

      const passwordHash = await hashPassword(credentials.password);

      // Mock: Suspended user
      prismaMock.user.findUnique.mockResolvedValue({
        id: 'user_456',
        email: credentials.email,
        passwordHash,
        status: 'suspended',
      } as any);

      const response = await request(app)
        .post('/v1/auth/login')
        .send(credentials)
        .expect(403);

      expect(response.body.error.code).toBe('ACCOUNT_SUSPENDED');
    });
  });

  describe('POST /v1/auth/refresh', () => {
    it('should refresh access token with valid refresh token', async () => {
      const userId = 'user_123';
      const refreshToken = generateRefreshToken(userId);

      // Mock: User exists
      prismaMock.user.findUnique.mockResolvedValue({
        id: userId,
        email: 'user@example.com',
        status: 'active',
        subscription: {
          planTier: 'professional',
          status: 'active',
        },
      } as any);

      const response = await request(app)
        .post('/v1/auth/refresh')
        .send({ refreshToken })
        .expect(200);

      expect(response.body).toHaveProperty('accessToken');
      expect(response.body.accessToken).toBeTruthy();
    });

    it('should reject refresh for suspended account', async () => {
      const userId = 'user_456';
      const refreshToken = generateRefreshToken(userId);

      // Mock: Suspended user
      prismaMock.user.findUnique.mockResolvedValue({
        id: userId,
        email: 'suspended@example.com',
        status: 'suspended',
      } as any);

      const response = await request(app)
        .post('/v1/auth/refresh')
        .send({ refreshToken })
        .expect(401);

      expect(response.body.error.code).toBe('INVALID_TOKEN');
    });
  });
});
```

**Integration Test Patterns:**

1. **Use Supertest**: Test HTTP endpoints without starting server
2. **Mock Database**: Use `prismaMock` to simulate DB responses
3. **Test HTTP Status Codes**: Verify correct response codes (200, 400, 401, 403)
4. **Test Response Structure**: Verify response body matches expected schema
5. **Test Error Codes**: Verify custom error codes (USER_EXISTS, INVALID_CREDENTIALS)
6. **Test Authentication Flow**: Login → Verify token → Refresh token

---

### 2. Example: Health Check Integration Tests

**File:** `api/__tests__/integration/routes/health.test.ts`

```typescript
/**
 * Integration Tests for Health Check Routes
 *
 * Tests the health, readiness, and liveness endpoints
 */

import request from 'supertest';
import { createServer } from '../../../server';
import { prismaMock, mockRedisClient } from '../../setup';
import { Application } from 'express';

describe('Health Routes', () => {
  let app: Application;

  beforeAll(() => {
    app = createServer();
  });

  describe('GET /health', () => {
    it('should return healthy status', async () => {
      const response = await request(app)
        .get('/health')
        .expect(200);

      expect(response.body).toMatchObject({
        status: 'healthy',
        service: 'translate-api',
        version: '1.0.0',
      });
      expect(response.body.timestamp).toBeDefined();
    });

    it('should have valid timestamp format', async () => {
      const response = await request(app)
        .get('/health')
        .expect(200);

      const timestamp = new Date(response.body.timestamp);
      expect(timestamp.toString()).not.toBe('Invalid Date');
    });
  });

  describe('GET /health/ready', () => {
    it('should return ready when all services are healthy', async () => {
      // Mock successful database query
      prismaMock.$queryRaw.mockResolvedValue([{ result: 1 }]);

      // Mock successful Redis ping
      mockRedisClient.ping.mockResolvedValue('PONG');
      mockRedisClient.quit.mockResolvedValue('OK');

      const response = await request(app)
        .get('/health/ready')
        .expect(200);

      expect(response.body).toMatchObject({
        status: 'ready',
        checks: {
          database: { status: 'healthy' },
          redis: { status: 'healthy' },
        },
      });
      expect(response.body.timestamp).toBeDefined();
    });

    it('should return 503 when database is unhealthy', async () => {
      // Mock database failure
      prismaMock.$queryRaw.mockRejectedValue(new Error('Connection refused'));

      // Mock successful Redis
      mockRedisClient.ping.mockResolvedValue('PONG');
      mockRedisClient.quit.mockResolvedValue('OK');

      const response = await request(app)
        .get('/health/ready')
        .expect(503);

      expect(response.body.status).toBe('not ready');
      expect(response.body.checks.database.status).toBe('unhealthy');
      expect(response.body.checks.database.error).toBeDefined();
    });

    it('should return 503 when Redis is unhealthy', async () => {
      // Mock successful database
      prismaMock.$queryRaw.mockResolvedValue([{ result: 1 }]);

      // Mock Redis failure
      mockRedisClient.ping.mockRejectedValue(new Error('Connection timeout'));

      const response = await request(app)
        .get('/health/ready')
        .expect(503);

      expect(response.body.status).toBe('not ready');
      expect(response.body.checks.redis.status).toBe('unhealthy');
      expect(response.body.checks.redis.error).toBeDefined();
    });
  });

  describe('GET /health/live', () => {
    it('should return alive status', async () => {
      const response = await request(app)
        .get('/health/live')
        .expect(200);

      expect(response.body).toMatchObject({
        status: 'alive',
      });
      expect(response.body.uptime).toBeDefined();
      expect(response.body.memory).toBeDefined();
      expect(response.body.timestamp).toBeDefined();
    });

    it('should include process uptime', async () => {
      const response = await request(app)
        .get('/health/live')
        .expect(200);

      expect(typeof response.body.uptime).toBe('number');
      expect(response.body.uptime).toBeGreaterThanOrEqual(0);
    });

    it('should always return 200 even if other services are down', async () => {
      // Mock service failures
      prismaMock.$queryRaw.mockRejectedValue(new Error('DB error'));
      mockRedisClient.ping.mockRejectedValue(new Error('Redis error'));

      // Liveness should still return 200
      const response = await request(app)
        .get('/health/live')
        .expect(200);

      expect(response.body.status).toBe('alive');
    });
  });

  describe('Health endpoint consistency', () => {
    it('should have consistent timestamp format across all endpoints', async () => {
      const health = await request(app).get('/health');
      const ready = await request(app).get('/health/ready');
      const live = await request(app).get('/health/live');

      const timestamps = [
        health.body.timestamp,
        ready.body.timestamp,
        live.body.timestamp,
      ];

      timestamps.forEach((timestamp) => {
        expect(new Date(timestamp).toString()).not.toBe('Invalid Date');
      });
    });

    it('should respond quickly (< 1 second)', async () => {
      const start = Date.now();

      await request(app).get('/health');

      const duration = Date.now() - start;
      expect(duration).toBeLessThan(1000);
    });
  });
});
```

**Health Check Test Patterns:**

1. **Test All Health Endpoints**: `/health`, `/health/ready`, `/health/live`
2. **Test Service Dependencies**: Database, Redis connectivity
3. **Test Failure Scenarios**: DB down, Redis down, both down
4. **Test Response Times**: Verify fast responses (< 1 second)
5. **Test Consistency**: Verify timestamp format, response structure
6. **Test Liveness Independence**: Liveness should always return 200

---

## Test Coverage Requirements

### 1. Coverage Thresholds

```javascript
// jest.config.js
coverageThresholds: {
  global: {
    branches: 70,      // 70% of if/else branches tested
    functions: 70,     // 70% of functions tested
    lines: 70,         // 70% of code lines tested
    statements: 70,    // 70% of statements tested
  },
}
```

**Coverage Targets by Module:**

| Module | Target Coverage | Priority |
|--------|-----------------|----------|
| **Auth Services** | 90%+ | Critical |
| **Credit Management** | 90%+ | Critical |
| **Payment Integration** | 85%+ | High |
| **Translation Service** | 85%+ | High |
| **Webhook Delivery** | 80%+ | High |
| **Admin Routes** | 75%+ | Medium |
| **Utilities** | 80%+ | Medium |
| **Email Service** | 70%+ | Low (external dep) |

---

### 2. Running Tests

```bash
# Run all tests
npm test

# Run only unit tests
npm run test:unit

# Run only integration tests
npm run test:integration

# Run only e2e tests
npm run test:e2e

# Run tests with coverage report
npm run test:coverage

# Run tests in watch mode (for development)
npm run test:watch

# Run tests for a specific file
npm test -- jwtService.test.ts

# Run tests matching a pattern
npm test -- --testNamePattern="should generate valid access token"
```

**Package.json Scripts:**

```json
{
  "scripts": {
    "test": "jest",
    "test:unit": "jest __tests__/unit",
    "test:integration": "jest __tests__/integration",
    "test:e2e": "jest __tests__/e2e",
    "test:coverage": "jest --coverage",
    "test:watch": "jest --watch",
    "test:ci": "jest --ci --coverage --maxWorkers=2"
  }
}
```

---

### 3. Coverage Reports

After running `npm run test:coverage`, view reports:

```bash
# Terminal summary
# (printed automatically)

# HTML report (detailed, interactive)
open coverage/index.html

# LCOV report (for CI/CD integration)
cat coverage/lcov.info
```

**HTML Coverage Report Features:**
- Line-by-line coverage visualization (green = covered, red = uncovered)
- Branch coverage (which if/else paths were tested)
- Function coverage (which functions were called)
- File-level summary with drill-down

---

## CI/CD Integration

### 1. GitHub Actions Workflow

**File:** `.github/workflows/test.yml`

```yaml
name: Test Suite

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run database migrations
        run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/test

      - name: Run tests with coverage
        run: npm run test:ci
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/test
          REDIS_HOST: localhost
          REDIS_PORT: 6379
          NODE_ENV: test

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info
          flags: unittests
          name: codecov-umbrella

      - name: Check coverage thresholds
        run: |
          COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
          if (( $(echo "$COVERAGE < 70" | bc -l) )); then
            echo "Coverage is below 70% ($COVERAGE%)"
            exit 1
          fi
```

**CI/CD Test Features:**

1. **Automated Testing**: Run tests on every push and PR
2. **Service Containers**: Spin up Postgres and Redis for integration tests
3. **Coverage Upload**: Send coverage reports to Codecov
4. **Coverage Gate**: Fail build if coverage drops below threshold
5. **Parallel Execution**: Use `--maxWorkers=2` for faster CI tests

---

### 2. Pre-commit Hooks (Husky)

**File:** `.husky/pre-commit`

```bash
#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

# Run tests before commit
npm run test:unit

# Run linter
npm run lint

# Check types
npm run type-check
```

**Installation:**

```bash
npm install --save-dev husky
npx husky install
npx husky add .husky/pre-commit "npm run test:unit"
```

---

## Testing Best Practices

### 1. Test Naming Convention

```typescript
describe('ModuleName', () => {
  describe('functionName', () => {
    it('should [expected behavior] when [condition]', () => {
      // Test implementation
    });
  });
});
```

**Examples:**

```typescript
// ✅ GOOD
it('should return 401 when JWT token is expired', () => {});
it('should allocate credits successfully when balance is sufficient', () => {});
it('should return null when authorization header is missing', () => {});

// ❌ BAD
it('works', () => {});
it('test login', () => {});
it('JWT', () => {});
```

---

### 2. Test Organization

```typescript
describe('CreditService', () => {
  // Setup
  beforeEach(() => {
    jest.clearAllMocks();
  });

  // Success path tests
  describe('allocateCredits', () => {
    it('should allocate credits successfully', () => {});
    it('should include payment ID when provided', () => {});
  });

  // Failure path tests
  describe('allocateCredits - error cases', () => {
    it('should reject negative allocation amount', () => {});
    it('should reject zero allocation amount', () => {});
    it('should handle database errors', () => {});
  });

  // Edge cases
  describe('allocateCredits - edge cases', () => {
    it('should handle first transaction for new user', () => {});
    it('should handle concurrent allocations', () => {});
  });
});
```

---

### 3. Mock Best Practices

```typescript
// ✅ GOOD: Mock only what's necessary
prismaMock.user.findUnique.mockResolvedValue({
  id: 'user_123',
  email: 'test@example.com',
  status: 'active',
} as any);

// ❌ BAD: Over-mocking
prismaMock.user.findUnique.mockResolvedValue({
  id: 'user_123',
  email: 'test@example.com',
  status: 'active',
  passwordHash: 'hash',
  emailVerified: true,
  createdAt: new Date(),
  updatedAt: new Date(),
  // ... 50 more fields
} as any);

// ✅ GOOD: Reset mocks in beforeEach
beforeEach(() => {
  jest.clearAllMocks();
});

// ❌ BAD: Leaking mocks between tests
// (no cleanup)
```

---

### 4. Assertion Best Practices

```typescript
// ✅ GOOD: Specific assertions
expect(response.body.error.code).toBe('USER_EXISTS');
expect(response.body.user.id).toBe('user_123');
expect(token.split('.')).toHaveLength(3);

// ❌ BAD: Vague assertions
expect(response.body.error).toBeTruthy();
expect(response.body.user).toBeDefined();

// ✅ GOOD: Test error messages
expect(() => verifyToken('invalid')).toThrow('Invalid token');

// ❌ BAD: Don't test exact error message (fragile)
expect(() => verifyToken('invalid')).toThrow('Invalid token: jwt malformed at position 5');
```

---

### 5. Test Data Factories

```typescript
// utilities/testFactories.ts

export const createMockUser = (overrides?: Partial<User>): User => ({
  id: 'user_123',
  email: 'test@example.com',
  status: 'active',
  emailVerified: true,
  createdAt: new Date(),
  updatedAt: new Date(),
  ...overrides,
});

export const createMockSubscription = (overrides?: Partial<Subscription>): Subscription => ({
  id: 'sub_123',
  userId: 'user_123',
  planTier: 'professional',
  status: 'active',
  ...overrides,
});

// Usage in tests
it('should handle suspended user', () => {
  const suspendedUser = createMockUser({ status: 'suspended' });
  prismaMock.user.findUnique.mockResolvedValue(suspendedUser);

  // Test logic
});
```

---

## Common Testing Pitfalls

### 1. Anti-Patterns to Avoid

| Anti-Pattern | Problem | Solution |
|--------------|---------|----------|
| **Testing Implementation Details** | Tests break when refactoring | Test behavior, not implementation |
| **Brittle Tests** | Tests fail on unrelated changes | Use flexible matchers (toMatchObject) |
| **Slow Tests** | CI takes too long | Mock external dependencies |
| **Flaky Tests** | Tests pass/fail randomly | Avoid timing dependencies, use deterministic mocks |
| **Test Duplication** | Same test logic repeated | Use test factories and shared utilities |
| **No Error Testing** | Only test success paths | Test failure paths and edge cases |
| **Over-Mocking** | Mocks don't reflect reality | Mock minimally, test integration points |

---

### 2. Debugging Failing Tests

```bash
# Run a single test file
npm test -- jwtService.test.ts

# Run a specific test by name
npm test -- --testNamePattern="should verify valid access token"

# Run tests in debug mode
node --inspect-brk node_modules/.bin/jest --runInBand

# View detailed error output
npm test -- --verbose

# Disable console suppression (if enabled)
# Comment out console mock in setup.ts
```

---

## Test Metrics Dashboard

### 1. Key Metrics to Track

| Metric | Target | Measurement |
|--------|--------|-------------|
| **Code Coverage** | 80%+ | Jest coverage report |
| **Test Execution Time** | < 2 minutes | CI pipeline duration |
| **Test Flakiness** | < 1% | Failed then passed without code changes |
| **Test Count** | 500+ tests | Jest summary output |
| **Coverage Trend** | Increasing | Weekly coverage reports |

---

### 2. Coverage by Module (Example)

```
---------------------------|---------|----------|---------|---------|
File                       | % Stmts | % Branch | % Funcs | % Lines |
---------------------------|---------|----------|---------|---------|
All files                  |   82.45 |    78.12 |   85.67 |   82.89 |
 auth/                     |   95.23 |    92.45 |   98.12 |   95.67 |
  jwtService.ts            |   98.45 |    96.78 |  100.00 |   98.90 |
  apiKeyService.ts         |   92.34 |    88.56 |   96.45 |   92.78 |
 services/                 |   88.12 |    84.56 |   90.23 |   88.45 |
  creditService.ts         |   94.56 |    91.23 |   96.78 |   94.89 |
  translationService.ts    |   82.34 |    78.45 |   85.67 |   82.67 |
  webhookService.ts        |   87.23 |    83.45 |   89.12 |   87.56 |
 routes/                   |   75.45 |    71.23 |   78.90 |   75.78 |
  auth.ts                  |   85.67 |    82.34 |   88.45 |   85.90 |
  translate.ts             |   72.34 |    68.45 |   75.67 |   72.56 |
  webhooks.ts              |   68.45 |    64.23 |   71.12 |   68.78 |
---------------------------|---------|----------|---------|---------|
```

---

## Validation Checklist

Before marking Skill 17 as complete, verify:

**Test Infrastructure:**
- [ ] Jest configured with TypeScript support
- [ ] Test setup file creates Prisma and Redis mocks
- [ ] Environment variables set for test environment
- [ ] Coverage thresholds configured (70%+)
- [ ] Test scripts added to package.json

**Unit Tests:**
- [ ] JWT Service tests (token generation, verification, extraction)
- [ ] Credit Service tests (allocation, deduction, balance)
- [ ] Encryption tests (hashing, comparison)
- [ ] Token calculation tests (estimation accuracy)
- [ ] All utility functions tested

**Integration Tests:**
- [ ] Auth routes tested (register, login, refresh)
- [ ] Health check routes tested (health, ready, live)
- [ ] Translation routes tested (estimate, submit)
- [ ] Webhook routes tested (delivery, signature verification)
- [ ] Admin routes tested (CRUD operations)

**Test Quality:**
- [ ] Success paths tested (happy path scenarios)
- [ ] Failure paths tested (error conditions)
- [ ] Edge cases tested (boundary conditions, null values)
- [ ] Security scenarios tested (token tampering, unauthorized access)
- [ ] Error codes verified (correct HTTP status codes)

**Coverage:**
- [ ] Coverage report generated successfully
- [ ] Coverage thresholds met (70%+ global)
- [ ] Critical modules have 85%+ coverage
- [ ] Coverage trends tracked in CI/CD

**CI/CD:**
- [ ] GitHub Actions workflow configured
- [ ] Tests run on every push and PR
- [ ] Coverage uploaded to Codecov
- [ ] Pre-commit hooks prevent untested commits

**Documentation:**
- [ ] Test patterns documented with examples
- [ ] Test factories created for common fixtures
- [ ] Debugging guide included
- [ ] Best practices documented

---

## Summary

This skill provides comprehensive testing infrastructure for the translate-press-zone backend API:

**Test Framework:**
- Jest with TypeScript support
- Supertest for HTTP endpoint testing
- jest-mock-extended for Prisma mocking
- 70%+ coverage thresholds enforced

**Test Categories:**
1. **Unit Tests**: Fast, isolated tests for business logic (< 50ms)
2. **Integration Tests**: API route tests with mocked DB/Redis (< 500ms)
3. **E2E Tests**: Full workflow tests with real dependencies (< 5s)

**Coverage Examples:**
- JWT Service: Token generation, verification, extraction (98%+ coverage)
- Auth Routes: Registration, login, refresh, password reset (85%+ coverage)
- Health Routes: Health checks, readiness, liveness (90%+ coverage)
- Credit Service: Allocation, deduction, balance tracking (94%+ coverage)

**CI/CD Integration:**
- GitHub Actions workflow for automated testing
- Codecov integration for coverage tracking
- Pre-commit hooks for local testing
- Coverage gates to prevent regressions

**Key Testing Patterns:**
- Mock Prisma and Redis to avoid external dependencies
- Test success paths, failure paths, and edge cases
- Use specific assertions (avoid vague toBeTruthy)
- Reset mocks in beforeEach hooks
- Use test factories for reusable fixtures

**Next Steps:**
- Write tests for remaining services (Translation, Gemini, PayPal)
- Increase coverage to 85%+ for critical modules
- Add E2E tests for full translation workflows
- Set up mutation testing with Stryker
- Create performance benchmarks with Artillery

---