# ChatGPT-Computer-MCP — Specification

**Status:** Canonical v0.2 specification
**Version:** 0.2.0
**Runtime:** Linux, Node.js 22+, TypeScript, ESM, pnpm 11
**Protocol:** MCP 2026-07-28
**Primary transport:** stdio

## 1. Purpose

`chatgpt-computer-mcp` is a thin visual-computer-use MCP for Linux. ChatGPT is the planner. This server only reports the current visual desktop state and executes explicit input actions requested by the MCP client.

It is intentionally separate from `chatgpt-mcp` and has no filesystem, shell, process, service, browser-navigation, local planner, OCR, model, or autonomous agent surface.

## 2. Product contract

The public MCP surface is exactly:

- `computer.info`
- `computer.observe`
- `computer.act`

Desktop implementation details such as `xdotool`, `xrandr`, `ffmpeg`, portals, PipeWire, and libei are private backend details and never appear as MCP tools.

The observation/action loop is explicit:

```text
ChatGPT
  -> computer.info      (when topology/capabilities matter)
  -> computer.observe   (one screenshot request)
  -> reason in ChatGPT
  -> computer.act       (one explicit action)
  -> computer.observe   (if another observation is needed)
```

The server does not perform hidden retries, hidden observe-act loops, automatic recovery, OCR, visual interpretation, or local task planning.

## 3. Non-goals

v0.2 does not:

- embed or call an LLM;
- perform OCR;
- maintain a task plan;
- infer UI intent from pixels;
- retry failed interactions automatically;
- batch an autonomous action sequence behind one MCP call;
- provide a shell or expose desktop helper programs;
- claim native Wayland control through X11 compatibility tools;
- bypass compositor/user-consent boundaries.

## 4. MCP protocol

The canonical MCP revision is `2026-07-28`.

The stdio entrypoint must use the MCP TypeScript SDK v2 `serveStdio(factory)` path so modern connections negotiate the 2026-07-28 era. Legacy 2025-era openings are rejected by default.

No application correctness depends on MCP session state. Backend state is limited to host topology/cache data that is not client identity or task state.

## 5. Architecture

```text
ChatGPT / MCP host
        |
        | MCP 2026-07-28 over stdio
        v
+---------------------------+
| MCP tool layer            |
| schemas + result mapping  |
+-------------+-------------+
              |
              v
+---------------------------+
| ComputerBackend interface |
| host-neutral typed seam   |
+-------------+-------------+
              |
              +------------------------------+
              |                              |
              v                              v
+---------------------------+   +---------------------------+
| X11Backend                |   | WaylandBackend            |
| xrandr / ffmpeg /         |   | portal + PipeWire + libei |
| xdotool internally        |   | compositor-approved path  |
+---------------------------+   +---------------------------+
```

`ComputerBackend` contains no MCP SDK types. MCP handlers know only the backend interface and typed desktop models.

## 6. Coordinate model

Coordinates are integers in the active backend's declared coordinate unit. `computer.info.coordinateUnit` is authoritative:

- X11: `physical-pixel`;
- Wayland: `logical-pixel` from the compositor/libei absolute region model.

Two coordinate spaces are accepted:

1. `desktop`: absolute coordinates in the backend virtual desktop.
2. `display`: coordinates relative to a named display's top-left corner.

A display-relative point is converted once to desktop coordinates before execution. Bounds are validated against the selected display and virtual desktop.

`computer.info` reports:

- the virtual desktop rectangle;
- every display ID/name;
- each display's desktop `x`, `y`, `width`, `height`;
- `pixelWidth` / `pixelHeight` for the capture stream/frame;
- `scale` between logical/display width and capture pixels;
- which display is primary when known;
- supported coordinate spaces.

The backend never silently equates Wayland logical coordinates with PipeWire frame pixels.

## 7. Tool contracts

### 7.1 `computer.info`

Read-only. Returns current backend/session/topology/capability metadata.

Minimum result:

```json
{
  "backend": "x11",
  "sessionType": "x11",
  "coordinateUnit": "physical-pixel",
  "coordinateSpaces": ["desktop", "display"],
  "virtualBounds": { "x": 0, "y": 0, "width": 5120, "height": 1080 },
  "displays": [
    { "id": "HDMI-1", "name": "HDMI-1", "x": 0, "y": 0, "width": 1920, "height": 1080, "primary": true, "pixelWidth": 1920, "pixelHeight": 1080, "scale": 1 }
  ],
  "capabilities": {
    "observe": true,
    "pointer": true,
    "keyboard": true,
    "text": true
  }
}
```

### 7.2 `computer.observe`

Captures one PNG screenshot. Default target is the entire virtual desktop.

Targets:

- whole desktop;
- one display by `displayId`;
- one rectangular region in desktop coordinates;
- one rectangular region relative to a display.

Input may choose whether the mouse cursor is drawn when the capture backend supports it.

The MCP result contains:

- an MCP `image` content item (`image/png`);
- structured capture metadata including exact desktop region, encoded `pixelWidth` / `pixelHeight`, byte size, backend, and capture timestamp.

The server does not inspect or interpret the image.

### 7.3 `computer.act`

Executes exactly one explicit action per call.

Supported action variants:

- `move`
- `click`
- `doubleClick`
- `scroll`
- `drag`
- `mouseDown`
- `mouseUp`
- `key`
- `text`

Pointer actions may include a desktop- or display-relative point. Actions that omit a point operate at the current pointer location.

Buttons: `left`, `middle`, `right`.

Scroll semantics:

- positive `deltaY`: down;
- negative `deltaY`: up;
- positive `deltaX`: right;
- negative `deltaX`: left.

Deltas are discrete wheel steps and are bounded by configuration.

Keyboard `key` accepts explicit key/chord names such as `ctrl+l`, `Return`, or `alt+F4`. This is an action semantic; backend helpers (`xdotool` or libei/XKB) remain private.

Text input is literal text and has a configured byte cap.

## 8. X11 backend

X11 remains the compatibility/legacy desktop backend.

### 8.1 Topology

Use `xrandr --listmonitors` to discover active displays and their framebuffer coordinates.

### 8.2 Screenshot capture

Preferred capture provider: FFmpeg `x11grab`, producing PNG bytes through stdout. Capture dimensions and offsets come from validated topology/target resolution.

The installer may install required host packages. The MCP interface does not expose capture commands or arguments.

### 8.3 Input

Use `xdotool` internally for X11 input emulation:

- pointer movement;
- button down/up/click/double-click;
- scroll wheel steps;
- drag;
- keyboard chords;
- literal text.

`xdotool` is an implementation detail behind `X11Backend`, never a public tool contract.

## 9. Wayland backend — v0.2

Native Wayland control uses compositor-approved interfaces rather than X11 compatibility or privileged input devices:

1. verify `RemoteDesktop.AvailableDeviceTypes` includes pointer + keyboard;
2. verify ScreenCast monitor sources and cursor-metadata mode are available;
3. create one `org.freedesktop.portal.RemoteDesktop` session;
4. select pointer/keyboard devices;
5. use `org.freedesktop.portal.ScreenCast.SelectSources` on that session to select monitor sources;
6. start the session and retain selected stream metadata;
7. call `ScreenCast.OpenPipeWireRemote` and consume authorized monitor streams;
8. call `RemoteDesktop.ConnectToEIS` and use libei as the input sender;
9. map PipeWire streams to libei absolute regions by `mapping_id`;
10. synchronize exposed logical display geometry to the matching libei input region;
11. encode exactly one requested observation as PNG, with cursor metadata composited only when requested;
12. terminate the portal authority when the MCP closes or the compositor revokes the session.

The TypeScript `WaylandBackend` implements the same `ComputerBackend` seam as X11. A native C helper owns GLib/GIO D-Bus, PipeWire, libei, libpng, and XKB interoperability; it is a backend implementation detail, not an MCP tool.

`computer.info` uses logical coordinates for Wayland and separately reports actual PipeWire frame pixel dimensions and scale. The backend does not assume portal stream `size`, PipeWire pixels, and libei coordinates are interchangeable.

Literal text uses libei's text capability when the installed runtime provides it; otherwise it falls back to the current libei/XKB keymap and fails clearly for unrepresentable characters.

Portal consent/revocation is authoritative. The MCP never bypasses denied permissions and never mixes `ConnectToEIS` with X11 input pretending to be native Wayland support.

## 10. Errors and execution behavior

Errors are stable, structured failures such as:

- `UNSUPPORTED_PLATFORM`
- `BACKEND_UNAVAILABLE`
- `DISPLAY_NOT_FOUND`
- `OUT_OF_BOUNDS`
- `INVALID_INPUT`
- `DEPENDENCY_MISSING`
- `ACTION_DISABLED`
- `OBSERVE_DISABLED`
- `OUTPUT_LIMIT`
- `OS_ERROR`

A failed action is returned to ChatGPT immediately. The server does not retry it or perform compensating actions unless ChatGPT explicitly sends another MCP call.

## 11. Configuration and security

`config.local.json`, `.secrets/`, tunnel profiles/credentials, generated service files, logs, screenshots, build output, and activation artifacts are local-only and gitignored.

Configuration controls at minimum:

- backend selection (`auto`, `x11`, or `wayland`);
- X display override;
- Wayland native-helper path and portal/frame timeouts;
- observation enablement and maximum PNG byte size;
- action enablement;
- maximum literal text bytes;
- maximum scroll steps per call.

No secret is required by the MCP server itself. Secure MCP Tunnel credentials belong to the tunnel client and remain outside tracked files.

## 12. Installation and ChatGPT connection

Primary connection:

```text
ChatGPT Developer Mode
        |
OpenAI Secure MCP Tunnel
        |
        | outbound HTTPS
        v
tunnel-client on Linux
        |
        | stdio
        v
ChatGPT-Computer-MCP
```

The plugin/app name in ChatGPT instructions is **ChatGPT-Computer-MCP**.

## 13. Quality gates

Before v0.2 publication:

- dependency install succeeds with pnpm 11;
- TypeScript strict typecheck passes;
- unit/integration tests pass;
- build passes;
- full `pnpm gate` passes;
- shell scripts pass `bash -n`;
- `package.json` is exactly version `0.2.0`;
- MIT license exists with copyright `Alex`;
- tracked files and staged diff are scanned for likely secrets;
- no local credentials/config are tracked;
- GitHub repository is private and default branch is `main`.