# ChatGPT-Computer-MCP

A separate Linux MCP dedicated to **visual computer use**. ChatGPT remains the planner; this server only observes the desktop and executes explicit input actions.

```text
ChatGPT / MCP client
        |
        | MCP 2026-07-28
        v
ChatGPT-Computer-MCP
        |
        | ComputerBackend
        +--> X11 backend
        `--> Wayland backend
```

The public MCP surface is intentionally only:

| Tool | Purpose |
| --- | --- |
| `computer.info` | Report backend, displays, virtual desktop bounds, coordinate spaces, and capabilities |
| `computer.observe` | Capture one PNG screenshot of the desktop, a display, or a region |
| `computer.act` | Execute one explicit pointer/keyboard/text action |

There is no embedded LLM, OCR requirement, local task planner, shell/filesystem surface, hidden retry loop, or hidden observe/action loop. `xdotool`, `xrandr`, FFmpeg, portals, PipeWire, and libei are backend implementation details, not MCP tools.

See [`SPEC.md`](./SPEC.md) for the canonical design, [`PLAN.md`](./PLAN.md) for delivery phases, and [`docs/WAYLAND.md`](./docs/WAYLAND.md) for the native Wayland design and operational requirements.

## Requirements

- Linux
- Node.js 22+
- pnpm 11.20.0
- X11 **or** a portal-capable Wayland desktop session
- X11 backend: `xrandr`, FFmpeg with `x11grab`, and `xdotool`
- Wayland backend: `xdg-desktop-portal`, PipeWire, libei, GLib/GIO, libpng, and libxkbcommon
- A C compiler + development headers are needed to build the native Wayland helper

Debian/Ubuntu package names are normally:

```sh
sudo apt-get install \
  x11-xserver-utils ffmpeg xdotool \
  build-essential pkg-config xdg-desktop-portal \
  libei-dev libpipewire-0.3-dev libspa-0.2-dev \
  libglib2.0-dev libpng-dev libxkbcommon-dev
```

The installer can install missing X11/Wayland host dependencies when `apt-get` and sudo/root are available.

## Architecture

```text
MCP tool schemas
      |
      v
ComputerBackend interface
      |
      +-- X11Backend
      |     xrandr -> display metadata
      |     ffmpeg/x11grab -> screenshot bytes
      |     xdotool -> input execution
      |
      `-- WaylandBackend
            xdg-desktop-portal RemoteDesktop + ScreenCast
            PipeWire -> authorized monitor frames
            RemoteDesktop.ConnectToEIS -> libei input
```

The observation/action cycle remains explicit:

```text
computer.observe -> ChatGPT reasons -> computer.act -> computer.observe ...
```

`computer.act` executes exactly one typed action. A `drag` is one explicit action composed of button-down, move, and button-up at the backend level; it does not trigger any observation or retry.

## Coordinate spaces

`computer.info` exposes the virtual desktop and each active display. Pointer and capture coordinates can use:

- `desktop`: absolute coordinates in the backend's desktop coordinate system;
- `display`: coordinates relative to one display's top-left corner.

`computer.info.coordinateUnit` makes the unit explicit. X11 uses `physical-pixel`; Wayland uses compositor/libei `logical-pixel`. Each display also reports `pixelWidth`, `pixelHeight`, and `scale` so PipeWire frame dimensions are not confused with logical input coordinates.

Example display-relative click:

```json
{
  "action": {
    "type": "click",
    "button": "left",
    "point": { "space": "display", "displayId": "DP-1", "x": 500, "y": 300 }
  }
}
```

Supported `computer.act` variants:

- `move`
- `click`
- `doubleClick`
- `scroll`
- `drag`
- `mouseDown`
- `mouseUp`
- `key`
- `text`

Scroll convention: positive Y = down, negative Y = up, positive X = right, negative X = left.

## Local development

```sh
pnpm install
pnpm typecheck
pnpm test
pnpm build
pnpm gate
```

Run over stdio:

```sh
cp config.example.json config.local.json
pnpm build
CHATGPT_COMPUTER_MCP_CONFIG="$PWD/config.local.json" node dist/src/stdio.js
```

stdout is reserved for MCP protocol traffic; diagnostics go to stderr.

## Configuration

`config.example.json` is safe to track. Copy it to `config.local.json` for machine-specific settings; the local file is gitignored.

```json
{
  "backend": "auto",
  "x11": { "display": null },
  "wayland": { "helperPath": null, "portalTimeoutMs": 120000, "frameTimeoutMs": 5000 },
  "observe": { "enabled": true, "maxImageBytes": 12582912 },
  "action": { "enabled": true, "maxTextBytes": 65536, "maxScrollSteps": 100 },
  "logLevel": "info"
}
```

`backend: "auto"` selects the native Wayland backend when `XDG_SESSION_TYPE=wayland`/`WAYLAND_DISPLAY` is present, otherwise X11 when `DISPLAY` is available. Set `backend: "x11"` only when intentionally using X11/XWayland; set `backend: "wayland"` to require the portal/PipeWire/libei path.

## Install + Secure MCP Tunnel

The included installer builds/tests the project, prepares a local config, checks X11/Wayland dependencies and builds the native Wayland helper, installs the current official OpenAI `tunnel-client` when necessary, creates a tunnel profile, runs diagnostics, and installs a persistent systemd user service.

Before running it, create:

1. an OpenAI-hosted MCP tunnel in Platform tunnel settings;
2. a runtime API key allowed to use that tunnel.

Then:

```sh
./install.sh
```

Non-interactive use:

```sh
export CONTROL_PLANE_TUNNEL_ID='tunnel_0123456789abcdef0123456789abcdef'
export CONTROL_PLANE_API_KEY='sk-...'
./install.sh --yes
```

Do not put a real API key directly on a shell command line or in a tracked file. The installer stores credentials under `.secrets/`, which is gitignored.

Operational commands:

```sh
./scripts/status.sh
./scripts/doctor.sh
./scripts/uninstall.sh
```

Detailed tunnel and ChatGPT setup: [`docs/CHATGPT.md`](./docs/CHATGPT.md).

## ChatGPT plugin/app

Use the app name **ChatGPT-Computer-MCP**.

Current ChatGPT flow:

1. Enable **Developer mode** in ChatGPT settings.
2. Open the ChatGPT Plugins page.
3. Create a developer-mode app.
4. Choose **Tunnel** under Connection.
5. Select/paste the tunnel configured for this machine.
6. Enable **ChatGPT-Computer-MCP** for the conversation.
7. First call `computer.info`, then `computer.observe`.

Because `computer.act` can click/type into real applications, review your ChatGPT confirmation settings and keep the MCP authority intentionally scoped.

## X11 limitations

- X11 input is implemented with `xdotool` internally; it is not part of the MCP API.
- Text input follows X11/XKB/`xdotool` behavior and therefore depends on the active desktop keyboard environment.
- Screenshot capture requires an accessible X display and FFmpeg `x11grab` support.
- Desktop permissions and X authorization still apply when run from a systemd user service.

## Wayland

Native Wayland support is implemented through compositor-approved interfaces:

- `org.freedesktop.portal.RemoteDesktop` for a user-authorized remote-desktop session;
- `org.freedesktop.portal.ScreenCast` for selected monitor streams;
- `ScreenCast.OpenPipeWireRemote` + PipeWire for frame capture;
- `RemoteDesktop.ConnectToEIS` + libei for pointer and keyboard input;
- portal/libei `mapping_id` regions for input/capture coordinate alignment.

The first Wayland operation may display a desktop portal chooser/consent dialog. The authorization lives for the MCP process/session and is closed when the MCP closes. If permission is revoked, the backend fails rather than bypassing compositor policy.

See [`docs/WAYLAND.md`](./docs/WAYLAND.md) for details and current limitations.

## Security / release hygiene

The repository ignores local configuration, secrets, tunnel state, generated service artifacts, screenshots, logs, build output, and activation bundles. `scripts/secret-check.sh` scans repository-controlled files for forbidden local artifacts and common credential patterns without printing secret values.

The GitHub Actions gate runs install, typecheck, tests, build, and the same full gate on Node.js 22 with pnpm 11.20.0.

## License

MIT — Copyright (c) 2026 Alex.