# Native Wayland backend

## Status

Implemented in v0.2.0. The Wayland backend does not use `xdotool`, XWayland, `ydotool`, or privileged `/dev/uinput` injection. It uses the compositor-approved portal + PipeWire + libei path.

## Architecture

```text
ComputerBackend
      |
      v
WaylandBackend
      |
      +-- xdg-desktop-portal RemoteDesktop
      |       session + pointer/keyboard authorization
      |
      +-- xdg-desktop-portal ScreenCast
      |       selected monitor streams + mapping metadata
      |
      +-- OpenPipeWireRemote -> PipeWire
      |       authorized visual frames
      |
      `-- RemoteDesktop.ConnectToEIS -> libei
              pointer + keyboard event injection
```

A small native helper implements the portal, PipeWire, libei, XKB, and PNG plumbing. The TypeScript MCP layer still exposes only `computer.info`, `computer.observe`, and `computer.act`.

## Session sequence

The helper keeps one portal session for the lifetime of the MCP process:

1. verify RemoteDesktop exposes pointer + keyboard devices;
2. verify ScreenCast exposes monitor sources and cursor-metadata mode;
3. create a RemoteDesktop session;
4. request pointer and keyboard devices with `SelectDevices`;
5. select monitor sources with `ScreenCast.SelectSources`;
6. call `RemoteDesktop.Start` and retain the selected stream metadata;
7. call `ScreenCast.OpenPipeWireRemote` and connect the selected PipeWire streams;
8. call `RemoteDesktop.ConnectToEIS` and initialize a libei sender;
9. match PipeWire streams to libei absolute regions with `mapping_id`;
10. keep that authority until the portal revokes/closes it or the MCP process exits.

The first operation that initializes the backend can therefore trigger a compositor portal dialog. The MCP does not try to suppress or bypass it.

## Coordinates and scaling

Wayland `computer.info` reports `coordinateUnit: "logical-pixel"`. Display `x`, `y`, `width`, and `height` describe the logical compositor/libei coordinate space used for input and region selection.

Each display additionally reports:

- `pixelWidth` / `pixelHeight`: actual PipeWire frame dimensions;
- `scale`: frame-pixel width divided by logical width.

The implementation does not assume logical coordinates equal PipeWire pixels. `mapping_id` joins each selected stream to the matching libei region; that libei region is the authority for input geometry when available. Screenshots are sampled/composited from the selected streams into the requested logical region and return their actual encoded `pixelWidth` / `pixelHeight`.

## Screenshot behavior

`computer.observe` reads authorized PipeWire monitor frames and returns exactly one PNG. It can capture:

- the selected virtual desktop;
- one selected display;
- a desktop-relative region;
- a display-relative region.

Cursor inclusion remains caller-controlled. The backend requests cursor metadata and composites it only when `includePointer` is true. It does not create an autonomous capture or analysis loop.

## Input behavior

`computer.act` translates the existing action union into libei events:

- absolute pointer movement;
- button down/up/click/double-click;
- discrete scroll;
- drag;
- keyboard chords;
- literal text.

For libei versions that expose the text capability, literal UTF-8 text uses it. Otherwise the helper uses the current libei-provided XKB keymap as a fallback and returns a clear error for characters that cannot be represented.

After `ConnectToEIS`, input goes through libei. The implementation does not mix that EIS connection with the legacy portal `Notify*` input methods.

## Consent and lifetime

Portal prompts, selected monitors, revoked sessions, locked screens, and compositor policy are real backend state. A denied/cancelled/revoked session produces a backend error. Closing the MCP closes the native helper and portal session.

`wayland.portalTimeoutMs` bounds user/portal requests; `wayland.frameTimeoutMs` bounds PipeWire/libei readiness and frame waits.

## Build/runtime requirements

Debian/Ubuntu development packages:

```sh
sudo apt-get install \
  build-essential pkg-config xdg-desktop-portal \
  libei-dev libpipewire-0.3-dev libspa-0.2-dev \
  libglib2.0-dev libpng-dev libxkbcommon-dev
```

`pnpm build` compiles `native/wayland-helper.c` to `dist/native/chatgpt-computer-wayland-helper`. `scripts/doctor.sh` checks portal reachability and native runtime dependencies when running in a Wayland session.

## Current scope

The v0.2 backend intentionally selects **monitor** sources. Window-only ScreenCast sources are not part of the public `computer.observe` target model. Desktop environments can differ in portal UI and policy, so a real Wayland-session smoke test is still required for each target compositor/distribution even though the helper is compiled and unit/integration-tested in CI.