Serve xa11y over MCP

xa11y mcp serves the accessibility tree, actions, input simulation, and screenshots as Model Context Protocol tools over stdio. A client launches it as a subprocess, so an agent can drive a desktop application without you writing any glue.

For the tool list, argument names, and result shapes, see the CLI reference.

Install the command

Any of the three packages provides it. Pick whichever matches the rest of your setup. The server is the same either way.

Rust

cargo install xa11y

Python

pip install xa11y

JavaScript

npm install -g @crowecawcaw/xa11y

Check it starts:

xa11y mcp

Run in a terminal it prints a one-line note to stderr and waits. That is correct: it is waiting for JSON-RPC on stdin. Press Ctrl-D to exit.

Register it with a client

Most clients read a JSON config naming the command to launch.

{
  "mcpServers": {
    "xa11y": {
      "command": "xa11y",
      "args": ["mcp"]
    }
  }
}

If the command is not on PATH, give an absolute path. The Python and Node packages install the same xa11y command into their environment’s bin directory, so a virtualenv or a local npm install needs the path to that copy rather than a global one.

Restart the client and confirm it lists twelve tools, starting with apps.

Grant accessibility permission first

The server can only see what the platform lets it see, and a client that launches it as a subprocess inherits the client’s own permissions. Grant them to the application that launches the server, not to xa11y itself.

Platform	Requirement
macOS	Accessibility permission for the client application
Windows	None for normal user sessions
Linux	An AT-SPI bus (at-spi-bus-launcher running)

See Install xa11y for the per-platform steps. Without the grant, the first tool call fails with a permission_denied result rather than an empty tree.

Drive an application

A typical sequence: find the application, read its structure, act on it.

1. apps returns running applications and their process ids.
2. tree with app or pid returns the structure, 12 levels deep by default.
3. find with a selector returns matching elements, each with bounds and a precomputed center.
4. action performs an accessibility action on a match.

To reach OS chrome instead of an application, call shell for the surfaces that are present and pass the kind you want as the shell argument to tree, find, or action. The listing is live, so a flyout shows up only while it is open, and a hidden Windows tray icon needs the taskbar’s “Show Hidden Icons” button pressed first. Kinds and the rest of the model are in Shell surfaces.

Prefer action over click. It calls the application’s own accessibility action, so it works regardless of window position, focus, or whether the element is on screen. Reach for click, key, and type only where no accessibility action exists — see Simulate input.

Two things about action are worth knowing before the first call. Its selector has to match one element: button in an application with eight of them is refused, with the eight listed, rather than applied to whichever came first. And its ok: true means the application accepted the call, not that the interface changed, so read the tree again when the effect matters.

Annotate a screenshot

A plain screenshot shows the model what the application looks like without saying which tree node is which pixel. When the tree is present but uninformative, annotate draws that correlation into the image.

Pass an array of selectors and a target:

{
  "pid": 4242,
  "annotate": ["button", "text_field"]
}

Each selector is one group, and every element it matches gets an outlined box with a short tag. The result keeps the PNG and gains a legend:

{
  "width": 2560,
  "height": 1440,
  "scale": 2.0,
  "bytes": 184320,
  "legend": [
    {
      "tag": "A1",
      "group": 1,
      "index": 1,
      "selector": "button:nth(1)",
      "role": "button",
      "name": "7",
      "bounds": { "x": 104, "y": 318, "width": 48, "height": 44 },
      "color": [230, 159, 0]
    }
  ],
  "omitted": [],
  "truncated": 0
}

A tag is a letter for the group and a 1-based number within it, so B7 is the seventh match of the second selector. That number is the :nth(n) argument, and selector is built from it already. A model that reads B7 off the image hands legend[i].selector to action or find with the same target, which is the round trip the feature exists for.

The boxes come from the accessibility tree. An application that exposes no tree gets no annotations and the capture comes back plain.

The target scopes the selectors. Cropping stays with x, y, width, and height, so the two compose: capture one window’s region, and annotate the elements inside it.

A target on its own is refused. app, pid, and shell are read only to resolve annotate selectors, so {"pid": 4242} with no annotate used to come back as a full-desktop capture reporting success, and the model that asked to target an application had no way to see that its argument did nothing. It now returns invalid_arguments, naming the key it could not use and offering both fixes: add annotate to box that target’s elements, or drop the target for a plain capture. An empty annotate array is the plain-capture path too, so it is refused on the same terms.

omitted lists everything that matched a selector without reaching the image, each with a reason of no_bounds, zero_area, or outside_capture. The three are described in Annotated screenshots.

Boxes are never clamped to the edge of the capture, because a clamped box claims pixels that belong to something else.

One caveat before you trust a box. The tree does not say which window is in front, so an element hidden behind another window still gets a box drawn over whatever is on screen there. Narrow the group with button[visible] when that matters, since only the caller knows what it meant.

Keep results small

Every result goes into the model’s context. Three arguments control the size:

• tree takes max_depth. Start at 2 or 3 to find the window you want, then walk into it.
• find takes limit, defaulting to 50.
• screenshot takes annotate, which describes at most 100 elements. A selector like * over a large window reaches that cap.

All three report whether they truncated, so a shortened result never reads as a complete one. For annotate, truncated counts the matches it did not reach, and narrowing the selectors is the way to see the rest.

Read a failure instead of retrying blind

A failed tool call comes back as a result with isError: true rather than a protocol error, so the model sees it and can correct itself. The payload carries a kind and, for lookup failures, a diagnosis naming what the search was looking for, what it last observed, and which near-miss elements were present.

A selector that matched nothing lists the candidates that were there, and one that matched too many lists what it matched, which is usually enough to fix the selector on the next call. See Errors and diagnosis.

Watch what an application does

Reading the tree twice tells you what changed, not when. events_start opens a subscription to an application’s accessibility events and returns a handle:

{
  "name": "events_start",
  "arguments": { "app": "Calculator", "kinds": ["focus_changed", "value_changed"] }
}

Events buffer from the moment that call returns, so open the subscription before the action you want to observe, act, then drain:

{ "name": "events_poll", "arguments": { "subscription_id": "sub_1" } }

A poll returns the buffered events oldest first, each with its kind, a sequence, at_ms since the subscription started, and the target element in the shape find returns. It does not block by default, so an empty events means nothing has happened yet. Pass timeout_ms (up to 15000) to wait for the first event when you are expecting a specific one.

Two fields say what the events cannot. dropped counts events lost to the buffer filling up since the previous poll: poll more often, narrow kinds, or accept the gap. live: false means the platform dropped the subscription, so no further event can arrive. It does not track whether the application is still running: no platform reports an exit that way, so an application that quits goes on reporting live: true with an empty events list. Call apps to check.

Call events_stop when you are done. A subscription nobody polls for five minutes is reclaimed, and its handle then comes back as a subscription_expired failure.

What is not there yet

Revision 2026-07-28 has subscriptions/listen, where the server pushes notifications instead of the model polling for them. That is the better shape for events, and the handle trio above is what works on every revision this server speaks, including the ones that have no such mechanism.