How xa11y Is Tested

xa11y drives the real accessibility tree of a running application. There is no way to mock that meaningfully, so the library is verified the same way it is used. Real GUI apps built on real toolkits are launched, and the tests assert against the trees their platform accessibility APIs actually expose. This page maps that test suite: what runs, and against what.

If you are deciding whether to depend on xa11y, this is the page that answers “how do I know it works?”

The suite at a glance

The library carries 700+ automated tests spanning Rust, Python, JavaScript, and the CLI. Roughly:

Suite	Approx. tests	Location	What it covers
Rust unit	~60	xa11y/tests/unit_test.rs	Selector engine, locators, roles, state sets, error mapping, serialization (against an in-memory mock provider)
Rust integration	~160	xa11y/tests/integ/	Live tree reads, action dispatch, events, screenshots against a running app, and the OS shell surfaces
Python binding unit	~240	xa11y-python/tests/	The PyO3 surface: types, actions, exceptions, GIL release, subscriptions
Python integration	~100	tests/suites/python/	Per-app compat, actions, events, errors, input simulation, screenshots
CLI integration	~80	tests/suites/cli/	The xa11y CLI end-to-end against running apps, plus the MCP server and its event subscriptions over every CLI launcher
JavaScript unit	~115	xa11y-js/__test__/unit/	The napi-rs binding surface
JavaScript integration	~40	tests/suites/js/	Per-app compat, actions, input simulation, screenshots
MCP client interop	~30	tests/mcp_client/	xa11y mcp driven by the official MCP Python SDK, in both protocol eras

Counts are approximate and grow over time, so read the figures above as a rough shape. The authoritative coverage index, recording which app is tested in which language for which feature, lives in tests/matrix.yaml.

Tested against real toolkits, not stubs

The integration suites run against a fleet of small test apps, each built on a different GUI toolkit, and each launched as a real process whose accessibility tree xa11y reads through the platform API. This is the part that matters: a passing test means xa11y handled a real AT-SPI2, AXUIElement, or UI Automation tree, not a fixture shaped to match the code.

Test app	Toolkit	Linux	macOS	Windows	Features covered
accesskit	Rust + winit (AccessKit)	✓	✓ †	✓	compat, actions, events, errors, screenshot, windows
qt	PySide6 (Qt 6)	✓	✓	✓	compat, actions, events, errors, windows
gtk	GTK4 (PyGObject)	✓	○	○	compat, actions, events, errors, windows
cocoa	Swift / AppKit	○	✓	○	compat, actions, events, errors, windows
tauri	Rust + WebView	✓	✓	✓	compat, actions, events, errors, input_sim, screenshot, windows
electron	Chromium / Node	✓	○	○	compat, actions, errors, input_sim, screenshot, windows
egui	Rust immediate-mode (eframe)	✓	✓	✓	compat, actions, events, errors, windows
winforms	.NET Windows Forms	○	○	✓	compat, actions, events, errors, windows
wpf	.NET WPF	○	○	✓	compat, actions, events, errors, windows

A hollow circle means the combination is not run in CI. † AccessKit on macOS runs locally through ./scripts/run_integ_tests_macos.sh rather than in CI, because the Rust integ suite needs an Accessibility (TCC) grant that hosted macOS runners cannot hold against a cargo-hashed binary path.

Reading the feature column: compat is tree structure, roles, and widget discovery; actions is press, focus, toggle, type, and expand/collapse; events is subscription and state-change detection; errors is invalid selectors, selector misses, wait timeouts, and unsupported actions; input_sim and screenshot are the platform input and capture surfaces; windows is window discovery, state inspection, and the window verbs (activate, minimize, maximize, restore, close, and move/resize). Not every language suite covers every feature listed for an app. The per-language breakdown is in tests/matrix.yaml, which is the authoritative index and the file this table summarises.

Which suite drives which app varies. Qt, GTK, Cocoa, Tauri, WinForms, and WPF are driven by the Python integration suite, with JS and CLI suites running the same scenarios alongside it. The AccessKit row is covered primarily by the Rust integration suite in xa11y/tests/integ/ rather than by the per-app compatibility suites, and its screenshot coverage comes from there; the JS suite also targets it for compat and actions. GTK is also driven by the Python suite. Electron is driven by the JS suite, which also covers AccessibilityNotEnabled detection by launching the app without --force-renderer-accessibility and asserting that xa11y says so rather than reporting an empty tree.

Tauri is the app carrying the input-simulation suite, so its three rows are what give each platform’s input backend an end-to-end test. Each of those platform claims is now checked: tests/matrix_check.py reads the CI matrix out of the workflow and fails the build when a declared platform has no cell running it.

Across those rows, xa11y is exercised against all three platform accessibility backends (AT-SPI2 on Linux, AXUIElement on macOS, UI Automation on Windows) and against toolkits that report their trees in meaningfully different ways: retained-mode native widgets, immediate-mode GUIs, two distinct web renderers, and two first-party Microsoft UIA providers rather than third-party bridges, by way of Windows Forms and WPF. The toolkit-specific quirks this surfaces are documented in Accessibility Quirks.

Shell surfaces are tested against a desktop that has them

Shell surfaces (the taskbar, a menu bar, a desktop panel) are found by three hand-written classifiers, one per platform, each keyed on a platform constant: a window class name on Windows, an AX attribute name on macOS, an AT-SPI window-type attribute on Linux. A classifier that stops matching returns an empty list, which reads exactly like a desktop that owns no shell UI. A test that skips when nothing is listed passes just as happily when the feature is broken.

So xa11y/tests/integ/shell.rs states what each platform’s desktop must vend and fails when it is missing. Windows must have a Shell_TrayWnd taskbar, macOS the menu bar of whichever application is in front, Linux a window-type:dock frame. The Linux one needs a fixture: a bare Xvfb display runs no desktop environment, so the harness launches test-apps/panel/panel.py, a GTK 3 dock window, and the test asserts it can reach the panel’s own button through the surface root.

One suite, every binding

The per-app integration tests aren’t duplicated per language. Python, JavaScript, and the CLI run the same feature suites (compat, actions, events, errors, input simulation, screenshots) against the same running app, driven by a shared harness (tests/harness/launch.py). Binding parity is proven by every binding passing identical scenarios against an identical target, rather than asserted by hand.

Checked against a real MCP client

The MCP server has two test suites, and the split is deliberate. tests/suites/cli/test_mcp.py speaks raw JSON-RPC and asserts the shapes this project believes the specification asks for. tests/mcp_client/ drives the same server with the official MCP Python SDK and asserts that a real client agrees.

Those are different questions. The SDK validates responses against wire models pinned to each protocol revision, and those models mark as required several fields the prose describes as MUST. The server returned tools/list and server/discover without their mandatory caching hints, and the raw-JSON suite passed the whole time, because it was checking one reading of the spec against itself. The SDK rejected the response outright on first contact.

Each suite covers what the other cannot. The SDK exercises what a real client actually sends, in both protocol eras. The raw-JSON suite exercises what the SDK refuses to send: malformed lines, unsupported protocol versions, notifications out of order.

The interop job runs on Linux only. The server is hand-rolled JSON-RPC over standard streams with no platform-specific code in it, so the answer does not vary by operating system; the accessibility layer beneath it is what does, and the integration matrix covers that.

One CLI, three launchers

The xa11y command comes with all three packages, and all three run the same Rust code: cargo install xa11y produces the binary, pip install xa11y installs a console script into it, and the npm package installs a Node bin that does the same. Nothing checked that they agreed until the MCP suite (tests/suites/cli/test_mcp.py), which runs every one of its tests once per launcher.

That parametrization found real drift on its first run: the Python wrapper mapped every failure to exit code 1, losing the documented 2 for usage errors, and prefixed those messages twice. The exit-code contract now lives in one Rust function that all three launchers call.

The integration cells set XA11Y_REQUIRE_ALL_CLI, which turns a missing launcher into a failure. A suite that quietly narrowed itself to whichever launcher happened to be built would look exactly like one that tested all three.

The CI matrix

Every push runs the suites across operating systems and toolkits in GitHub Actions (.github/workflows/ci.yml). The integ job alone is a matrix of fifteen OS × app cells (e.g. ubuntu × {accesskit, qt, gtk, tauri, electron, egui}, macos × {cocoa, tauri, qt, egui}, windows × {tauri, qt, egui, winforms, wpf}), each standing up a headless display, accessibility bus, and the app under test before running the Python, JS, and CLI suites against it. Alongside it run the Rust unit and integration jobs per OS, two wire-level input jobs (Linux uinput read back through libevdev, Windows SendInput read back through low-level hooks), bindings builds and typechecks, cross-compile checks, license auditing, docs build with link checking, and the fuzzers below.

Fuzzing and robustness

The selector engine and tree operations are continuously fuzzed with libFuzzer targets in xa11y/fuzz/ (run on every CI push), and a separate live provider fuzzer (xa11y-fuzz/) stress-tests the provider interface against a running app with randomized action sequences.

Correctness is a stated discipline

The breadth above is backed by a small set of firm engineering tenets that every new piece of provider or binding code is held to: no silent fallbacks, action fidelity (an advertised action invokes the real platform action, never a substitute), errors that carry their own diagnosis, and blocking calls that release the host runtime’s lock. These are written out in full, with anti-patterns, in Architecture & Design and restated verbatim in the repo’s CLAUDE.md so human reviewers and automated ones apply them identically.

Missing a toolkit you care about?

The test-app fleet is deliberately chosen so that each app covers an accessibility surface the others don’t. If you work with a common UI framework that’s meaningfully different from everything in the table above, that’s a coverage gap worth closing. A different accessibility backend, a renderer family not yet represented, or a platform combination we don’t exercise all qualify. Please open an issue describing the framework and how its accessibility tree differs. New test-app coverage is very welcome.