Input Simulation
xa11y can drive a target application in two distinct ways:
- Accessibility actions are
locator.press(),locator.type_text(),locator.toggle(), and the rest. These call the platform accessibility API directly. They are deterministic and work even when the target window is not focused, which makes them the preferred way to drive a UI. - Input simulation is
InputSim, which generates OS-level pointer and keyboard events at the system event layer (CGEvent on macOS, SendInput on Windows, XTest on X11).
Prefer accessibility actions wherever possible. Reach for input simulation only for gestures that have no accessibility equivalent: drag-and-drop, scroll wheels, global shortcuts, or tests that must exercise the event loop itself.
The mechanisms are not bridged. A failed accessibility action never falls back to input simulation, and input simulation never inspects the accessibility tree for you. If you want to click an element with synthesised input, you compute its bounds via the a11y API and hand a point or Element to InputSim.
When to use which
Section titled “When to use which”| Scenario | Mechanism |
|---|---|
| Press a button, toggle a checkbox, expand a menu | Accessibility action (locator.press(), locator.toggle(), locator.expand()) |
| Type text into a focused field with IME support | Accessibility action (locator.type_text()) |
| Drag-and-drop between two elements | Input simulation (mouse.drag) |
| Scroll a list with the wheel | Input simulation (mouse.scroll) |
| Global shortcut without a focused element | Input simulation (keyboard.chord) |
| Click at a fixed pixel coordinate | Input simulation (mouse.click((x, y))) |
Permissions
Section titled “Permissions”| Platform | Requirement |
|---|---|
| macOS | Accessibility + Input Monitoring for the terminal/IDE running your code |
| Windows | None for normal user sessions. Input to an elevated window needs an elevated caller |
| Linux (X11) | XTest extension enabled on the X server. Zero setup. |
| Linux (Wayland / headless) | Write access to /dev/uinput: membership in the input group (sudo usermod -aG input $USER then re-login), or a udev rule that grants it. xa11y opens /dev/uinput and registers a virtual evdev keyboard+pointer, the same mechanism used by xdotool --using-uinput, ydotool, wtype, Steam Input, and Wine. The privilege grants global input read/write access, comparable to macOS Input Monitoring. |
On macOS, input simulation usually also requires the target window to be foregrounded. Activate the window explicitly before synthesising input.
xa11y picks the Linux backend at runtime. Pure X11 sessions use XTest;
Wayland and mixed Wayland/XWayland sessions use uinput. Headless Linux also
uses uinput. Set XA11Y_LINUX_INPUT_BACKEND=x11|wayland to override that
choice. There is no compile-time feature flag, and both backends are always
built into xa11y-linux. The uinput path is compositor-agnostic, so it works
on every Wayland desktop (GNOME, KDE Plasma, sway, Hyprland, Cosmic, weston)
and on headless Linux servers. Errors from a missing input group surface as
PermissionDenied with a message saying what to do about it. A missing
uinput kernel module surfaces as Unsupported.
Basic usage
Section titled “Basic usage”use xa11y::*;
fn main() -> Result<()> { let sim = input_sim()?;
// Click at a screen coordinate sim.mouse().click(Point::new(400, 300))?;
// Click at the centre of an element's current bounds let app = App::by_name("Finder", std::time::Duration::from_secs(5))?; let button = app.locator("button[name='New Folder']").element()?; sim.mouse().click(&button)?;
// Type text into whatever has keyboard focus sim.keyboard().type_text("Hello, world!")?;
// Keyboard shortcut — Cmd/Ctrl+S sim.keyboard().chord(Key::Char('s'), &[Key::Meta])?;
Ok(())}import xa11y
sim = xa11y.input_sim()
# Click at a screen coordinatesim.click((400, 300))
# Click at the centre of an element's current boundsapp = xa11y.App.by_name("Finder")button = app.locator("button[name='New Folder']").element()sim.click(button)
# Type text into whatever has keyboard focussim.type_text("Hello, world!")
# Keyboard shortcut — Cmd/Ctrl+Ssim.chord("s", ["Meta"])import { inputSim, App } from '@crowecawcaw/xa11y';
const sim = inputSim();
// Click at a screen coordinateawait sim.click([400, 300]);
// Click at the centre of an element's current boundsconst app = await App.byName('Finder');const button = await app.locator("button[name='New Folder']").element();await sim.click(button);
// Type text into whatever has keyboard focusawait sim.typeText('Hello, world!');
// Keyboard shortcut — Cmd/Ctrl+Sawait sim.chord('s', ['Meta']);Targets
Section titled “Targets”Every pointer method accepts either a screen-space point or an Element:
- Point is an absolute desktop coordinate. Windows uses physical pixels. macOS uses points. Linux uses logical pixels, or physical pixels where the HiDPI scale can’t be read reliably (see platform details). Origin is the top-left of the primary display, and negative values are valid on multi-monitor setups.
- Element uses the centre of the element’s
boundsat the time of the call. Rectangles are half-open. Right and bottom anchors use the last coordinate inside the bounds rather than the exclusive far edge.InputSimdoes not re-read the accessibility tree for you, so re-fetch via a locator first if the UI may have moved.
Check a refused pointer target
Section titled “Check a refused pointer target”If a pointer action returns InvalidActionData (InvalidActionDataError in
Python and JS, invalid_action_data in MCP), inspect its geometry diagnosis.
The built-in Windows, macOS, and X11 backends check that the requested point
lies inside an active display. This applies to raw points and element anchors;
gaps between monitors are outside every display. For Wayland and headless
uinput, raw points are checked against the compositor’s outputs, or the
XA11Y_SCREEN_WIDTH / XA11Y_SCREEN_HEIGHT override. Element targets remain
unsupported there.
On Linux the diagnosis also says where the display rectangles came from, in
display_source (displaySource in JS). An X server without RandR 1.5 has no
monitor list, so xa11y checks the root window’s geometry there and records
that rule. An X server that reports no active monitors at all fails with a
Platform error, since no point could pass.
The initial check happens before input is posted, including held modifiers,
so its failure reports delivery: not_delivered. Both drag endpoints are
checked before the gesture starts. Display query failures retain their
original error kind instead of becoming a target miss.
Read error.pointer_target in Python or error.pointerTarget in JS for the
requested point, original element bounds and anchor, and the point and display
rectangles in the checked native coordinate space. The display snapshot is
limited to 32 rectangles and reports how many were omitted; validation uses
the complete set. In Rust, read pointer_target from the Diagnosis returned
by Error::diagnosis; MCP includes it under
diagnosis.pointer_target. The CLI prints the geometry with the failure.
For raw points, choose a coordinate on a display. For element targets, move the window onto a display and fetch a fresh element before retrying. Display membership alone does not establish that a target is visible in its viewport or still at those coordinates when input arrives. Another window can cover it.
Moves while a button is held through the same input provider can cross
display gaps. Releases also remain available, so a geometry check cannot
strand a captured gesture. A click count of zero remains a no-op. Custom Rust
input providers retain their existing behavior unless they override
InputProvider::validate_pointer_target.
Clicking something that is still moving
Section titled “Clicking something that is still moving”An element sliding in, or a dialog still settling after a layout change, has
bounds that are already stale by the time the click is sent. Fetch it with the
locator’s wait_stable() (waitStable() in JS) instead of element(). It
returns once visible bounds are observed unchanged for at least 750 ms, so
InputSim uses the last observed stable position. Native snapshots can lag
the screen, and a pause does not guarantee that motion will not resume:
panel_button = app.locator('button[name="Save"]').wait_stable(timeout=5)sim.click(panel_button)The CLI spelling is xa11y find ... -o center --stable. See the
locator reference for the timeout diagnosis.
On a Linux Wayland session, Element targets are refused with Unsupported. Wayland clients don’t know their global position, so the bounds AT-SPI reports are not screen coordinates. Use a semantic action such as press or focus there, or pass a raw point. See platform details.
To click a location selected from a screenshot, convert the image pixel with
that capture’s image_to_desktop / imageToDesktop method first. On Windows
this also checks that the display layout has not changed since capture.
In Rust, additional anchors are available via ClickOptions.anchor (Anchor::TopLeft, Anchor::Offset { dx, dy }, etc.).
let m = sim.mouse();
m.click((100, 100))?; // left clickm.double_click(&element)?; // double clickm.right_click((100, 100))?; // right clickm.move_to((200, 200))?; // move without clickingm.drag((50, 50), (200, 200))?; // left-drag with 150ms durationm.scroll((400, 300), ScrollDelta::vertical(-3))?; // 3 ticks down
// Shift-click with explicit optionsm.click_with( ClickTarget::from(&element), ClickOptions::new() .held([Key::Shift]) .anchor(Anchor::TopLeft),)?;
// Slow drag for apps that need more framesm.drag_with( (50, 50), (200, 200), DragOptions::new().duration(Duration::from_millis(500)),)?;sim.click((100, 100)) # left clicksim.double_click(element) # double clicksim.right_click((100, 100)) # right clicksim.move_to((200, 200)) # move without clickingsim.drag((50, 50), (200, 200)) # left-dragsim.scroll((400, 300), dy=-3) # 3 ticks downsim.scroll((400, 300), dx=2, dy=0) # 2 ticks rightawait sim.click([100, 100]); // left clickawait sim.doubleClick(element); // double clickawait sim.rightClick([100, 100]); // right clickawait sim.moveTo([200, 200]); // move without clickingawait sim.drag([50, 50], [200, 200]); // left-dragawait sim.scroll([400, 300], 0, -3); // dx=0, dy=-3 (3 ticks down)Scroll units are platform “ticks”, typically one notch of a physical scroll wheel. Positive dx scrolls right. Positive dy scrolls content down, which moves the viewport up.
Keyboard
Section titled “Keyboard”Modifier keys (Shift, Ctrl, Alt, Meta) are ordinary keys, so pass them alongside character keys. Meta is the platform’s “command” modifier, spelled Cmd on macOS, Win on Windows, and Super on Linux.
On Windows, letters and digits are positional virtual keys (VK_A, VK_1), as keyboard shortcuts expect. Any other character key is looked up on the keyboard layout of the foreground window. If the layout needs a modifier to type the character (Shift for / on a German layout), the call returns Unsupported. xa11y does not add the modifier for you.
let k = sim.keyboard();
// Tap a single keyk.press(Key::Enter)?;k.press(Key::Char('a'))?;
// Shortcuts — modifiers are just keys held during the tapk.chord(Key::Char('a'), &[Key::Meta])?; // Cmd/Ctrl+Ak.chord(Key::Char('z'), &[Key::Meta, Key::Shift])?; // Cmd/Ctrl+Shift+Z
// Uppercase letters — hold Shift explicitlyk.chord(Key::Char('a'), &[Key::Shift])?; // 'A'
// Hold and release manuallyk.down(Key::Shift)?;k.press(Key::Char('a'))?;k.press(Key::Char('b'))?;k.up(Key::Shift)?;
// Type literal text (handles case and IME for you)k.type_text("Hello, world!")?;Key::Char rejects ASCII uppercase letters at the API boundary to prevent the common bug where chord(Key::Char('K'), &[Key::Meta]) is read as “Cmd+K” but would mean “Cmd+Shift+K” under auto-shift semantics. For arbitrary text, call type_text.
# Tap a single keysim.press("Enter")sim.press("a")
# Shortcuts — modifiers are just keys listed in `held`sim.chord("a", ["Meta"]) # Cmd/Ctrl+Asim.chord("z", ["Meta", "Shift"]) # Cmd/Ctrl+Shift+Z
# Type literal text (handles case and IME for you)sim.type_text("Hello, world!")Key names: single printable characters are literal ("a", "7", ";"); named keys use their Pascal name ("Enter", "Escape", "ArrowUp", "F5"); modifiers are "Shift", "Ctrl", "Alt", "Meta". Aliases: "Control", "Option", "Cmd"/"Command", "Esc", "Up"/"Down"/"Left"/"Right".
Uppercase letters are rejected by press/chord — pass "a" with "Shift" held, or use type_text for arbitrary text.
// Tap a single keyawait sim.press('Enter');await sim.press('a');
// Shortcuts — modifiers are just keys listed in `held`await sim.chord('a', ['Meta']); // Cmd/Ctrl+Aawait sim.chord('z', ['Meta', 'Shift']); // Cmd/Ctrl+Shift+Z
// Type literal text (handles case and IME for you)await sim.typeText('Hello, world!');Key names match the Python binding: literal characters, Pascal-cased named keys ('Enter', 'ArrowUp', 'F5'), modifiers 'Shift' / 'Ctrl' / 'Alt' / 'Meta'.
On every platform, type_text types each line break as a press of the Enter key. A "\r\n" pair is one line break, and a lone "\r" or "\n" is one line break each. In a single-line field, Enter can trigger the default button, as it does when a person types it.
Example: drag-and-drop
Section titled “Example: drag-and-drop”Drag-and-drop has no accessibility equivalent on most platforms. Resolve both endpoints via the a11y tree first, then synthesise the drag.
On macOS a drag takes about 0.7 seconds longer than its duration. The pointer holds still for 200 ms after the button goes down, so the drag source can start its session. It holds still again for 500 ms at the drop point before release, so the target registers the drop.
let app = App::by_name("Finder", std::time::Duration::from_secs(5))?;let source = app.locator("list_item[name='notes.txt']").element()?;let target = app.locator("list_item[name='Archive']").element()?;
let sim = input_sim()?;sim.mouse().drag(&source, &target)?;app = xa11y.App.by_name("Finder")source = app.locator("list_item[name='notes.txt']").element()target = app.locator("list_item[name='Archive']").element()
sim = xa11y.input_sim()sim.drag(source, target)const app = await App.byName('Finder');const source = await app.locator("list_item[name='notes.txt']").element();const target = await app.locator("list_item[name='Archive']").element();
const sim = inputSim();await sim.drag(source, target);Error handling
Section titled “Error handling”Input simulation raises five of xa11y’s error variants:
PermissionDeniedwhen the OS denied the synthesis permission. On Linux this is the missinginputgroup; macOS cannot report a missing Input Monitoring grant, so the events are dropped silently instead. On Windows it means the target window belongs to a process at a higher integrity level, such as an elevated app. UIPI drops such input and still reports it as delivered. xa11y refuses it up front instead, naming both integrity levels. When the target’s level cannot be read, as with another user’s process, the input is sent and UIPI may still drop it. A target that holdsuiAccessis not checked either, and UIPI can drop input to it from a same-level sender that lacksuiAccess. Key and button releases are never refused, so a held key can always be released. A drag checks both of its ends before it presses the button.Unsupportedwhen the platform has no backend for the operation, when the Linuxuinputkernel module is unavailable, when aCharkey is on no key of the active Windows layout, or when that layout needs a modifier to produce it.NoElementBoundswhen the targetElementhas no bounds. Re-fetch it, or anchor on a point.InvalidActionDatafor an uppercaseCharkey, a malformed key name, or a bad target tuple.Platformfor a raw OS failure such as an FFI return code.
Arguments are parsed before the first OS call, so a rejected argument can never leave a half-delivered gesture behind with a key pressed and never released.
Note that NoElementBounds surfaces as a plain ValueError in Python rather
than as an xa11y exception class. See Errors for the
full mapping.