Skip to content
xa11y

Input Simulation

xa11y can drive a target application in two distinct ways:

  • Accessibility actions are locator.press(), locator.type_text(), locator.toggle(), and the rest. These call the platform accessibility API directly. They are deterministic and work even when the target window is not focused, which makes them the preferred way to drive a UI.
  • Input simulation is InputSim, which generates OS-level pointer and keyboard events at the system event layer (CGEvent on macOS, SendInput on Windows, XTest on X11).

Prefer accessibility actions wherever possible. Reach for input simulation only for gestures that have no accessibility equivalent: drag-and-drop, scroll wheels, global shortcuts, or tests that must exercise the event loop itself.

The mechanisms are not bridged. A failed accessibility action never falls back to input simulation, and input simulation never inspects the accessibility tree for you. If you want to click an element with synthesised input, you compute its bounds via the a11y API and hand a point or Element to InputSim.

When to use which

Section titled “When to use which”
Scenario Mechanism
Press a button, toggle a checkbox, expand a menu Accessibility action (locator.press(), locator.toggle(), locator.expand())
Type text into a focused field with IME support Accessibility action (locator.type_text())
Drag-and-drop between two elements Input simulation (mouse.drag)
Scroll a list with the wheel Input simulation (mouse.scroll)
Global shortcut without a focused element Input simulation (keyboard.chord)
Click at a fixed pixel coordinate Input simulation (mouse.click((x, y)))

Permissions

Section titled “Permissions”
Platform Requirement
macOS Accessibility + Input Monitoring for the terminal/IDE running your code
Windows None for normal user sessions
Linux (X11) XTest extension enabled on the X server. Zero setup.
Linux (Wayland / headless) Membership in the input group (sudo usermod -aG input $USER then re-login). xa11y opens /dev/uinput and registers a virtual evdev keyboard+pointer, the same mechanism used by xdotool --using-uinput, ydotool, wtype, Steam Input, and Wine. The privilege grants global input read/write access, comparable to macOS Input Monitoring.

On macOS, input simulation usually also requires the target window to be foregrounded. Activate the window explicitly before synthesising input.

xa11y picks the Linux backend at runtime. Pure X11 sessions use XTest; Wayland and mixed Wayland/XWayland sessions use uinput. Headless Linux also uses uinput. Set XA11Y_LINUX_INPUT_BACKEND=x11|wayland to override that choice. There is no compile-time feature flag, and both backends are always built into xa11y-linux. The uinput path is compositor-agnostic, so it works on every Wayland desktop (GNOME, KDE Plasma, sway, Hyprland, Cosmic, weston) and on headless Linux servers. Errors from a missing input group surface as PermissionDenied with a message saying what to do about it. A missing uinput kernel module surfaces as Unsupported.

Basic usage

Section titled “Basic usage”
use xa11y::*;


fn main() -> Result<()> {
    let sim = input_sim()?;


    // Click at a screen coordinate
    sim.mouse().click(Point::new(400, 300))?;


    // Click at the centre of an element's current bounds
    let app = App::by_name("Finder", std::time::Duration::from_secs(5))?;
    let button = app.locator("button[name='New Folder']").element()?;
    sim.mouse().click(&button)?;


    // Type text into whatever has keyboard focus
    sim.keyboard().type_text("Hello, world!")?;


    // Keyboard shortcut — Cmd/Ctrl+S
    sim.keyboard().chord(Key::Char('s'), &[Key::Meta])?;


    Ok(())
}

Targets

Section titled “Targets”

Every pointer method accepts either a screen-space point or an Element:

  • Point is an absolute desktop coordinate. Windows uses physical pixels. macOS uses points. Linux uses logical pixels, or physical pixels where the HiDPI scale can’t be read reliably (see platform details). Origin is the top-left of the primary display, and negative values are valid on multi-monitor setups.
  • Element uses the centre of the element’s bounds at the time of the call. Rectangles are half-open. Right and bottom anchors use the last coordinate inside the bounds rather than the exclusive far edge. InputSim does not re-read the accessibility tree for you, so re-fetch via a locator first if the UI may have moved.

To click a location selected from a screenshot, convert the image pixel with that capture’s image_to_desktop / imageToDesktop method first. On Windows this also checks that the display layout has not changed since capture.

In Rust, additional anchors are available via ClickOptions.anchor (Anchor::TopLeft, Anchor::Offset { dx, dy }, etc.).

Mouse

Section titled “Mouse”
let m = sim.mouse();


m.click((100, 100))?;                   // left click
m.double_click(&element)?;              // double click
m.right_click((100, 100))?;             // right click
m.move_to((200, 200))?;                 // move without clicking
m.drag((50, 50), (200, 200))?;          // left-drag with 150ms duration
m.scroll((400, 300), ScrollDelta::vertical(-3))?;  // 3 ticks down


// Shift-click with explicit options
m.click_with(
    ClickTarget::from(&element),
    ClickOptions::new()
        .held([Key::Shift])
        .anchor(Anchor::TopLeft),
)?;


// Slow drag for apps that need more frames
m.drag_with(
    (50, 50),
    (200, 200),
    DragOptions::new().duration(Duration::from_millis(500)),
)?;

Scroll units are platform “ticks”, typically one notch of a physical scroll wheel. Positive dx scrolls right. Positive dy scrolls content down, which moves the viewport up.

Keyboard

Section titled “Keyboard”

Modifier keys (Shift, Ctrl, Alt, Meta) are ordinary keys, so pass them alongside character keys. Meta is the platform’s “command” modifier, spelled Cmd on macOS, Win on Windows, and Super on Linux.

let k = sim.keyboard();


// Tap a single key
k.press(Key::Enter)?;
k.press(Key::Char('a'))?;


// Shortcuts — modifiers are just keys held during the tap
k.chord(Key::Char('a'), &[Key::Meta])?;              // Cmd/Ctrl+A
k.chord(Key::Char('z'), &[Key::Meta, Key::Shift])?;  // Cmd/Ctrl+Shift+Z


// Uppercase letters — hold Shift explicitly
k.chord(Key::Char('a'), &[Key::Shift])?;             // 'A'


// Hold and release manually
k.down(Key::Shift)?;
k.press(Key::Char('a'))?;
k.press(Key::Char('b'))?;
k.up(Key::Shift)?;


// Type literal text (handles case and IME for you)
k.type_text("Hello, world!")?;

Key::Char rejects ASCII uppercase letters at the API boundary to prevent the common bug where chord(Key::Char('K'), &[Key::Meta]) is read as “Cmd+K” but would mean “Cmd+Shift+K” under auto-shift semantics. For arbitrary text, call type_text.

Example: drag-and-drop

Section titled “Example: drag-and-drop”

Drag-and-drop has no accessibility equivalent on most platforms. Resolve both endpoints via the a11y tree first, then synthesise the drag.

let app = App::by_name("Finder", std::time::Duration::from_secs(5))?;
let source = app.locator("list_item[name='notes.txt']").element()?;
let target = app.locator("list_item[name='Archive']").element()?;


let sim = input_sim()?;
sim.mouse().drag(&source, &target)?;

Error handling

Section titled “Error handling”

Input simulation raises five of xa11y’s error variants:

  • PermissionDenied when the OS denied the synthesis permission. On Linux this is the missing input group; macOS cannot report a missing Input Monitoring grant, so the events are dropped silently instead.
  • Unsupported when the platform has no backend for the operation, or when the Linux uinput kernel module is unavailable.
  • NoElementBounds when the target Element has no bounds. Re-fetch it, or anchor on a point.
  • InvalidActionData for an uppercase Char key, a malformed key name, or a bad target tuple.
  • Platform for a raw OS failure such as an FFI return code.

Arguments are parsed before the first OS call, so a rejected argument can never leave a half-delivered gesture behind with a key pressed and never released.

Note that NoElementBounds surfaces as a plain ValueError in Python rather than as an xa11y exception class. See Errors for the full mapping.