Skip to content
xa11y

Screenshots

xa11y can capture pixel-level snapshots of the screen alongside its accessibility APIs. This is useful for attaching screenshots to test failures, feeding vision-language models with the same frame the agent is reasoning about, or visually diffing a region around a specific control.

The entry points are:

  • Full screen captures whatever the platform treats as the whole screen. On Windows that is the virtual desktop, which spans every monitor. macOS captures one display. Linux captures the X11 root window or the Wayland portal’s desktop image.
  • Region captures an explicit rectangle in desktop coordinates (physical pixels on Windows, logical points elsewhere).
  • Element captures the pixels under an accessibility element’s current bounds.

Screenshots are decoupled from the accessibility and input layers. Capture never raises or activates the target window. Whatever pixels sit at the requested coordinates are returned, including any occluding windows. Bring the target to the foreground yourself if you need a clean capture.

Permissions

Section titled “Permissions”
Platform Requirement
macOS Screen & System Audio Recording for the terminal/IDE
Windows None for normal user sessions
Linux (X11) DISPLAY set to a reachable X server
Linux (Wayland) xdg-desktop-portal with screenshot support (user consent dialog may appear)

Capturing

Section titled “Capturing”
use xa11y::*;


fn main() -> Result<()> {
    // Full screen
    let shot = screenshot()?;
    shot.save_png("full.png")?;


    // Explicit region
    let shot = screenshot_region(Rect { x: 0, y: 0, width: 400, height: 300 })?;
    shot.save_png("corner.png")?;


    // Pixels under an accessibility element
    let app = App::by_name("Calculator", std::time::Duration::from_secs(5))?;
    let display = app.locator("static_text[name='Result']").element()?;
    let shot = screenshot_element(&display)?;
    shot.save_png("result.png")?;


    Ok(())
}

Working with the pixel buffer

Section titled “Working with the pixel buffer”

A Screenshot carries raw RGBA8 pixels, dimensions, a compatibility scale, and the mapping frozen when the pixels were captured. width and height always describe the actual returned image, including a crop or resize, and pixels.len() equals width * height * 4. scale remains useful for old single-display code, but one scalar cannot describe a mixed-DPI desktop or a non-uniform resize. Use desktop_to_image / image_to_desktop for points and desktop_rect_to_image for rectangles. JavaScript uses desktopToImage, imageToDesktop, and desktopRectToImage. A rectangle that crosses displays can produce multiple image rectangles.

The conversion methods fail explicitly when mapping is unavailable (for example, a Rust Screenshot::new built from arbitrary pixels). On Windows they also reject a capture after the display layout or DPI arrangement has changed; capture a fresh image before using one of its pixels as an input target. This does not detect an ordinary app or window moving.

let shot = screenshot()?;
println!("{} x {} @ {}x", shot.width, shot.height, shot.scale);


// Encode as PNG bytes (e.g. to upload or embed)
let png_bytes: Vec<u8> = shot.to_png()?;


// Or write directly to disk
shot.save_png("out.png")?;


// Raw RGBA for custom processing
let rgba: &[u8] = &shot.pixels;


// Pixel in this exact image -> desktop point suitable for input.
let target = shot.image_to_desktop(Point::new(20, 30))?;


// Cropping and resizing carry the mapping forward.
let thumbnail = shot.crop(Rect { x: 0, y: 0, width: 800, height: 600 })?
    .resize(400, 300)?;

Migrating from v0.14

Section titled “Migrating from v0.14”

Code that treated Screenshot.scale as a universal desktop-to-image multiplier should use desktop_to_image, image_to_desktop, and desktop_rect_to_image instead. A single scale cannot represent a capture that crosses displays with different DPI settings. Captured screenshots now retain the exact mapping that produced their pixels, and crop and resize carry that mapping forward. scale remains for compatibility; it is 1.0 on Windows and a single scale hint elsewhere.

On Windows, bounds, input, window moves and sizes, and capture regions now use physical desktop pixels. This is a breaking coordinate change from v0.14’s logical Windows units: convert any stored DIP coordinates before passing them to xa11y. Screenshot.scale is 1.0 on Windows, and full-screen image dimensions match the physical virtual desktop extent. Right and bottom input anchors select the last point inside the element’s half-open bounds.

Example: screenshot on test failure

Section titled “Example: screenshot on test failure”

A common pattern is to capture the full screen (or the app under test) whenever an assertion fails, and attach it to the test report.

fn capture_on_fail(name: &str, test: impl FnOnce() -> Result<()>) -> Result<()> {
    match test() {
        Ok(()) => Ok(()),
        Err(e) => {
            if let Ok(shot) = screenshot() {
                let _ = shot.save_png(format!("target/failures/{name}.png"));
            }
            Err(e)
        }
    }
}

Error handling

Section titled “Error handling”

Capture raises four of xa11y’s error variants:

  • PermissionDenied when macOS Screen Recording is not granted, or the Wayland portal denied consent.
  • Unsupported when no capture backend covers the session, which a missing DISPLAY or an unresponsive Wayland portal will both produce.
  • NoElementBounds when the element passed to screenshot_element has no bounds.
  • Platform for a raw OS or FFI failure during capture or PNG encode.

The Python example above catches xa11y.XA11yError, which covers PermissionDenied, Unsupported, and Platform. NoElementBounds surfaces as a plain ValueError in Python, so catch that too when capturing by element. See Errors for the full mapping.