Vercel Labs Open-Sources agent-desktop for Agent Computer Use

Why it matters

Vercel Labs has released agent-desktop, an open-source tool that lets AI agents operate native desktop applications by reading operating-system accessibility trees instead of taking screenshots. Written in Rust and also offered as a C library, it runs on macOS, Windows and Linux across common CPU architectures, and any shell-capable coding agent can call it without needing a vision model. The tool is deliberately minimal: it embeds no language model and has no graphical interface, leaving the observe-act-verify loop to the agent itself. Agents receive structured snapshots with stable references to interface elements, and actions run headlessly through the native accessibility APIs without moving the pointer or stealing focus, so a person can keep using the machine while an agent works. The design contrasts with screenshot-based computer use, where images are costly to process. The builder reports large token savings on dense applications, but these figures are self-reported and no independent benchmark exists yet. Apps lacking accessibility trees remain out of reach.