Agent Desktop

agent-desktop is a native desktop automation CLI built in Rust, designed specifically for AI agents to observe, decide, and act on any desktop application. It provides structured access to applications through OS accessibility trees — no screenshots, no pixel matching, no browser required. The tool outputs machine-readable JSON with deterministic element references, making it ideal for agentic workflows that require reliable, repeatable UI interactions.

• Native Rust CLI: Fast, single binary with no runtime dependencies — install via npm install -g agent-desktop or build from source with Cargo.
• 53 commands: Covers observation, interaction, keyboard, mouse, notifications, clipboard, and window management for comprehensive desktop control.
• Progressive skeleton traversal: Achieves 78–96% token reduction on dense apps via shallow overview and targeted drill-down, minimizing LLM context usage.
• Snapshot & refs system: AI-optimized workflow using deterministic element references (@e1, @e2) that persist until the next snapshot, enabling reliable act-verify loops.
• AX-first interactions: Every action exhausts pure accessibility API strategies before falling back to mouse events, maximizing reliability.
• Structured JSON output: All commands return machine-readable responses with error codes and recovery hints for robust agent error handling.
• C-ABI cdylib (FFI): Load libagent_desktop_ffi once from Python, Swift, Go, Ruby, Node, or C instead of forking the CLI per call — prebuilt binaries ship with every release.
• Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an OS accessibility tree is supported.
• Batch command execution: Run multiple commands in a single call with --stop-on-error support for efficient multi-step agent workflows.
• Cross-platform FFI binaries: Prebuilt cdylib artifacts available for macOS arm64/x86_64, Linux x86_64/arm64, and Windows x86_64.