Give your AI
eyes and hands

Stage 1 · SnapshotBuilder

FREE — no LLM cost

Parallel OCR + accessibility tree capture in one pass. Produces a unified perception object with every text element, button, input field, and their exact screen coordinates. Also runs spatial layout analysis — detecting toolbar zones, content area center, sidebars, and status bar from element clustering. The LLM gets a text-based map of the screen without needing vision.

Stage 2 · TextNavigator ★

CHEAP — any text model ($0.25/1M tokens with Haiku)

Primary reasoning stage. A cheap text LLM reads the OCR snapshot, spatial layout, and app-specific guide, then decides the next action: click, type, drag, key press, scroll, or done. One action per LLM call, verified by re-scanning. Handles 90% of tasks without ever taking a screenshot.

Stage 3 · VisionFiller

EXPENSIVE — vision model, last resort only

Fires only when Stage 2 can't handle it (CAPTCHAs, complex spatial tasks, visual content). Takes screenshots, sends to a vision LLM, executes tool calls. Supports Anthropic native Computer Use and any OpenAI-compatible vision model (GPT-4o, Gemini, Kimi k2.5).

Pre-processor + Task Classifier

Before the pipeline runs: a regex parser decomposes compound tasks ("type X, then save as Y" → 2 subtasks). A zero-cost classifier categorizes each subtask — mechanical (router handles it for free), navigation (Ctrl+L + URL), reasoning (Stage 2), or spatial (needs vision). No LLM wasted on tasks a regex can route.

Textual Scaffolding (Spatial Layout Analysis)

ClawdCursor's breakthrough: instead of sending screenshots, it analyzes WHERE elements are clustered on screen and builds a text-based spatial map. The LLM reads: Content area: center = (1920, 1200) — CLICK HERE. This lets any text model — even a 7B parameter local model — know exactly where to click in Google Docs, Excel, Paint, or any app. No vision required.

App Guides — 86+ apps

Community-contributed JSON instruction manuals. Keyboard shortcuts, workflows, UI layout hints, and tips. Excel (116 shortcuts), Paint, Notepad, Outlook, Spotify, Discord, Figma, and more. Install with clawdcursor guides install excel. Loaded automatically when the app is detected.

Smart Tools (Blind Agent)

Click buttons by name, type into fields by label, read screen text — all without screenshots. smart_click, smart_type, smart_read use accessibility + OCR fallback automatically.

Shortcuts Engine

Built-in keyboard shortcut database. shortcuts_list discovers available shortcuts for any app. shortcuts_execute fires them instantly. Zero LLM cost.

Ground-Truth Verifier

Re-scans the screen after every action. Done verification checks if task keywords appear. Drawing tasks get lenient verification. The agent can't hallucinate success.

Three Transport Modes

clawdcursor start (full agent + REST), clawdcursor serve (tools only, bring your own brain), clawdcursor mcp (MCP stdio for Claude Code, Cursor, Windsurf, Zed).

Truly Universal

13 providers auto-detected. Display-agnostic (720p to 5K, any DPI). Cross-platform (Windows, macOS, Linux). Provider-agnostic — zero hardcoded model names. Community-driven app knowledge. If a human can do it on screen, clawdcursor can too.