The fallback execution layer: when there's no API, CLI or file edit for the job, your AI uses the screen through clawdcursor. It picks the fastest way into each app: the page itself in browsers, the accessibility tree in desktop apps, text recognition when there's no tree, and screenshots last. After every action it tells your AI whether it worked.
clawdcursor is a free, open-source MCP server. Your AI connects to it and gets six tools to use a real Windows, macOS or Linux computer. It sees the parts of an app by name, such as "Email field" or "Confirm button", and acts on them: type into Email, press Confirm, choose High.
Where an app lists its parts, it reads the text exactly, not from a picture.
It clicks and types by name, so a moved or resized window does not throw it off. Clicking at a screen position is the fallback.
After each change it reads the result back and tells your AI one of three things: confirmed, unverified or failed.
Uninterrupted workflow. When your AI needs you for a step only you can take, such as a sign-in, a 2FA code or a choice, the screen glows blue and a card says what it needs. You do that one step, and your AI carries on with the task where it left off, with nothing to redo. If you take the mouse or keyboard mid-task, it waits and then picks up again.
Works with any AI that can call tools:
Our own benchmarks, each run on one machine. The scope of every result is stated with it. None of it is a general claim.
macOS, one machine, 2.2, 13 scored tasks per model. The whole run took 10 min 44 s for both models.
| Measure | Claude Sonnet 5.5 | Claude Haiku 5.5 |
| Tasks done | 11 of 13 | 13 of 13 |
| Within the call budget | 13 of 13 | 12 of 13 |
| Tool calls | 56 (a scripted reference run made 75) | 86 |
| Screenshots | 4 | 4 |
| False "done" | 0 | 0 |
| Loops | 0 | 0 |
| Cost for the 13 tasks | $0.51 | $0.048 |
| Seconds per task (median) | 18 s (229 s for all 13) | 22 s (321 s for all 13) |
| Actions by name or label / by screen position | 24 / 6 | 29 / 10 |
The scripted reference run (the "oracle") also needs 4 screenshots: the drawing-canvas tasks have no named parts. A task's seconds include starting the agent and the model's own thinking, not only clawdcursor.
Windows, one machine, 2.2. The agent never took the window you were using: foreground sampled every 50 ms, 0 steals. It made 0 screenshot calls. Details: REPORT-windows.md.
Reading the screen. Windows 11, 4K display, one machine, measured 2026-10-08: text recognition of one window takes 55–85 ms, and of the whole 4K screen about 0.54 s.
Waiting for a person. In a sign-up test the agent waited 4 min 43 s for the person to type a password. It made no action meanwhile, and finished on its own about 7 s after they clicked Resume.
Our own benchmark: the Codex agent on Windows, 15 tasks.
Where the app has an accessibility tree, the agent acted by name about 100% of the time. 12 of the 15 tasks stayed within their call budget. All 4 screenshots were on the two drawing-canvas tasks, which is the fewest those tasks allow. The run also found one false "ok", an action reported as done that was not: it was fixed after that run.
That run was narrow: one OS, one agent, 15 tasks we built for the test. A release now ships only when the benchmark passes. How it measures: bench/README.md.
Your AI tries these in order and uses the first that works. Your AI makes every decision; clawdcursor carries it out and checks it.
It opens its own Chrome or Edge window, with its own profile, and works on the web page itself through Playwright. A page already open in your own browser is read through the accessibility tree.
Apps list their buttons, fields and menus by name for screen readers. clawdcursor reads that list and acts on each part by name.
When an app lists nothing, it reads the words on the screen (OCR) and can click a visible label.
Only when nothing else works, such as on drawing canvases and 3D views. Your AI then gets a picture of the screen.
confirmed (it saw the change), unverified (nothing could check it) or failed (with the reason and what to try next).
refused means one of the protections below stopped the action.
| MCP | Model Context Protocol: the standard way to plug tools into an AI app. clawdcursor is the tool side; your AI app (Claude Code, Codex, Cursor and others) connects to it. |
| Accessibility tree | The list of buttons, fields and menus an app gives the system, each with a name. Screen readers for blind people use the same list. |
| Acting by name | Pressing "Send" because it is called Send, not because it sits at a spot on the screen. |
| OCR | Text recognition: reading the words in a picture of the screen. It runs on your computer. |
| Playwright | A library that controls Chrome and Edge from the inside, through the browser's own debugging connection. Apps built on a browser engine (Electron or WebView2) can be reached the same way. |
| Compact tools | The six tools your AI sees with --compact: accessibility, system, window, browser, computer and batch. Without it there are 89 smaller tools, one per action. |
It depends on apps exposing accessibility info. Most do: web pages, native apps, Office and system dialogs.
Drawing canvases, games and CAD views have no named parts. There it needs text recognition or screenshots.
The numbers above come from one machine each. Windows ran one agent (2.0) and macOS two models (2.2). Linux is supported but not benchmarked yet.
clawdcursor runs with your own permissions: anything you can do with a mouse and keyboard, your AI can do through it. These hold on every call:
While your AI works, a small pill on screen says what it is doing ("Typing in Edge"). Press Stop on it, or run clawdcursor stop. It stays stopped until you resume. Pause on the pill holds every act until you press Resume (Windows, macOS).
An amber outline around the agent's window: it works in the background and you keep using the computer. An amber glow on the whole screen: it is using the mouse and keyboard. A blue glow on the whole screen: it needs you for a step.
At a sign-in or a 2FA code, your AI hands that one step to you on a card and waits. Nothing else runs until you finish.
It never types into an AI assistant's window by accident, so text meant for an app does not land in a chat.
Password field values are never read through the accessibility tree, on Windows, macOS and Linux. Keys that lock the computer or end your session are never sent.
It runs on your computer. The HTTP server listens on 127.0.0.1 only and needs a token. The usual setup opens no port at all.
It waits while you use the computer (Windows, macOS). It never types or moves the mouse while your hands are on it: it waits until you have been idle for 1 s. If you take over mid-action, nothing more is sent; after 5 s it carries on if you are idle, otherwise it waits for Resume on the pill.
No built-in policy engine. clawdcursor does not decide what your AI may do: your AI and your MCP host's approval prompts do. If your host approves every tool call on its own, nothing asks before your AI sends, deletes or pays, unless you turn on confirm mode (from 2.1,
#373): with CLAWD_CONFIRM=1, or the Claude Desktop extension's "Ask before Send / Delete / Pay / Close" setting, it waits on the pause card until you click Resume.
The full threat model is in SECURITY.md.
2.0.0 sends nothing. From 2.1, clawdcursor sends anonymous error reports by default, so bugs get found and fixed.
start, when the server starts: OS major version, CPU architecture, Node.js major version, which AI app started it (from a fixed list; anything else is "other"), and compact or granular tools.call_failed, when a call fails, times out or reads nothing: the tool and action name, the result code, the outcome, how long it took, and whether the app was a browser, an Electron app or a native app.slow_call, when a call takes over 10 seconds: the tool and action name, how long it took, and its slow internal steps (clawdcursor's own helper programs, by name, with their time).exit, when the server exits during a call, or was killed or crashed: the exit reason (from a fixed list), and the tool and action that were cut off and how long they had run.crash, on an uncaught error: the error class and code, and the stack reduced to file and line in clawdcursor's own code.
Reports go over HTTPS to telemetry.clawdcursor.com, a Cloudflare Worker run by the project. It checks every event against the same fixed lists and drops anything else, does not read or store your IP address or request headers, and deletes events after 90 days.
To turn them off, run clawdcursor optout (clawdcursor optin turns them back on), ask your AI to opt out for you (it can only turn them off), or set CLAWD_TELEMETRY=0 or DO_NOT_TRACK=1. They are always off in CI and in test runs.
From 2.1, clawdcursor telemetry prints the status and every field it sends.
Install the plugin:
claude plugin marketplace add AmrDab/clawdcursor
claude plugin install clawdcursor@clawdcursor
Open the .mcpb extension from the latest release. No Node.js needed.
Install with npm (below), then add clawdcursor mcp --compact as an MCP server in your app.
$
Then run clawdcursor consent --accept. On macOS also run clawdcursor grant. To update: npm i -g clawdcursor@latest (from 2.1: clawdcursor update).
Same tools over two transports. clawdcursor never runs a model: your agent's host brings it.
Your agent's host starts clawdcursor over stdio. No daemon, no port. Config shapes for Codex, VS Code, Zed and opencode are in the README.
{
"mcpServers": {
"clawdcursor": {
"command": "clawdcursor",
"args": ["mcp", "--compact"]
}
}
}
HTTP MCP on 127.0.0.1:3847/mcp, for an agent that speaks MCP over HTTP. Every request needs the bearer token in ~/.clawdcursor/token. Loopback only; use an SSH tunnel from another machine, on the same local port as the daemon's.
2.0.0 is a breaking release. It was first published as 1.6.0 and renumbered to follow semver; 1.6.0 is deprecated on npm.
The built-in agent loop, the task tool, clawdcursor task and every provider, model and API-key setting are gone. Your agent drives the six tools.
Tiers, "confirm" and blocked labels are removed: a click by coordinates went around them. The protections above replace them.
It works beside you. On Windows and macOS it never types or moves the mouse while you use the computer, and the screen shows three states: an amber outline, an amber glow, a blue glow. The pill has Pause and Resume. On Windows the agent works in its own window without taking yours.
Everything else is in the changelog.
| Compound | Purpose | Actions |
|---|---|---|
computer |
Mouse, keyboard, screenshots. Raw I/O. | click · double_click · right_click · middle_click · triple_click · hover · move · move_relative · scroll · scroll_horizontal · drag · drag_path · mouse_down · mouse_up · type · key · key_press · key_down · key_up · wait · screenshot · screenshot_region · zoom |
accessibility |
Drive UI by element name, not pixel. Survives DPI, resize, layout shifts. | read_tree · find · get_element · focused · invoke · focus · set_value · get_value · expand · collapse · toggle · select · state · list_children · wait_for · compile_ui · find_button · find_field · smart_click · smart_type · smart_read |
window |
Launch, focus, resize. App-level state. | list · active · focus · maximize · minimize · restore · close · resize · list_displays · screen_size · open_app · open_file · open_url · switch_tab · navigate |
system |
Page text, OCR, clipboard, shortcuts, undo, webview detection, launching, and the pause that hands a step to you. | clipboard_read · clipboard_write · copy_all_text · report · system_time · ocr · undo · shortcuts_list · shortcuts_run · pause_for_user · detect_webview · relaunch_with_cdp · build_uri · open_uri · open_app · open_file · open_url |
browser |
Playwright over the Chrome DevTools Protocol: the web page itself, in its own Chrome or Edge window. Also Electron / WebView2 apps whose accessibility tree is sparse. | navigate · connect · page_context · read_text · click · type · select_option · evaluate · wait_for · list_tabs · switch_tab · scroll |
batch |
Several calls in one round trip. Halts with a per-step trace on a refusal, an error or a failed expect. |
steps: [{ name, arguments, expect? }] |
Compact (about 7,000 tokens of catalog; granular is about 19,000):computer({ "action": "key", "combo": "mod+s" }).
Granular: key_press({ "key": "mod+s" }).
Both run the same tools behind the same guards. Leave out --compact for the granular surface.
See schema.snapshot.json for every parameter.
# Install & setup
clawdcursor consent # one-time desktop-control authorization (always required)
clawdcursor grant # macOS only: Accessibility + Screen Recording prompts. MCP setup ends here.
clawdcursor doctor # diagnose the install: version, consent, screen capture, accessibility, permissions
clawdcursor status # readiness check (consent, permissions, whether the daemon is running)
clawdcursor versions # every installed copy and its version
clawdcursor update # update the global npm install, list the other copies to update (from 2.1)
clawdcursor telemetry # whether error reports are on, and every field sent (from 2.1)
clawdcursor optout # turn error reports off; optin turns them back on (from 2.1)
clawdcursor register-skill # register the usage skill in detected agents again
# Run
clawdcursor mcp # stdio MCP server; your agent's host starts it
clawdcursor mcp --compact # same, with the 6 compact tools (recommended)
clawdcursor agent # HTTP MCP server at 127.0.0.1:3847/mcp (--compact, --port, --no-vision)
clawdcursor stop # stop every running instance; agents stay stopped until resume
clawdcursor resume # continue the daemon's paused task, or give control back after a stop (--end to end the task)
clawdcursor report # send a redacted report of the last session, after a preview
clawdcursor uninstall # remove all clawdcursor config and data
Free and open source (MIT). Works with any AI that can call tools. Runs on your computer; from 2.1 it sends anonymous error reports, off with clawdcursor optout.