v2.2.0 · free and open source · Windows, macOS, Linux

Gives any AI control
over the GUI.

The fallback execution layer: when there's no API, CLI or file edit for the job, your AI uses the screen through clawdcursor. It picks the fastest way into each app: the page itself in browsers, the accessibility tree in desktop apps, text recognition when there's no tree, and screenshots last. After every action it tells your AI whether it worked.

What it does

Your AI sees each app by name, and acts on it.

clawdcursor is a free, open-source MCP server. Your AI connects to it and gets six tools to use a real Windows, macOS or Linux computer. It sees the parts of an app by name, such as "Email field" or "Confirm button", and acts on them: type into Email, press Confirm, choose High.

🔎

Reads what is there

Where an app lists its parts, it reads the text exactly, not from a picture.

🖱️

Acts by name

It clicks and types by name, so a moved or resized window does not throw it off. Clicking at a screen position is the fallback.

✅

Checks every step

After each change it reads the result back and tells your AI one of three things: confirmed, unverified or failed.

Uninterrupted workflow. When your AI needs you for a step only you can take, such as a sign-in, a 2FA code or a choice, the screen glows blue and a card says what it needs. You do that one step, and your AI carries on with the task where it left off, with nothing to redo. If you take the mouse or keyboard mid-task, it waits and then picks up again.

Works with any AI that can call tools:

Claude Code Claude Desktop Codex Cursor Windsurf Zed local models
Proof

What we measured.

Our own benchmarks, each run on one machine. The scope of every result is stated with it. None of it is a general claim.

Speed and accuracy on macOS (2.2)

macOS, one machine, 2.2, 13 scored tasks per model. The whole run took 10 min 44 s for both models.

Measure Claude Sonnet 5.5 Claude Haiku 5.5
Tasks done 11 of 13 13 of 13
Within the call budget 13 of 13 12 of 13
Tool calls 56 (a scripted reference run made 75) 86
Screenshots 4 4
False "done" 0 0
Loops 0 0
Cost for the 13 tasks $0.51 $0.048
Seconds per task (median) 18 s (229 s for all 13) 22 s (321 s for all 13)
Actions by name or label / by screen position 24 / 6 29 / 10

The scripted reference run (the "oracle") also needs 4 screenshots: the drawing-canvas tasks have no named parts. A task's seconds include starting the agent and the model's own thinking, not only clawdcursor.

Own-window work on Windows (2.2)

14/14
On-screen checks passed
0
Focus steals
0
Screenshots

Windows, one machine, 2.2. The agent never took the window you were using: foreground sampled every 50 ms, 0 steals. It made 0 screenshot calls. Details: REPORT-windows.md.

Reading the screen, and waiting for a person

Reading the screen. Windows 11, 4K display, one machine, measured 2026-10-08: text recognition of one window takes 55–85 ms, and of the whole 4K screen about 0.54 s.

Waiting for a person. In a sign-up test the agent waited 4 min 43 s for the person to type a password. It made no action meanwhile, and finished on its own about 7 s after they clicked Resume.

Earlier measurement (2.0)

Our own benchmark: the Codex agent on Windows, 15 tasks.

14/15
Tasks done
4
Screenshots in all 15 tasks
0
Loops

Where the app has an accessibility tree, the agent acted by name about 100% of the time. 12 of the 15 tasks stayed within their call budget. All 4 screenshots were on the two drawing-canvas tasks, which is the fewest those tasks allow. The run also found one false "ok", an action reported as done that was not: it was fixed after that run.

That run was narrow: one OS, one agent, 15 tasks we built for the test. A release now ships only when the benchmark passes. How it measures: bench/README.md.

How it works

The fastest way in first. Screenshots last.

Your AI tries these in order and uses the first that works. Your AI makes every decision; clawdcursor carries it out and checks it.

1 The page

Its own browser window

It opens its own Chrome or Edge window, with its own profile, and works on the web page itself through Playwright. A page already open in your own browser is read through the accessibility tree.

2 The tree

Desktop apps, your browser

Apps list their buttons, fields and menus by name for screen readers. clawdcursor reads that list and acts on each part by name.

3 Text recognition

No tree

When an app lists nothing, it reads the words on the screen (OCR) and can click a visible label.

4 Screenshot

Last

Only when nothing else works, such as on drawing canvases and 3D views. Your AI then gets a picture of the screen.

After every action clawdcursor reads the result back and ends its answer with one word: confirmed (it saw the change), unverified (nothing could check it) or failed (with the reason and what to try next). refused means one of the protections below stopped the action.

The words, in plain terms

MCP Model Context Protocol: the standard way to plug tools into an AI app. clawdcursor is the tool side; your AI app (Claude Code, Codex, Cursor and others) connects to it.
Accessibility tree The list of buttons, fields and menus an app gives the system, each with a name. Screen readers for blind people use the same list.
Acting by name Pressing "Send" because it is called Send, not because it sits at a spot on the screen.
OCR Text recognition: reading the words in a picture of the screen. It runs on your computer.
Playwright A library that controls Chrome and Edge from the inside, through the browser's own debugging connection. Apps built on a browser engine (Electron or WebView2) can be reached the same way.
Compact tools The six tools your AI sees with --compact: accessibility, system, window, browser, computer and batch. Without it there are 89 smaller tools, one per action.
Limits

Where it is weaker.

🧩

It needs apps to describe themselves

It depends on apps exposing accessibility info. Most do: web pages, native apps, Office and system dialogs.

🎨

Apps that draw their own screen

Drawing canvases, games and CAD views have no named parts. There it needs text recognition or screenshots.

📏

Measured on Windows and macOS

The numbers above come from one machine each. Windows ran one agent (2.0) and macOS two models (2.2). Linux is supported but not benchmarked yet.

Safety

What protects you.

clawdcursor runs with your own permissions: anything you can do with a mouse and keyboard, your AI can do through it. These hold on every call:

⏹️

A Stop button

While your AI works, a small pill on screen says what it is doing ("Typing in Edge"). Press Stop on it, or run clawdcursor stop. It stays stopped until you resume. Pause on the pill holds every act until you press Resume (Windows, macOS).

🟧

Three screen states

An amber outline around the agent's window: it works in the background and you keep using the computer. An amber glow on the whole screen: it is using the mouse and keyboard. A blue glow on the whole screen: it needs you for a step.

🟦

A pause card for you

At a sign-in or a 2FA code, your AI hands that one step to you on a card and waits. Nothing else runs until you finish.

💬

No typing into AI chats

It never types into an AI assistant's window by accident, so text meant for an app does not land in a chat.

🔒

Passwords are not read

Password field values are never read through the accessibility tree, on Windows, macOS and Linux. Keys that lock the computer or end your session are never sent.

🔑

Local only

It runs on your computer. The HTTP server listens on 127.0.0.1 only and needs a token. The usual setup opens no port at all.

It waits while you use the computer (Windows, macOS). It never types or moves the mouse while your hands are on it: it waits until you have been idle for 1 s. If you take over mid-action, nothing more is sent; after 5 s it carries on if you are idle, otherwise it waits for Resume on the pill.

No built-in policy engine. clawdcursor does not decide what your AI may do: your AI and your MCP host's approval prompts do. If your host approves every tool call on its own, nothing asks before your AI sends, deletes or pays, unless you turn on confirm mode (from 2.1, #373): with CLAWD_CONFIRM=1, or the Claude Desktop extension's "Ask before Send / Delete / Pay / Close" setting, it waits on the pause card until you click Resume. The full threat model is in SECURITY.md.

Error reports

Error reports, from 2.1.

2.0.0 sends nothing. From 2.1, clawdcursor sends anonymous error reports by default, so bugs get found and fixed.

What is sent

  • With every report: a random install id made on your computer (not derived from your hardware or accounts), the clawdcursor version and your OS family.
  • start, when the server starts: OS major version, CPU architecture, Node.js major version, which AI app started it (from a fixed list; anything else is "other"), and compact or granular tools.
  • call_failed, when a call fails, times out or reads nothing: the tool and action name, the result code, the outcome, how long it took, and whether the app was a browser, an Electron app or a native app.
  • slow_call, when a call takes over 10 seconds: the tool and action name, how long it took, and its slow internal steps (clawdcursor's own helper programs, by name, with their time).
  • exit, when the server exits during a call, or was killed or crashed: the exit reason (from a fixed list), and the tool and action that were cut off and how long they had run.
  • crash, on an uncaught error: the error class and code, and the stack reduced to file and line in clawdcursor's own code.
  • Every value is a number or comes from a fixed list; tool and action names are clawdcursor's own.

What is never sent

  • Error messages, typed text, the clipboard, screen or OCR text, screenshots, window titles, the names of the apps your AI works in, URLs, element names, file paths outside clawdcursor, user names, emails or IP addresses.

Reports go over HTTPS to telemetry.clawdcursor.com, a Cloudflare Worker run by the project. It checks every event against the same fixed lists and drops anything else, does not read or store your IP address or request headers, and deletes events after 90 days. To turn them off, run clawdcursor optout (clawdcursor optin turns them back on), ask your AI to opt out for you (it can only turn them off), or set CLAWD_TELEMETRY=0 or DO_NOT_TRACK=1. They are always off in CI and in test runs. From 2.1, clawdcursor telemetry prints the status and every field it sends.

Install

Install it for your AI app.

Claude Code

Install the plugin:

claude plugin marketplace add AmrDab/clawdcursor
claude plugin install clawdcursor@clawdcursor

Claude Desktop

Open the .mcpb extension from the latest release. No Node.js needed.

Codex, Cursor and others

Install with npm (below), then add clawdcursor mcp --compact as an MCP server in your app.


$ 

Then run clawdcursor consent --accept. On macOS also run clawdcursor grant. To update: npm i -g clawdcursor@latest (from 2.1: clawdcursor update).

Connect

How will your agent talk to it?

Same tools over two transports. clawdcursor never runs a model: your agent's host brings it.

clawdcursor mcp recommended

Your agent's host starts clawdcursor over stdio. No daemon, no port. Config shapes for Codex, VS Code, Zed and opencode are in the README.

{
  "mcpServers": {
    "clawdcursor": {
      "command": "clawdcursor",
      "args": ["mcp", "--compact"]
    }
  }
}
6 / 89
Compact / Granular tools
stdio
Transport

clawdcursor agent HTTP daemon

HTTP MCP on 127.0.0.1:3847/mcp, for an agent that speaks MCP over HTTP. Every request needs the bearer token in ~/.clawdcursor/token. Loopback only; use an SSH tunnel from another machine, on the same local port as the daemon's.

:3847
HTTP MCP
2.0.0

What changed, and what is next.

2.0.0 is a breaking release. It was first published as 1.6.0 and renumbered to follow semver; 1.6.0 is deprecated on npm.

🧰

A tool layer

The built-in agent loop, the task tool, clawdcursor task and every provider, model and API-key setting are gone. Your agent drives the six tools.

🚧

No policy gate

Tiers, "confirm" and blocked labels are removed: a click by coordinates went around them. The protections above replace them.

🧭

2.2

It works beside you. On Windows and macOS it never types or moves the mouse while you use the computer, and the screen shows three states: an amber outline, an amber glow, a blue glow. The pill has Pause and Resume. On Windows the agent works in its own window without taking yours.

Everything else is in the changelog.

Tools 6 compact tools, or 89 granular The 6 compact tools are the recommended surface. The 89 granular tools expose one tool per action.
Compound Purpose Actions
computer Mouse, keyboard, screenshots. Raw I/O. click · double_click · right_click · middle_click · triple_click · hover · move · move_relative · scroll · scroll_horizontal · drag · drag_path · mouse_down · mouse_up · type · key · key_press · key_down · key_up · wait · screenshot · screenshot_region · zoom
accessibility Drive UI by element name, not pixel. Survives DPI, resize, layout shifts. read_tree · find · get_element · focused · invoke · focus · set_value · get_value · expand · collapse · toggle · select · state · list_children · wait_for · compile_ui · find_button · find_field · smart_click · smart_type · smart_read
window Launch, focus, resize. App-level state. list · active · focus · maximize · minimize · restore · close · resize · list_displays · screen_size · open_app · open_file · open_url · switch_tab · navigate
system Page text, OCR, clipboard, shortcuts, undo, webview detection, launching, and the pause that hands a step to you. clipboard_read · clipboard_write · copy_all_text · report · system_time · ocr · undo · shortcuts_list · shortcuts_run · pause_for_user · detect_webview · relaunch_with_cdp · build_uri · open_uri · open_app · open_file · open_url
browser Playwright over the Chrome DevTools Protocol: the web page itself, in its own Chrome or Edge window. Also Electron / WebView2 apps whose accessibility tree is sparse. navigate · connect · page_context · read_text · click · type · select_option · evaluate · wait_for · list_tabs · switch_tab · scroll
batch Several calls in one round trip. Halts with a per-step trace on a refusal, an error or a failed expect. steps: [{ name, arguments, expect? }]

Compact (about 7,000 tokens of catalog; granular is about 19,000):computer({ "action": "key", "combo": "mod+s" }). Granular: key_press({ "key": "mod+s" }). Both run the same tools behind the same guards. Leave out --compact for the granular surface. See schema.snapshot.json for every parameter.

CLI Every command For humans diagnosing an install. Agents connect via MCP.
# Install & setup
clawdcursor consent          # one-time desktop-control authorization (always required)
clawdcursor grant            # macOS only: Accessibility + Screen Recording prompts. MCP setup ends here.
clawdcursor doctor           # diagnose the install: version, consent, screen capture, accessibility, permissions
clawdcursor status           # readiness check (consent, permissions, whether the daemon is running)
clawdcursor versions         # every installed copy and its version
clawdcursor update           # update the global npm install, list the other copies to update (from 2.1)
clawdcursor telemetry        # whether error reports are on, and every field sent (from 2.1)
clawdcursor optout           # turn error reports off; optin turns them back on (from 2.1)
clawdcursor register-skill   # register the usage skill in detected agents again

# Run
clawdcursor mcp              # stdio MCP server; your agent's host starts it
clawdcursor mcp --compact    # same, with the 6 compact tools (recommended)
clawdcursor agent            # HTTP MCP server at 127.0.0.1:3847/mcp (--compact, --port, --no-vision)
clawdcursor stop             # stop every running instance; agents stay stopped until resume
clawdcursor resume           # continue the daemon's paused task, or give control back after a stop (--end to end the task)
clawdcursor report           # send a redacted report of the last session, after a preview
clawdcursor uninstall        # remove all clawdcursor config and data

Let your AI use your computer.

Free and open source (MIT). Works with any AI that can call tools. Runs on your computer; from 2.1 it sends anonymous error reports, off with clawdcursor optout.

Star on GitHub