← Ayo Osunjuyigbe

· AI

Claude can control your desktop now. Confirm before it clicks.

Anthropic shipped computer use on the 22nd. The write-up sounds like a new tool. It is not. There is no hosted desktop. You send Claude a screenshot. It sends back pixel coordinates and a key. Your loop moves the mouse. You are the hands.

I have been wiring models to bash and a browser for a year. That was still a menu I wrote. This is the other direction. They taught the model to use software that was never designed for a model. The screen is now retrieved content. Anything on it is in the prompt.

What shipped

The model is claude-3-5-sonnet-20241022. Same price as June’s 3.5 Sonnet. Computer use is a public beta on the Anthropic API, Bedrock, and Vertex. You send the beta header computer-use-2024-10-22 and a tool of type computer_20241022 named computer, plus the width and height of the display you are going to show it.

Then the loop:

  1. You take a screenshot of a machine you control.
  2. You put that image in the next user turn.
  3. Claude returns tool_use: screenshot, mouse_move, left_click, type, key, a drag if you are lucky.
  4. You run it — the reference demo shells out to xdotool inside an Ubuntu container with VNC.
  5. You send the result back. Usually another picture.

The new model skill is coordinates. Older vision models could narrate a button. They could not say click 412, 880. Anthropic trained this one to count pixels. Their research post is blunt about why: without that, mouse commands look like asking the model how many A’s are in banana.

They also shipped a bash tool and a text-editor tool in the same kit. The interesting one is still the desktop. bash is a tool you scoped. The desktop is every tool the OS already had.

Anthropic’s own sentence, not mine: until now, developers made tools fit the model. Now the model fits the tools people already click.

It is bad at the computer. That is not the point

OSWorld is the benchmark they used: real Ubuntu/Windows/macOS tasks, 369 of them in the paper, screenshot-only. Humans who do not even know the software sit around 72%. The new Sonnet got 14.9% at the usual 15-step cap, 22% if you let it take about 50 steps. Next-best screenshot agent they cite: about 8%. The OSWorld paper itself, from April, had GPT-4V-class screenshot agents in the mid-single digits.

So: state of the art, and still fails most of the exam. Scrolling, dragging, zooming are messy. The model sees a flipbook of screenshots, not a video, so a toast or a two-second dialog can vanish between frames. While they were recording the launch demos, Claude stopped a long screen recording and deleted the footage. In another take it left the coding task and opened Yellowstone photos.

They told developers to start with low-risk tasks. Believe them. Replit is using it to look at an app while Agent builds it, not to file your taxes. The launch named a pile of design partners. That is exploration. It is not a reliability number.

The SWE-bench jump (33.4% → 49.0%) is a different model improvement. Do not mix it into Claude can use a computer now. Computer use is 15% on a desktop exam and a new attack surface.

The pixels are untrusted input

Anthropic’s Trust & Safety write-up names the failure on day one. Prompt injection. The model reads screenshots from a machine that can open the internet. A page, a PDF, an image, a Slack thread in the corner of the screen can print instructions. The model has already been trained to treat what I see as the next step. Sometimes it will follow the page instead of you.

Their docs, in the red box: Claude will, in some circumstances, follow commands found in content even when they conflict with the user’s instructions. Isolate it from sensitive data and actions.

That is the whole security model. You did not add a browser tool with a domain allowlist. You added eyes. The allowlist is whatever the window manager will show.

I am not going to write a recipe. The shape is enough. If the agent can see a tab, that tab is in context. If the agent can click, a sentence on that tab can become a click. Confirm before send, before install, before a transfer. A safety classifier is not a substitute for a human on those three.

They kept the model at ASL-2. Computer use, they argue, mostly lowers the barrier to skills the model already had; it does not jump a catastrophic-risk tier. Fair as a scaling-policy sentence. Useless as a product sentence. The harm that matters this month is the agent clicking the thing the banner told it to click, on a machine that also has your cookie jar.

They trained on a calculator and a text editor. No internet in training, on purpose. Then they handed developers a Docker image that can apt-get and open Firefox. Generalization is the demo. It is also how a prompt the model never saw in training ends up on the screen.

What I run

I will try the official container. I will not point this at the laptop I bank on.

The machine is a throwaway. Ubuntu in Docker, as they shipped. No host mounts of ~/.ssh. No password manager extension. No just this once on the daily browser profile. If the model needs a login, that account is a robot account, and I am ready to burn it.

I execute. It proposes. Same rule as any tool loop. A mouse_move plus left_click on Send or Confirm payment or Allow paste waits for me. Anthropic can nudge the model off government sites and social posts. They said they would, because of the election next week. That does not click Send for me.

The prompt does not get to outrank the screen. The task text is one block. The pixels are another. Treat every screenshot like an email you would not mark as trusted.

Low-risk means reversible. Fill a form in a staging admin. Walk a QA script. Read a page and take notes in the container. Do not let it download and run something it found in a tab. Do not give it the production AWS console and a vague fix the error.

I wanted a better computer-use API for years. This is not that API. This is a vision model with a coordinate head and a loop I still have to write. The loop is the product. The screenshot is the prompt. If you cannot watch the VNC pane, you are not ready to let it click.

References