Computer Use

What Is Computer Use?

Computer use is an AI agent capability where the model operates a computer through its graphical interface: it looks at screenshots, moves the cursor, clicks, types, and scrolls, exactly as a person would. This lets an agent work with any application on screen, including software that offers no API for structured access.

Key Takeaways

• The interface is pixels and input events. The model perceives the screen visually and acts through synthetic mouse and keyboard events, so nothing about the target application needs to change.
• Its value is universality. Legacy desktop software, virtual desktops, admin consoles, and long-tail web apps all become automatable without integration work.
• It is the slowest and least reliable tool an agent has. Every step costs a screenshot round trip, so well-designed agents reach for APIs first and the GUI last.
• Sandboxing is non-negotiable. An agent that can click anything can also click "Delete," so computer use runs in a VM or isolated desktop with scoped credentials.

How It Works

The loop is perception and action. The harness captures a screenshot and sends it to the model along with the goal and the history so far. The model reasons about what is on screen, then emits a low-level action: move the cursor to coordinates, click, double-click, drag, type text, press a key combination, scroll. The harness executes the action against a real or virtual display, waits for the UI to settle, captures a fresh screenshot, and the cycle repeats. Frontier models are trained specifically for this grounding task, mapping visual elements to precise coordinates and recognizing state changes like dialogs, spinners, and validation errors. That training shows in benchmarks: Claude 3.5 Sonnet, the first frontier model to offer computer use, scored 14.9% on OSWorld's screenshot-only category in 2024, nearly double the next-best system's 7.8% [1].

Because each cycle involves vision inference, computer use is an order of magnitude slower than direct tool calling. Production agents therefore treat it as one tool among several. If the task can be done through an API, a database query, or a CLI, those routes win on speed and determinism; the GUI path is reserved for the systems that offer nothing better. Reliability engineering focuses on verification, having the agent confirm from the next screenshot that the intended change actually happened, and on recovery, since misclicks, focus issues, and unexpected popups are routine. The distance to human reliability is still large: on the OSWorld benchmark of real computer tasks, published in 2024, humans succeed at over 72% of tasks while the best AI model at launch achieved only 12.24% [2].

The security model deserves emphasis. A computer-use agent inherits whatever the desktop session can do, and everything it reads on screen enters its context, which makes prompt injection through hostile page content a live threat. Standard practice is a disposable VM, a dedicated low-privilege account, no access to personal sessions, and human confirmation before irreversible actions.

Example

A finance team needs monthly vendor invoices pulled from eleven supplier portals, most of which have no API and two of which are aging desktop-era web apps. An agent with computer use logs into each portal inside a sandboxed browser VM using vault-scoped credentials, navigates to the billing section, sets the date filter, downloads the PDFs, and files them into the accounting share with a normalized naming scheme. When one portal has redesigned its navigation since last month, the agent does not break the way a scripted bot would; it reads the new layout from the screenshot, finds the relocated "Statements" menu, and completes the run, noting the change in its log.

What People Get Wrong

The recurring mistake is treating computer use as the default way agents should interact with software, because it looks the most general. Generality is exactly what makes it a last resort. A GUI session is slow, stateful, and fragile compared to an API call, and it multiplies token cost through screenshots. Teams that route everything through a virtual desktop build agents that are impressive in demos and painful in production. The discipline is a tool hierarchy: structured interfaces first, browser automation next, raw computer use only for the software that leaves no alternative.

FAQ

How is computer use different from a browser agent? Scope. A browser agent is confined to web pages and can often read the page's DOM structure directly, which is faster and more precise. Computer use operates the whole operating system from pixels, covering desktop applications, file managers, and anything else on screen. Many stacks combine them, using DOM-based browsing where possible and falling back to screen control elsewhere.

Is computer use AI reliable enough for production? For bounded, verifiable workflows, yes, with engineering around it: deterministic checkpoints, screenshot-based verification after each critical step, retry logic, and human review of outcomes. Long free-form sessions across unfamiliar interfaces still fail too often to run unwatched. Extra iteration helps but does not close the gap; Anthropic reported in 2024 that allowing Claude 3.5 Sonnet more steps per task raised its OSWorld score to 22.0%, still far below human performance [3].

Why not just script the UI with traditional automation tools? Scripted UI automation encodes exact selectors and coordinates, so it breaks on any interface change and must be maintained per application. A computer-use agent perceives the interface each time, which trades speed for resilience and removes most of the maintenance burden.

Sources

1. Anthropic. "Claude 3.5 Sonnet scored 14.9% on OSWorld's screenshot-only category, nearly double the next-best system's 7.8%." https://www.anthropic.com/news/3-5-models-and-computer-use. Accessed August 2026.
2. OSWorld (arXiv). "Humans succeed at over 72.36% of OSWorld tasks; the best AI model at launch achieved 12.24%." https://arxiv.org/abs/2404.07972. Accessed August 2026.
3. Anthropic. "With more steps allowed, Claude 3.5 Sonnet's OSWorld score rose to 22.0%." https://www.anthropic.com/news/3-5-models-and-computer-use. Accessed August 2026.