Computer Use

computer use · browser agent · browser use · computer control · gui agent · screen control agent

Definition

Computer use is an AI agent's ability to drive a computer interface — cursor, clicks, forms — as a human does, which allows it to buy on websites that expose no dedicated API or protocol.

What computer use is

Classic AI agents interact with services through APIs or dedicated protocols. Computer use takes a
fundamentally different route: the agent receives a screenshot of the screen, interprets it as a
visual scene and drives the pointer and keyboard exactly as a person would.

From the site’s point of view, such an agent is indistinguishable from a real visitor in a browser —
it opens pages, scrolls them, clicks buttons, fills in forms and walks through the ordinary checkout
process.

How it works technically

Screen capture — the agent receives a screenshot of the browser’s current state.
Interface recognition — a multimodal model identifies the elements: buttons, fields, text,
prices.
Action planning — the agent decides what to do next to reach the goal (add to basket, go to
payment).
Execution — it issues a command: click(x, y), type(“John”), scroll(down, 300px).
Verification — the next screenshot confirms that the action landed correctly.

Important: every step needs its own call to the model. A typical purchase of 10–15 steps takes
considerably longer than a transaction through an API.

Compared with protocols

Parameter	Computer Use	Protocols (ACP / YCP)
Requires integration by the store	No	Yes
Reliability	Medium	High
Transaction speed	Seconds to minutes	Milliseconds
Works with CAPTCHA	Limited	N/A
Supports 3-D Secure and bank authentication	Problematic	Through a payment mandate
A markup change breaks the agent	Yes	No

Its place in the agentic commerce ecosystem

Computer use is a transitional mechanism. It lets agents work with the existing web without any
preparation on the retailer’s side. As the industry adopts standards — ACP, MCP, YCP — the need for
browser agents will fall.

For a retailer that means something specific: stores that have not implemented agent protocols can
still end up inside AI agent purchase flows, just with lower reliability and no control over the
process.

Barriers and limitations

Anti-bot protection — Cloudflare, reCAPTCHA and their peers block automated access.
3-D Secure and bank verification — require actions a browser agent cannot perform.
Fragility — a page redesign breaks the flow; there is no markup standard for agents.
Terms of service — automated access is often prohibited outright.
Cost — the load on AI infrastructure is far higher than for API calls.

Frequently asked questions

How does computer use differ from a normal API integration?

An API integration is structured: the agent calls defined methods and receives data in a machine-readable format. Computer use works with the visual interface — the agent sees a screenshot and decides where to click. That is slower and more fragile, because markup changes, but it works on any site with no preparation.

How reliable is computer use for real purchases?

Less reliable than protocols such as ACP or YCP. Any change to a site's markup can break the flow. On complex checkout pages with CAPTCHAs or 3-D Secure authentication, the agent often meets a barrier it cannot pass.

Are there legal limits on using computer use in e-commerce?

Many sites forbid automated access in their terms of service. On top of that, paying through a browser agent raises verification and fraud questions for the payment networks. Protocols such as ACP and the payment mandate were designed partly to answer those problems.

Why bother with computer use at all if protocols exist?

Protocols require integration work from the merchant, which costs time and resources. Computer use lets an agent work with any existing site today, without waiting for the store to ship an API. It is a transitional mechanism.