Computer use is the ability of an AI agent to operate software through its user interface. The agent observes a page or desktop app, chooses an action, and checks the result. Tools such as Vercel's agent-browser and agent-desktop connect that reasoning to working software.
For example, an agent asked to retrieve a monthly report could open a supplier portal, select the reporting period, and download the file. The application might expose no API for that report. Computer use gives the agent another way to complete the task through the interface already available to a person.
Copy link to headingHow does computer use work?
Computer use repeats a cycle of observation, action, and verification. The model chooses what to attempt, and an execution tool performs the action and returns evidence for the next decision.
Suppose the task is to export October's orders. The agent first needs to establish which account is open and whether it has reached the orders page. It then selects the date range and requests an export. Completion requires more evidence than a successful click: the downloaded file should contain the requested period and belong to the intended account.
The observations can take different forms:
An element reference identifies a target from an observation. It isn't a permanent identifier for the application's underlying business object. If the interface changes, the agent may need to observe it again before acting. agent-browser's snapshot workflow follows this pattern explicitly.
Verification also needs to match the task. Seeing a new address in a text field confirms that the text was entered, but it doesn't establish that the application saved it. The agent should look for the saved record or another independent confirmation before reporting completion.
Copy link to headingDoes computer use always mean looking at screenshots?
Computer use can work through structured UI information, screenshots, or both. Screenshots help a model interpret the visible screen, but many applications expose controls in a form that tools can address directly.
For web pages, agent-browser returns text snapshots with references for later commands. For native applications, agent-desktop returns the operating system's accessibility tree as JSON. Button names and supported actions give the agent more explicit targets than coordinates alone.
Screenshot-based systems are useful when the task depends on appearance. To check whether a checkout panel overlaps its submit button, a coding agent needs to inspect the rendered result. The structured observation can identify both elements without showing that one obscures the other.
The AI SDK Computer Use template demonstrates a visual approach with Claude, a remote desktop in Vercel Sandbox, and a Next.js interface. Its desktop includes Chrome and software for screenshots and input. Those components supply the environment in which the model's requested actions run.
Copy link to headingHow is computer use different from API tools and scripts?
Computer use describes how an agent interacts with an application. Tool calling describes how a model requests an operation from your code. The agent can request a computer-use action through a tool call.
Use an API when it offers the operation and evidence you need. It can give you a defined request format, structured errors, and a record ID to verify. With a graphical interface, the agent must also track the active page and any open dialogs.
Mixed workflows can use both approaches. An agent can look up an order through an API, open the application's UI to reproduce the customer's problem, and use a separate tool to attach its findings to a ticket.
The Model Context Protocol, or MCP, is another separate concept. It is a way to connect an application to tools and resources. An MCP integration can expose browser controls, but it can also expose database queries or other operations that never touch a UI.
Copy link to headingWhat are useful computer-use tasks for agents?
Copy link to headingCheck a code change in a real application
Unit tests alone don't establish that a user can complete a checkout. An agent can open a preview deployment, follow the relevant flow, and collect evidence of what happened. Give it a test account and connect the preview to test services so it can reach the confirmation page without placing a real order.
In the eve software-factory browser integration, an agent uses a hosted browser to reproduce bugs and verify preview deployments. Its browser sessions are attended, with a person able to watch or take over. Findings can guide later unattended runs.
Copy link to headingRetrieve information from a portal without an export API
Suppose a supplier offers invoices through a dashboard while exposing only catalog data through its API. The browser agent can select an account and reporting period, then collect the available documents. The useful output is a verified set of files with their source and period, so missing months or duplicate invoices remain visible.
In an authenticated browser-agent workflow, eve uses Vercel Connect to access KERNEL. The user signs in through KERNEL's hosted login, which saves a browser session the agent can use without receiving the password.
Copy link to headingWork in a native desktop application
An operations agent could turn an approved service request into a draft work order in a native desktop app. It could enter the customer details and requested work, then leave the form for review. agent-desktop needs the app to expose the required controls and properties through its accessibility tree.
The review should show the resulting document or form. Click transcripts are useful for debugging, but they don't tell the reviewer whether the agent entered the correct customer number or preserved the document's formatting.
Copy link to headingInvestigate a visual or account-specific failure
Some support issues appear only after signing in as a particular test role or opening a specific dialog. An agent can investigate the interface and return reproduction steps alongside screenshots. This is useful when the initial report doesn't identify which action causes the failure.
Keep reproduction separate from changes to production data. Test order submissions and messaging in a test environment, or obtain authorization first.
Copy link to headingWhat does a browser interaction look like in code?
The following commands demonstrate the tool operations an agent can choose. Install agent-browser and its browser dependency in a development environment:
On Linux, agent-browser install --with-deps also installs the browser's system libraries. Then open a public page and inspect its interactive elements:
Read the returned snapshot. After confirming that it contains the link named "Learn more", follow that link and inspect the destination:
The named session keeps these commands attached to the same browser. The semantic locator identifies the link by its role and accessible name. An agent could instead choose an element reference from the snapshot, but it must use the reference returned by its own run.
This sequence contains no model call. In an agent application, your model receives the observation and requests the next tool operation. The AI SDK provides model and tool execution capabilities for building that application; the browser tool performs the UI operations.
Copy link to headingWhere does computer use need additional controls?
Computer use inherits the permissions of the session it operates. An authenticated browser can expose private information and allow consequential changes. Restrict the account, environment, and available actions to the task, and define in advance which operations require human approval.
Content inside a page or document is also untrusted input. Instructions in that content may conflict with the user's request. Your application should keep that content separate from its own instructions and enforce access restrictions outside the model. agent-browser offers optional navigation and action controls, which you must configure for your workload.
Applications can also change while the agent is working. Dialogs may appear, other people may edit records, and sessions may expire. The agent needs a way to stop and report uncertainty. For tasks where a mistake would be difficult to reverse, completing a draft for human review can be the appropriate end state.
Copy link to headingFrequently asked questions
Copy link to headingWhat does computer use mean for an AI agent?
Computer use means an agent can inspect and operate software through its interface to carry out a task. The agent application supplies browser or desktop tools, and the agent uses their results to decide what to do next.
Copy link to headingDo computer-use agents need screenshots?
No. Computer-use agents can inspect structured descriptions of web elements or native application controls. Screenshots provide visual evidence for tasks such as checking whether a panel obscures a button.
Copy link to headingDoes computer use replace APIs?
No. An authorized API is often the better route for an operation with a defined contract and a structured result. Computer use adds access to workflows that depend on the interface, and an agent can combine both approaches.
Copy link to headingAre agent-browser and agent-desktop complete AI agents?
Both projects provide tools that an agent invokes. Your agent application supplies the model, task instructions, and execution loop, then checks whether the tool results satisfy the user's request.