Use browser automation when an agent needs to operate a web page, and desktop automation when it needs native application controls. Vercel's agent-browser and agent-desktop support those respective paths. Check for an authorized API first, then choose the UI tool that can observe and verify the task.
The distinction follows the control you need to operate. An agent may move between web content and native menus or file pickers within the same workflow.
Copy link to headingWhich interface should an agent use?
Start with the required outcome and the evidence that would prove it happened. If the task is to download a report, you need the correct file. If it is to reproduce a rendering bug, an API response won't establish what appeared on screen.
Uploading a file does not always require desktop access. Browser tools may be able to set the web page's file input directly. Use native dialog automation when the task requires that dialog or the browser integration cannot perform the upload.
Keep tasks that only need web access in a browser session to limit the available interfaces. Its account may still permit consequential changes.
Copy link to headingWhat does agent-browser provide?
agent-browser is a CLI for operating a browser one command at a time. It can navigate pages, inspect their structure, and interact with controls. Its page snapshots give the agent text and element references instead of requiring it to process a full page's raw HTML for each decision.
The browser remains available between commands. Named sessions let you address separate browser instances, which is useful when two tasks need different cookies or accounts. Your application still needs to decide which user owns each session and when to close it.
For a preview check, the agent might open a test deployment, find a form by its accessible labels, submit synthetic data, and inspect the confirmation. It can also capture a screenshot to check appearance. Vercel's OpenCode example uses browser verification in a coding workflow.
To inspect a public Vercel page, install the CLI and browser, then run:
The snapshot identifies interactive elements, and the screenshot records the rendered page. Define what a correct result looks like for your application before asking the model to evaluate either observation.
Copy link to headingWhat does agent-desktop provide?
agent-desktop reads native accessibility information on macOS, Windows, and Linux. Its snapshots return JSON with controls and references the agent can use to request actions. The tool runs the requested operation, while the calling agent owns the observation and decision loop.
The operating system exposes a desktop application's accessibility tree to assistive software. This tree can describe text fields, buttons, and menus, along with their current values. Coverage varies by application and platform, and custom-drawn controls may expose too little information for semantic automation.
agent-desktop acts on references through accessibility APIs by default, without taking foreground focus or synthesizing physical input. This headless mode still depends on access to the desktop environment. Physical input for these actions requires headed mode and the appropriate platform support.
Use Node.js 18 or later to install the tool and check the environment:
On macOS, grant Accessibility permission through the documented setup. On Linux, run the tool inside the user's graphical session with the accessibility bus available. Windows actions are constrained by the process's integrity level. The platform installation requirements explain these differences.
With GNOME Calculator open on a supported Linux desktop, inspect its controls:
Find the button named "7" in that output. Copy its full ref_id, including the snapshot prefix, into the following command. The sample reference is illustrative and must be replaced:
Inspect the resulting display before taking another action. The calculator can update asynchronously, so wait for the expected value with a timeout rather than treating an immediate snapshot as final. Repeating the click before confirming its effect could enter 77.
Copy link to headingWhy do the two tools use different references?
Browser references such as @e1 and desktop references such as @s8f3k2p9:e11 belong to different tools and observation models. You cannot pass one to the other or copy a tutorial's reference into an unrelated run.
Refresh agent-browser's snapshot after navigation or meaningful page changes before reusing references. agent-desktop uses references with a snapshot ID and checks the saved target identity against the live control before acting.
If a target is ambiguous or stale, observe it again and select a target the tool can distinguish. Check the action's delivery and retry status before repeating it. When delivery is uncertain, inspect the application's state before deciding what to do next.
Don't generate a long list of future clicks from the initial screen and assume every target will keep the same meaning. Save dialogs, validation messages, and updated tables can change what the next action needs to be.
Copy link to headingWhen should an agent use screenshots?
Use screenshots when the task depends on visual information that structured observations don't capture. Examples include an obscured button, a broken chart, or an incorrect arrangement of elements at a particular window size.
For actions, prefer a meaningful element target when the tool exposes one. The control's name and role can help distinguish it from nearby controls. Coordinate-based input depends on the current viewport and screen geometry, so take a fresh observation after a window moves or the interface changes.
Screenshot-driven computer use also needs an execution environment that can capture images and deliver input. Vercel's AI SDK Computer Use template supplies a remote desktop in Sandbox for that purpose. It is a separate implementation from agent-desktop's native accessibility approach.
Screenshots show the interface at one moment. For a saved file, an exported report, or a submitted record, also verify the resulting artifact or record.
Copy link to headingHow should an agent handle tasks that cross applications?
Consider a report that must move from a supplier portal into a native spreadsheet application. Use browser automation to select and download the report. Check the file's period and account before handing it to the desktop step, then inspect the opened document before editing it.
The handoff needs an explicit artifact, such as a file path and a recorded validation result. Passing only a message saying "the report is ready" leaves the next step to guess which download to open. If the tools run on separate machines, your application must transfer the file through an authorized storage path.
Electron apps combine web technology with desktop packaging, so choose the tool for the control you need to operate. agent-desktop can inspect the app through accessibility APIs even when no browser connection is available.
For a compatible app you can launch fresh, agent-desktop can open and verify a loopback DevTools port. Connect agent-browser to that port to operate web content, and use agent-desktop for native menus and dialogs. Using web technology does not by itself mean an app exposes a usable browser endpoint.
For development workflows on Vercel, you can add a hosted browser to an existing agent. The example uses attended sessions so a person can resolve login or account questions before returning control to the agent.
Copy link to headingWhat do browser and desktop tools leave to your application?
Neither tool decides what a user has authorized. Saved logins can permit operations beyond the current request, so your application needs to constrain actions and tie approval to the specific proposed change.
The tools also don't supply a complete agent framework. The AI SDK provides model execution and tool approvals that you can combine with browser or desktop operations. Those capabilities serve different responsibilities from identifying a button or reading a window.
Hosted execution depends on the environment the tool needs. Browsers can run in a suitable worker or through a remote provider. agent-desktop requires a compatible operating system, the target application, and graphical and accessibility services. Check that your runtime provides those services before choosing it.
Choose a small representative task before expanding access. If the tool can't observe the control or distinguish completion from failure, adding more instructions won't repair the missing evidence. Use another integration, improve the application's accessibility, or keep that step with a person.
Copy link to headingFrequently asked questions
Copy link to headingCan agent-desktop replace agent-browser for websites?
Use agent-browser for the web page's content and controls. agent-desktop is suited to native application elements, which can include browser menus and operating system dialogs around that page.
Copy link to headingCan agent-browser operate a native desktop application?
No. Browser automation operates through a supported browser connection. Native apps need a desktop integration, although Electron apps may also expose web content through a supported DevTools connection.
Copy link to headingWhat should an agent do when a reference stops working?
Observe the affected page or application again and select the target from the new result. Before retrying a consequential action, check whether the previous attempt already changed the application.
Copy link to headingDoes headless desktop automation work without a desktop session?
Headless accessibility actions avoid taking focus or sending physical input, but they still need the platform's desktop services. On Linux, agent-desktop requires access to the user's graphical session and accessibility bus.