AI SDK Computer Use

An open-source AI chatbot demonstrating computer use capabilities with Anthropic Claude Sonnet 4.5, Vercel Sandboxes, and the AI SDK by Vercel.

Features · How It Works · Deploy Your Own · Running Locally

Features

• Streaming text responses powered by the AI SDK.
• Anthropic Claude Sonnet 4.5 with computer use and bash tool capabilities.
• Remote desktop environment running in a Vercel Sandbox with Chrome, a window manager, and VNC streaming.
• shadcn/ui components for a modern, responsive UI powered by Tailwind CSS.
• Built with the latest Next.js App Router.

How It Works

The app spins up a Vercel Sandbox from a pre-built snapshot that includes:

• Xvnc — a virtual X11 display server
• openbox — a lightweight window manager
• noVNC + websockify — streams the desktop to the browser via WebSocket
• Google Chrome — auto-launched so the AI agent has a browser ready
• xdotool + ImageMagick — for mouse/keyboard control and screenshots

When a user sends a message, Claude uses the computer tool (screenshot, click, type, scroll) and the bash tool (run shell commands) to interact with the sandbox desktop. The noVNC stream is displayed in a resizable iframe alongside the chat.

Architecture

1
User ↔ Next.js Chat UI ↔ AI SDK ↔ Claude Sonnet 4.5
2
                                        ↓
3
                                  Vercel Sandbox
4
                              ┌─────────────────────┐
5
                              │  Xvnc (:99)         │
6
                              │  openbox             │
7
                              │  Chrome              │
8
                              │  websockify → noVNC  │
9
                              └─────────────────────┘
10
                                        ↓
11
                              noVNC iframe in browser

Deploy Your Own

You can deploy your own version to Vercel by clicking the button below:

Running Locally

Prerequisites

• Node.js 18+
• A Vercel account (for Sandbox access)
• An Anthropic API key

1. Install dependencies

1
pnpm install

2. Set up Vercel credentials

Install the Vercel CLI and link your project:

1
pnpm install -g vercel
2
vercel link
3
vercel env pull

This creates a .env.local file with VERCEL_OIDC_TOKEN for Sandbox authentication.

Alternatively, set VERCEL_TOKEN, VERCEL_TEAM_ID, and VERCEL_PROJECT_ID manually in your .env.local.

3. Create a sandbox snapshot

The snapshot pre-installs the desktop environment (Xvnc, Chrome, openbox, noVNC, xdotool, ImageMagick) so sandboxes boot in seconds.

1
npx tsx lib/sandbox/create-snapshot.ts

This takes ~10 minutes. When done, it outputs a snapshot ID. Add it to your .env.local:

1
SANDBOX_SNAPSHOT_ID=snap_xxxxxxxxxxxxx

4. Add your Anthropic API key

1
ANTHROPIC_API_KEY=sk-ant-...

5. Start the dev server

1
pnpm dev

Open http://localhost:3000 to use the computer use agent.

Environment Variables

* Either VERCEL_OIDC_TOKEN (via vercel env pull) or the VERCEL_TOKEN + team/project IDs are required for Sandbox authentication.