Build realtime voice agents on AI Gateway

AI Gateway now supports audio/voice. You can add realtime voice, text to speech, and speech to text with the same calls you already use for text, image, and video, routed through AI Gateway alongside every other modality.

Audio launches with models from OpenAI and xAI. Each call gets the same provider routing, observability, spend controls, and bring-your-own-key support you already use for your other models.

These capabilities are in beta and available in AI SDK 7.

Link to headingGetting started

Realtime, speech, and transcription model are supported on AI SDK 7.

npm install ai @ai-sdk/react @ai-sdk/gateway

Link to headingRealtime voice agents

Realtime turns your app into something a user can hold a conversation with. When they speak, the model responds right away. Because it replies in the moment instead of waiting for a full turn, users can interrupt and talk over it the way they would with a person. It fits voice assistants, customer support agents, hands-free tools, and anywhere a user would rather talk than type.

What sets it apart from chaining models together is that a single realtime model hears audio and produces audio directly, instead of running a speech-to-text, then language model, then text-to-speech pipeline.

In the browser, the useRealtime hook manages the WebSocket connection, microphone capture, and audio playback.

The connection is authenticated with your AI Gateway credential, so you mint a short-lived token on the server and hand the browser only that token. Your API key never reaches the client. Add a route that mints the token:

1
import { gateway } from '@ai-sdk/gateway';
2

3
export async function POST() {
4
  const { token, url } = await gateway.experimental_realtime.getToken({
5
    model: 'openai/gpt-realtime-2',
6
  });
7
  return Response.json({ token, url, tools: [] });
8
}

Then connect from a client component:

1
'use client';
2
import { experimental_useRealtime as useRealtime } from '@ai-sdk/react';
3
import { gateway } from '@ai-sdk/gateway';
4
import { useMemo } from 'react';
5

6
export default function Page() {
7
  const model = useMemo(
8
    () => gateway.experimental_realtime('openai/gpt-realtime-2'),
9
    [],
10
  );
11
  const { status, connect, startAudioCapture } = useRealtime({
12
    model,
13
    api: { token: '/api/realtime/token' },
14
    sessionConfig: { voice: 'alloy', turnDetection: { type: 'server-vad' } },
15
  });
16
  // Call connect(), then startAudioCapture(stream) with a microphone MediaStream.
17
}

The hook captures the microphone, streams the audio to the model through AI Gateway, and plays back the spoken reply. Outside the browser, you can drive the session over a WebSocket yourself with getWebSocketConfig, serializeClientEvent, and parseServerEvent. See the realtime reference for that path.

Link to headingInside a realtime session

A realtime session works differently from a normal model call:

• Turn-taking and interruptions. turnDetection: { type: 'server-vad' } lets the server decide when the user has stopped speaking, and lets the user talk over the model to cut a reply short (barge-in), with no client-side silence timers.

• Tools mid-conversation. The model emits a tool call mid-reply, you run it and return the result as a client event, and it folds the answer into what it says next without ending the turn.

Link to headingText to speech

Generate spoken audio from text with generateSpeech. Pass a voice and an output format, then write the result to a file:

1
import { generateSpeech } from 'ai';
2
import { writeFile } from 'node:fs/promises';
3

4
const result = await generateSpeech({
5
  model: 'xai/grok-tts',
6
  text: 'Thanks for trying out AI Gateway.',
7
  voice: 'eve',
8
  outputFormat: 'mp3',
9
});
10
await writeFile('speech.mp3', result.audio.uint8Array);

Link to headingSpeech to text (transcription)

Transcribe recordings into text with transcribe. The audio can be a buffer, a base64 string, or a URL:

1
import { transcribe } from 'ai';
2
import { readFile } from 'node:fs/promises';
3

4
const result = await transcribe({
5
  model: 'openai/whisper-1',
6
  audio: await readFile('audio.mp3'),
7
});
8
console.log(result.text);

Speech and transcription are complementary, so they compose. You can generate audio with one model and read it back with the other, which is a quick way to check both ends of an audio pipeline.

Link to headingPlayground

You can also try audio models without writing any code. Open the models page, click into a model, and interact with it right in the browser. Talk to a realtime model to hold a voice conversation, or send text or audio to a speech or transcription model and read or play back the result.

Link to headingRouting audio through AI Gateway

Audio calls behave like every other model call on AI Gateway. You use one API key across providers, see requests and usage in observability, apply the same budgets and spend limits, and bring your own provider keys when you need to. Adding speech to an app that already uses AI Gateway for text/images/videos can now all be done in the same place.

Link to headingMore information

• Realtime quickstart

• Speech quickstart (text to speech and speech to text)

• Realtime reference

• See all audio models