AI Gateway Model Modalities

Modalities are the kinds of input and output models work with. Through AI Gateway you can generate text, images, and video, transcribe and synthesize speech, hold realtime voice conversations, create embeddings, rerank documents, and answer typed questions about state, all through a unified API so you don't need separate integrations for each provider. For cross-cutting abilities like reasoning and web search, see Models & Providers.

For image understanding, documents, recorded audio, video analysis, and function calling, see Inputs & Tools.

• Text apps: Generate and stream text for chat, content, and agents with Text Generation
• Visual content apps: Generate product images, marketing assets, or UI mockups with Image Generation
• Video content: Create videos from text prompts, images, or video input with Video Generation
• Voice agents: Build low-latency, two-way voice conversations with Realtime
• Transcription: Turn recorded audio into text with Speech to Text
• Spoken audio: Generate voiceovers and spoken responses from text with Text to Speech
• Semantic search: Generate vector embeddings for search, similarity matching, and RAG with Embeddings
• Improved retrieval: Re-score candidate documents by relevance for better RAG results with Reranking
• Structured decisions: Answer typed questions about shared state with Decision

Generate and stream text from hundreds of models through a single API. Text is the default modality, with support for tool calling and structured output.

import { generateText } from 'ai';
 
const { text } = await generateText({
  model: 'openai/gpt-6-astra',
  prompt: 'What is the capital of France?',
});

See the Text Generation docs for streaming, tool calling, and structured output.

Generate images using AI models through a single API. Requests route to the best available provider, with authentication and response formatting handled automatically.

import { gateway } from '@ai-sdk/gateway';
import { generateImage } from 'ai';
 
const { image } = await generateImage({
  model: gateway.imageModel('openai/gpt-image-2'),
  prompt: 'A serene mountain landscape at sunset',
});

Supported providers include OpenAI (GPT Image), Google (Nano Banana), and multimodal LLMs with image capabilities. See the Image Generation docs for implementation details.

Generate videos from text prompts, images, or video input using AI models through a single API. Control resolution, duration, aspect ratio, and audio generation across providers.

import { experimental_generateVideo as generateVideo } from 'ai';
 
const { videos } = await generateVideo({
  model: 'google/veo-3.1-generate-001',
  prompt: 'A serene mountain landscape at sunset with clouds drifting by',
  aspectRatio: '16:9',
  resolution: '1920x1080',
  duration: 8,
});

Supported providers include Google (Veo 3.1), KlingAI (motion control), and Wan. See the Video Generation docs for implementation details.

AI Gateway supports audio understanding, transcription, speech generation, and realtime voice. Use this table to pick the one that matches what you have and what you want to produce:

Speech to Text and Text to Speech process recorded or generated audio in a single request. Realtime handles live conversation, so reach for it when you need a back-and-forth voice session rather than a transcript or a finished audio file.

Generate vector embeddings for semantic search, similarity matching, and RAG, then re-score candidate documents by relevance to improve retrieval accuracy. See the Embeddings docs and Reranking docs for implementation details.

Ask typed questions about shared state and get structured answers back: boolean probabilities, a choice from a named set, or a score along an ordered scale. Many questions can be answered in one request. See the Decision docs for implementation details.

• Generate text for chat, content, and agents
• Generate your first image
• Generate your first video
• Build a voice agent with Realtime
• Transcribe audio with Speech to Text
• Generate speech with Text to Speech
• Generate embeddings for semantic search and RAG
• Rerank documents to improve retrieval accuracy
• Make decisions to get structured answers to typed questions

For cross-cutting abilities, see Models & Providers. For monitoring and spend, see Observability and Spend. For data privacy and governance, see Security and Compliance.