AI Gateway Video Input
Send a video clip and a question to a video-capable model to summarize events or inspect a sequence of actions. To create or edit a video, see video generation.
Set AI_GATEWAY_API_KEY, install your SDK or API client, and save a short MP4 clip as boat.mp4 in your working directory. These examples use google/gemini-3.6-flash and request text output.
Video support, sampling, duration limits, and audio-track handling depend on the model and provider. Start with a short clip and check the selected model's limits before sending a longer recording.
These examples send the MP4 bytes and request a description of the clip. The SDK examples print the description; cURL prints the JSON response containing it:
import { readFileSync } from 'node:fs';
import { generateText } from 'ai';
const { text } = await generateText({
model: 'google/gemini-3.6-flash',
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Describe what happens in this video in one sentence.',
},
{
type: 'file',
data: readFileSync('boat.mp4'),
mediaType: 'video/mp4',
},
],
},
],
});
console.log(text);import asyncio
from pathlib import Path
import ai
async def main():
model = ai.get_model("google/gemini-3.6-flash")
messages = [
ai.user_message(
"Describe what happens in this video in one sentence.",
ai.file_part(Path("boat.mp4").read_bytes(), media_type="video/mp4"),
)
]
async with ai.stream(model, messages) as stream:
async for event in stream:
if isinstance(event, ai.events.TextDelta):
print(event.chunk, end="", flush=True)
print()
asyncio.run(main())import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env.AI_GATEWAY_API_KEY,
baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('boat.mp4').toString('base64');
const response = await client.chat.completions.create({
model: 'google/gemini-3.6-flash',
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Describe what happens in this video in one sentence.',
},
{
type: 'file',
file: {
filename: 'boat.mp4',
file_data: `data:video/mp4;base64,${data}`,
},
},
],
},
],
});
console.log(response.choices[0]?.message.content);import base64
import os
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_API_KEY"],
base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("boat.mp4").read_bytes()).decode("ascii")
response = client.chat.completions.create(
model="google/gemini-3.6-flash",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what happens in this video in one sentence.",
},
{
"type": "file",
"file": {
"filename": "boat.mp4",
"file_data": f"data:video/mp4;base64,{data}",
},
},
],
}
],
)
print(response.choices[0].message.content)FILE_BASE64=$(base64 < boat.mp4 | tr -d '\n')
printf '{
"model": "google/gemini-3.6-flash",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what happens in this video in one sentence."
},
{
"type": "file",
"file": {
"filename": "boat.mp4",
"file_data": "data:video/mp4;base64,%s"
}
}
]
}
]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/chat/completions \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @-import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env.AI_GATEWAY_API_KEY,
baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('boat.mp4').toString('base64');
const response = await client.responses.create({
model: 'google/gemini-3.6-flash',
input: [
{
role: 'user',
content: [
{
type: 'input_text',
text: 'Describe what happens in this video in one sentence.',
},
{
type: 'input_file',
filename: 'boat.mp4',
file_data: `data:video/mp4;base64,${data}`,
},
],
},
],
});
console.log(response.output_text);import base64
import os
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_API_KEY"],
base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("boat.mp4").read_bytes()).decode("ascii")
response = client.responses.create(
model="google/gemini-3.6-flash",
input=[
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Describe what happens in this video in one sentence.",
},
{
"type": "input_file",
"filename": "boat.mp4",
"file_data": f"data:video/mp4;base64,{data}",
},
],
}
],
)
print(response.output_text)FILE_BASE64=$(base64 < boat.mp4 | tr -d '\n')
printf '{
"model": "google/gemini-3.6-flash",
"input": [
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Describe what happens in this video in one sentence."
},
{
"type": "input_file",
"filename": "boat.mp4",
"file_data": "data:video/mp4;base64,%s"
}
]
}
]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/responses \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @-See the AI SDK file-part guide and Python beta multimodal-message guide for constructing SDK messages. The table below compares those messages with the HTTP formats.
| Format | Input representation | Details |
|---|---|---|
| AI SDK 7 | file part with mediaType: 'video/mp4' | Pass bytes, a data URL, or a supported URL. See file parts. |
| AI SDK for Python (beta) | ai.file_part(..., media_type="video/mp4") | See Python file parts. |
| Chat Completions | file with filename and a file_data data URL | AI Gateway maps the file to the selected provider. See file requests. |
| Messages API | No video content part | Use another format shown here. For an image-only model, extract selected frames and send them with vision; frame images do not preserve audio or motion. |
| Responses / OpenResponses | input_file with filename and a file_data data URL; OpenResponses also accepts input_video with video_url | For non-OpenAI models, AI Gateway translates the request through OpenResponses. Native Responses models have their own input support. |
These examples use AI Gateway's compatible endpoints with a Gemini model. They do not imply that the same file works with every OpenAI model or the direct OpenAI API. See API format differences before changing models or endpoints.
Inline base64 increases the request size. Check the selected model's context and file-size limits, and keep any fallback models compatible with the input type. When using a hosted URL, make sure the provider can retrieve it without your application's browser session.
For request details and failures, inspect AI Gateway logs. See Inputs & Tools to choose another capability.
Was this helpful?