Skip to content
Docs

AI Gateway Vision and Image Input

Send an image and a question to a vision-capable model to describe a scene, read a chart, or inspect a screenshot. To create or edit images, see image generation.

Before you start

Set AI_GATEWAY_API_KEY and install your SDK or API client. These examples use AI SDK 7 and the independent Python beta. Save a PNG as image.png in your working directory.

Choose a model that supports image input. Check its accepted image types, resolution, and image-count limits before sending a request. A text-only fallback cannot process the same image message.

Analyze an image

These examples request a description of the local image. The SDK examples print the description; cURL prints the JSON response containing it:

vision.ts
import { readFileSync } from 'node:fs';
import { generateText } from 'ai';
 
const { text } = await generateText({
  model: 'anthropic/claude-sonnet-5',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe the image in one sentence.' },
        {
          type: 'file',
          data: readFileSync('image.png'),
          mediaType: 'image/png',
        },
      ],
    },
  ],
});
 
console.log(text);
vision_ai.py
import asyncio
from pathlib import Path
import ai
 
 
async def main():
    model = ai.get_model("anthropic/claude-sonnet-5")
    messages = [
        ai.user_message(
            "Describe the image in one sentence.",
            ai.file_part(Path("image.png").read_bytes(), media_type="image/png"),
        )
    ]
    async with ai.stream(model, messages) as stream:
        async for event in stream:
            if isinstance(event, ai.events.TextDelta):
                print(event.chunk, end="", flush=True)
    print()
 
 
asyncio.run(main())
vision-chat.ts
import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('image.png').toString('base64');
 
const response = await client.chat.completions.create({
  model: 'anthropic/claude-sonnet-5',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Describe the image in one sentence.',
        },
        {
          type: 'image_url',
          image_url: {
            url: `data:image/png;base64,${data}`,
          },
        },
      ],
    },
  ],
});
 
console.log(response.choices[0]?.message.content);
vision_chat.py
import base64
import os
from pathlib import Path
from openai import OpenAI
 
client = OpenAI(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
 
response = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the image in one sentence."},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{data}"},
                },
            ],
        }
    ],
)
 
print(response.choices[0].message.content)
vision-chat.sh
FILE_BASE64=$(base64 < image.png | tr -d '\n')
 
printf '{
  "model": "anthropic/claude-sonnet-5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe the image in one sentence."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,%s"
          }
        }
      ]
    }
  ]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/chat/completions \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @-
vision-messages.ts
import { readFileSync } from 'node:fs';
import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh',
});
const data = readFileSync('image.png').toString('base64');
 
const response = await client.messages.create({
  model: 'anthropic/claude-sonnet-5',
  max_tokens: 1024,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Describe the image in one sentence.',
        },
        {
          type: 'image',
          source: {
            type: 'base64',
            media_type: 'image/png',
            data: data,
          },
        },
      ],
    },
  ],
});
 
for (const block of response.content) {
  if (block.type === 'text') console.log(block.text);
}
vision_messages.py
import base64
import os
from pathlib import Path
from anthropic import Anthropic
 
client = Anthropic(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh",
)
data = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
 
response = client.messages.create(
    model="anthropic/claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the image in one sentence."},
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": data,
                    },
                },
            ],
        }
    ],
)
 
for block in response.content:
    if block.type == "text":
        print(block.text)
vision-messages.sh
FILE_BASE64=$(base64 < image.png | tr -d '\n')
 
printf '{
  "model": "anthropic/claude-sonnet-5",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe the image in one sentence."
        },
        {
          "type": "image",
          "source": {
            "type": "base64",
            "media_type": "image/png",
            "data": "%s"
          }
        }
      ]
    }
  ]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/messages \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  --data-binary @-
vision-responses.ts
import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('image.png').toString('base64');
 
const response = await client.responses.create({
  model: 'anthropic/claude-sonnet-5',
  input: [
    {
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Describe the image in one sentence.',
        },
        {
          type: 'input_image',
          detail: 'auto',
          image_url: `data:image/png;base64,${data}`,
        },
      ],
    },
  ],
});
 
console.log(response.output_text);
vision_responses.py
import base64
import os
from pathlib import Path
from openai import OpenAI
 
client = OpenAI(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
 
response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe the image in one sentence."},
                {
                    "type": "input_image",
                    "detail": "auto",
                    "image_url": f"data:image/png;base64,{data}",
                },
            ],
        }
    ],
)
 
print(response.output_text)
vision-responses.sh
FILE_BASE64=$(base64 < image.png | tr -d '\n')
 
printf '{
  "model": "anthropic/claude-sonnet-5",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Describe the image in one sentence."
        },
        {
          "type": "input_image",
          "detail": "auto",
          "image_url": "data:image/png;base64,%s"
        }
      ]
    }
  ]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/responses \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @-

API format differences

See the AI SDK file-part guide and Python beta multimodal-message guide for constructing SDK messages. The table below compares those messages with the HTTP formats.

FormatInput representationDetails
AI SDK 7file part with an image/* mediaTypeThe older image part is deprecated in SDK 7. See version compatibility.
AI SDK for Python (beta)ai.file_part(..., media_type="image/png")Use the independent Python beta SDK.
Chat Completionsimage_url content partSend an HTTPS URL or data URL. See image requests.
Messages APIimage content part with sourceThe base64 source carries an explicit media_type. See Messages images.
Responses / OpenResponsesinput_image with image_urlSee Responses images and OpenResponses images for detail controls.

Handling larger inputs

Inline base64 increases the request size. Check the selected model's context and file-size limits, and keep any fallback models compatible with the input type. When using a hosted URL, make sure the provider can retrieve it without your application's browser session.

For request details and failures, inspect AI Gateway logs. See Inputs & Tools to choose another capability.

Last updated September 8, 2026
Previous
Inputs & Tools
Next
File Input

Was this helpful?

supported.