Skip to content
Docs

OpenAI Responses Images and PDFs with AI Gateway

Vision-capable models accept images and PDFs alongside your prompt. Replace the plain string in input with an array of content parts: input_text for the prompt, input_image for an image, input_file for a document.

image-input.ts
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const response = await client.responses.create({
  model: 'anthropic/claude-opus-5',
  input: [
    {
      role: 'user',
      content: [
        { type: 'input_text', text: 'Describe this image in one sentence.' },
        {
          type: 'input_image',
          image_url:
            'https://assets.vercel.com/image/upload/v1662130559/nextjs/Icon_light_background.png',
          detail: 'auto',
        },
      ],
    },
  ],
});
 
console.log(response.output_text);
image_input.py
import os
from openai import OpenAI
 
client = OpenAI(
    api_key=os.getenv('AI_GATEWAY_API_KEY'),
    base_url='https://ai-gateway.vercel.sh/v1',
)
 
response = client.responses.create(
    model='anthropic/claude-opus-5',
    input=[
        {
            'role': 'user',
            'content': [
                {'type': 'input_text', 'text': 'Describe this image in one sentence.'},
                {
                    'type': 'input_image',
                    'image_url': 'https://assets.vercel.com/image/upload/v1662130559/nextjs/Icon_light_background.png',
                    'detail': 'auto',
                },
            ],
        },
    ],
)
 
print(response.output_text)
image-input.sh
curl https://ai-gateway.vercel.sh/v1/responses \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "input": [
      {
        "role": "user",
        "content": [
          { "type": "input_text", "text": "Describe this image in one sentence." },
          {
            "type": "input_image",
            "image_url": "https://assets.vercel.com/image/upload/v1662130559/nextjs/Icon_light_background.png",
            "detail": "auto"
          }
        ]
      }
    ]
  }'

Base64-encoded images

For images that are not publicly reachable, send a data URI instead of a URL. The gateway forwards it to the provider, so no fetch happens from the provider's side:

image-base64.ts
import fs from 'node:fs';
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const imageBase64 = fs.readFileSync('./diagram.png').toString('base64');
 
const response = await client.responses.create({
  model: 'anthropic/claude-opus-5',
  input: [
    {
      role: 'user',
      content: [
        { type: 'input_text', text: 'What does this diagram show?' },
        {
          type: 'input_image',
          image_url: `data:image/png;base64,${imageBase64}`,
          detail: 'auto',
        },
      ],
    },
  ],
});
console.log(response.output_text);

A URL source must be reachable without authentication. If the host blocks the request, the gateway returns a 400 naming the upstream status rather than falling back, so use base64 for anything behind a login or a signed URL.

PDF documents

Send a PDF with an input_file part. The model reads the document's text directly rather than working from a rasterized page:

pdf-input.ts
import fs from 'node:fs';
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const pdfBase64 = fs.readFileSync('./report.pdf').toString('base64');
 
const response = await client.responses.create({
  model: 'anthropic/claude-opus-5',
  input: [
    {
      role: 'user',
      content: [
        { type: 'input_text', text: 'What total does this document state?' },
        {
          type: 'input_file',
          filename: 'report.pdf',
          file_data: `data:application/pdf;base64,${pdfBase64}`,
        },
      ],
    },
  ],
});
 
console.log(response.output_text);
pdf_input.py
import base64
import os
from openai import OpenAI
 
client = OpenAI(
    api_key=os.getenv('AI_GATEWAY_API_KEY'),
    base_url='https://ai-gateway.vercel.sh/v1',
)
 
with open('report.pdf', 'rb') as f:
    pdf_base64 = base64.b64encode(f.read()).decode()
 
response = client.responses.create(
    model='anthropic/claude-opus-5',
    input=[
        {
            'role': 'user',
            'content': [
                {'type': 'input_text', 'text': 'What total does this document state?'},
                {
                    'type': 'input_file',
                    'filename': 'report.pdf',
                    'file_data': f'data:application/pdf;base64,{pdf_base64}',
                },
            ],
        },
    ],
)
 
print(response.output_text)
pdf-input.sh
PDF_B64=$(base64 -i report.pdf)
 
curl https://ai-gateway.vercel.sh/v1/responses \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "input": [
      {
        "role": "user",
        "content": [
          { "type": "input_text", "text": "What total does this document state?" },
          {
            "type": "input_file",
            "filename": "report.pdf",
            "file_data": "data:application/pdf;base64,'"$PDF_B64"'"
          }
        ]
      }
    ]
  }'

Detail parameter

detail controls the resolution the model analyzes the image at:

  • auto - Let the model decide
  • low - Lower resolution, fewer input tokens, faster
  • high - Higher resolution, more input tokens, better for small text and fine detail

Next steps

Last updated September 8, 2026
Previous
Reasoning
Next
Compaction

Was this helpful?

supported.