AI Gateway Inputs and Tools
Choose a guide for the input you have or the function you want a model to call. Each guide includes examples for supported API formats and explains their differences.
| Capability | Use it to | Example input |
|---|---|---|
| Vision | Describe images, read charts, or inspect screenshots | PNG image |
| File Input | Summarize documents or extract information | PDF document |
| Tool Use | Connect a model to application functions | Function schema and tool results |
| Audio Input | Ask questions about recorded speech | MP3 recording |
| Video Input | Summarize events or inspect actions in a clip | MP4 video |
For SDK concepts, start with AI SDK message prompts and tool calling, or the Python beta messages and agents guides.
A model must support the input type and the selected provider must accept that request format. The model catalog describes model capabilities. Check model-specific limits as well as the format notes in each guide, including for any configured fallback models.
Examples group AI SDK 7 TypeScript and AI SDK for Python (beta) under one AI SDK tab. Chat Completions, Messages API, and Responses / OpenResponses follow, with TypeScript, Python, and cURL language switchers where supported. See SDKs and APIs for installation, authentication, and endpoint differences.
These guides focus on understanding existing input. For other tasks, see:
- Image generation to create or edit an image.
- Video generation to create or edit a video.
- Speech to text for a dedicated transcript.
- Text to speech to generate spoken audio.
- Realtime for a live voice conversation.
- Coding agents to connect an existing coding tool to AI Gateway.
Was this helpful?