Time to first token (TTFT) measures the wait between sending a request to a large language model (LLM) and receiving its first streamed token. It helps explain how responsive an AI feature feels. Some benchmarks count reasoning tokens, but users wait until response text appears on screen. A model can generate text faster than people read and still feel slow if the first word takes seconds to appear.
To reduce that wait, first measure where the time goes.
Copy link to headingWhat is TTFT in LLM inference?
Client-observed TTFT includes the time for the request to reach the server, wait for an inference slot, process the prompt, and return the first token. During prefill, the model processes prompt tokens and builds its key-value (KV) cache. This cache stores attention state that the model reuses during generation. Prompt caching can reuse some of that state across requests. Benchmarks such as vLLM's benchmark tool report TTFT separately from later generation time because the phases behave differently. Check when each timer starts and which token ends the measurement.
Copy link to headingWhat is the difference between TTFT and tokens per second?
TTFT measures the wait for the first token. Output tokens per second measures how fast later tokens arrive. Prefill contributes to TTFT, and decode generates output incrementally after it. Prefill is often compute-bound, while decode is often limited by memory bandwidth. An optimization for one phase may therefore have little effect on the other. Anthropic's fast mode focuses its speed improvements on output tokens per second rather than TTFT.
Each metric measures a different part of the response:
For chat, track the wait for visible text and how consistently the response streams. For voice, also measure the wait for audible speech. For long reports and batch jobs, track output tokens per second and end-to-end latency. A high-throughput model can still be a poor choice for chat if users wait too long for its response to begin.
Copy link to headingWhy does time to first token matter?
When streamed prose arrives faster than someone can read it, the initial wait can matter more to perceived responsiveness than further increases in output speed. Throughput still matters for code, long outputs, and downstream processing. Startup delay affects each use case differently:
Chat turns: Users may wait for visible text on each turn, so startup latency repeatedly affects the conversation.
Voice agents: In a cascaded voice agent, speech recognition, model response, text buffering, and speech synthesis all add to the wait before users hear a response.
Inline completions: A code suggestion becomes less useful if it arrives after the person has continued typing.
Agent tool loops: Sequential model calls each add startup latency. Those delays accumulate across a tool loop, especially when calls include long contexts.
Measure these stages separately to find which one needs attention.
Copy link to headingCore components of time to first token
A request travels over the network, may wait in the provider's queue, and goes through prefill. The model may then generate reasoning before response text. Returning and rendering that text adds to the wait users see. The model, provider, region, and serving configuration affect these stages.
Copy link to headingNetwork hops and region
Network latency is the time data spends traveling between the client, your server, any gateway, and the provider. A chat request sent through all four crosses three outbound network legs before reaching the model, then returns through them. Distance, connection setup, and network conditions affect transit time. Deploying a function far from the provider's inference region can add delay to every request. The same model can also respond at different speeds in different provider regions.
An LLM gateway adds a network hop, so measure the overhead it introduces. It also gives you one place to compare providers and control routing. Those routing choices can reduce total latency, which is the tradeoff to assess.
Copy link to headingQueueing and batching
Queueing is the time a request waits for the provider to schedule inference. With light traffic, prefill may account for much of TTFT. With heavier traffic, queueing can account for a larger share of the delay, while scheduling and prefill performance may also change.
Batching trades first-token speed against the pace of an active stream. With chunked prefill in vLLM, a higher max_num_batched_tokens budget can improve TTFT because each batch processes more prefill tokens. A smaller budget can improve inter-token latency because less prefill work delays ongoing decode work. Hosted model APIs typically do not expose this setting. Applications can still reduce concurrency, choose another serving tier where available, or use another provider.
Copy link to headingPrompt length and model size
Prompt length and model size affect prefill time and therefore the minimum time needed to return the first token. The model must process the required context or reuse that work from cache before it can produce output. Longer uncached prompts generally take longer to process. Under comparable serving conditions, models with more active parameters generally need more compute per prompt token.
Application code gives you direct control over prompt length. Retrieval-augmented generation (RAG) results, chat history, and tool schemas can increase the prompt size on every turn. Removing unnecessary content can shorten prefill, but removing useful context can reduce answer quality. Compare prompt sizes across turns, remove redundant history and unused tool definitions, and test answer quality before cutting retrieved material.
Copy link to headingReasoning tokens and visible output
A model generates reasoning tokens as it works through a problem. Those tokens can arrive before answer text, so a low TTFT may still leave users waiting. An interface that hides reasoning may not display any model-generated text until a preamble or an answer begins.
Benchmarks do not all end the measurement at the same event. If a model exposes reasoning, TTFT may count its first reasoning token. NVIDIA's AIPerf benchmarking tool reports TTFT for the first token of any type. It also reports time to first output token (TTFO) for the first non-reasoning output token. TTFO is closer to the wait users see when an interface hides reasoning. It still measures receipt rather than rendering, and it may count a preamble before the substantive answer.
Copy link to heading4 best practices to measure and reduce TTFT for AI apps
Client timing and tail percentiles help you find slow requests. Traces separate network, application, and model time so you can see whether prompt or reasoning changes would address the delay.
Copy link to headingMeasure the first visible token at the client
Server-side TTFT excludes the network trip between the browser and your server. Measure at the client to capture the wait users see. Provider and gateway timings help explain how much of that wait occurs elsewhere in the request.
Record when the request leaves the browser and when the first non-empty response text renders. Exclude reasoning and metadata chunks. Decide whether preambles count. If they do, also measure the wait for the substantive answer. When comparing client and server timings, match the same request across traces and use the same output event to end both timers. Subtracting timers that end at different events does not isolate network or route overhead.
Copy link to headingTrack p95 and p99 TTFT instead of averages
An average can look acceptable while the 99th percentile (p99) remains several seconds. P99 is the latency at or below which approximately 99% of requests fall. Traffic bursts can increase this value as some requests wait longer in queues.
Set route-specific alerts at the 95th percentile (p95) and the 99th percentile (p99), and use the median (p50) as the baseline. Use the same start and end events for every measurement. Separate model time from tool, application, and network time to identify the source of the slowest requests.
Copy link to headingKeep the prompt prefix stable
Changing content early in a prompt leaves less of its prefix available for reuse. When the opening matches content cached from a recent request, prompt caching lets the provider reuse that work. OpenAI reports first-token latency reductions of up to approximately 80%. The improvement rose from 7% for 1,024-token prompts to 67% for prompts above 150,000 tokens.
Keep static instructions and tool definitions unchanged. Where the API permits, place changing RAG results and conversation content after the reusable prefix. A timestamp or user identifier near the top breaks the prefix match from that point onward. Requests must also meet the provider's minimum prompt length and cache retention requirements.
Copy link to headingSet reasoning effort per route
Different routes may need different reasoning settings. On supported OpenAI reasoning models, a short preamble can give users feedback before a longer task finishes. An earlier preamble does not establish that the substantive answer starts sooner. For Claude, setting thinking.display: 'omitted' can reduce time to first text if the model would otherwise stream thinking. Some models already use this default. On supported Gemini models, lowering thinking_level to low can reduce latency.
For interactive chat, test whether lower reasoning effort preserves answer quality. Use preambles when progress updates help users. Background agents may benefit from higher effort when task quality matters more than startup latency. Measure the wait for both the first visible update and a useful result.
Copy link to headingHow Vercel helps AI teams measure and lower time to first token
AI Gateway provides provider-level TTFT metrics, routing controls, and request logs. It can prioritize providers based on observed median TTFT, with provider health and other routing settings also influencing the choice. For your own functions, Fluid compute reduces cold-start overhead before the route calls the model.
Copy link to headingLive TTFT metrics per provider
A public benchmark may not represent your workload. AI Gateway measures TTFT and throughput from live AI Gateway traffic and publishes provider-level aggregates in its dashboard and REST API. Use these aggregates to compare providers, then measure performance on your own application traffic.
The REST API reports p50 and p95 first-token latency over a rolling one-hour window. P95 can reveal slow requests that the median hides. These aggregates exclude traffic that uses your own provider keys. The provider latency metrics docs explain the fields, which requests count, and the reporting window.
Copy link to headingProvider sorting by live TTFT
A hardcoded provider order stays the same as provider performance changes. AI Gateway can rank a model's eligible providers by observed median TTFT, lowest first. Provider health and other routing controls also affect the order. To enable this ranking in AI SDK, set sort: 'ttft' in the gateway's provider options:
import { streamText } from 'ai';
const result = streamText({ model: 'deepseek/deepseek-v4-pro', prompt: 'Summarize this support ticket in two sentences.', providerOptions: { gateway: { sort: 'ttft', }, },});AI Gateway calculates the ranking for each request from the available metrics. Providers in order move ahead of that ranking. Default session affinity can then put the provider holding the conversation's cache first. Set sessionAffinity: false to disable that preference. Provider health also affects the ranking, so the chosen provider will not always have the lowest TTFT for a particular request. AI Gateway can fall back when a provider fails or is unavailable. For bring-your-own-key (BYOK) credentials, a configured provider timeout can also trigger failover before streaming begins. The first reasoning token clears that timeout, even if answer text has not arrived.
Copy link to headingTTFT logs for every request
AI Gateway records first-token timing across routes, so you need less custom instrumentation. Its logs report TTFT for streamed model requests that produce tokens. Open a request to see its duration, provider-attempt timing, inference region, and whether zero data retention applied. You still need client instrumentation to measure when response text renders.
The Requests summaries show P50 duration and P50 TTFT for each project and API key. Use those medians as a baseline and monitor p95 and p99 separately for slow requests. Open an individual request to see which model and provider handled it and how long each provider attempt took.
Copy link to headingAutomatic prompt caching for repeat prompts
OpenAI, Google, and DeepSeek cache repeated prompt prefixes when requests meet their eligibility rules. Anthropic, MiniMax, and Alibaba require cache-control configuration. With caching: 'auto' in providerOptions.gateway, AI Gateway adds cache markers for supported providers. These include Anthropic served directly or through Vertex or Bedrock, as well as MiniMax and Alibaba. Cache markers enable caching but do not guarantee a hit.
Anthropic's default cache lifetime is five minutes. Each request that reuses an entry refreshes its lifetime, measured from the start of the request that reads or writes it. Once an entry expires, a later request must rebuild that cached work. For conversations with longer gaps, AI Gateway's Responses API supports a one-hour lifetime. Set both caching: 'auto' and cache_ttl: '1h' at the top level of the same request. Setting caching only through providerOptions.gateway.caching does not activate cache_ttl. The one-hour lifetime costs more to write to cache.
Copy link to headingWarm functions with Fluid compute
During a cold start, the platform initializes a function instance and loads application code before the route can call the model. Users wait longer if that initialization happens during their request. In September 2025, Vercel reported that 99.37% of requests avoided cold starts with Fluid compute. This was a platform-wide result. It does not guarantee the same rate for every application.
Bytecode caching reduces compilation work during later cold starts on supported Node.js versions starting with Node.js 20. Vercel applies it only in production, so preview and development deployments do not include this optimization. Measure production TTFT separately. Fluid compute has been on by default for new projects since April 23, 2025. For older projects, check the setting and enable it if needed.
Copy link to headingMeasure the first token people see and ship on Vercel
Network transit, application startup, a busy provider, a long prompt, reasoning, and buffering can all delay a visible response. Measure each stage before choosing a fix. Browser timing captures the wait users see, and traces help explain it. Chat needs a responsive start and useful answers. Background jobs often prioritize completion time and task quality.
Vercel provides tools to measure these delays and change how requests run:
Live provider metrics: Compare available median and tail first-token metrics, then check performance on your own traffic.
TTFT provider sorting: Prioritize eligible providers by observed median TTFT. Provider health, explicit ordering, and session affinity also affect the choice.
Per-request TTFT logs: Inspect a slow model request's provider attempts, timing, and reported inference region.
Automatic prompt caching: Add cache markers for supported providers that require them. Requests must still meet each provider's eligibility rules.
Fluid compute: Reduce cold-start overhead on your own functions.
Start a new project and send model calls through AI Gateway, or browse templates for an AI chat app you can instrument and measure.
Copy link to headingFrequently asked questions about time to first token
Copy link to headingWhat is a good time to first token?
A first visible response within two seconds can be a starting target for streamed chat. The right target depends on the workload. MLCommons' MLPerf Inference v4.0 Llama 2 70B server benchmark used a TTFT limit of two seconds. It measured first-token delivery rather than browser rendering or the usefulness of the answer. Set a target for each route and percentile, and specify whether reasoning tokens or preambles count. Long-context and background workloads may tolerate longer waits.
Copy link to headingDoes streaming reduce time to first token?
Streaming can shorten the visible wait because it sends output in chunks as the response is generated. It does not inherently reduce the time the model needs to generate its first token or finish the answer. Provider, proxy, or client buffering can delay rendering those chunks.
Copy link to headingHow is TTFT different from time to first byte?
Time to first byte (TTFB) measures the interval between the start of a request and receipt of the response's first byte. TTFT measures the interval to the first model token that meets the metric's definition. Specify the start and end events for each measurement. A route can send a loading shell immediately and still wait seconds for model output. Streaming the shell can improve TTFB and first contentful paint (FCP) without reducing prefill time or the wait for model-generated text.