Fine-tuning vs RAG: when to use each

Most teams building AI features hit the same fork early: do you teach the model new facts through retrieval, or reshape how it behaves through training? The reflexive answer has been "start with RAG," and it has been right often enough to stop feeling like a choice. I'd push back on that. Reflexive advice ages badly in a field where the base models shift underneath your architecture every few months, and this particular piece of advice is aging right now.

The split across the ecosystem tells you the question is unsettled. Our State of AI survey puts 60% of teams on RAG, 12% on fine-tuning, and 20% on no customization at all. We've watched both ends of that distribution up close, because we built v0 on an architecture that uses retrieval and fine-tuning together. The teams I've seen stall aren't the ones who picked wrong between RAG and fine-tuning. They're the ones who never diagnosed which problem they actually had. So before the comparison, two definitions, because the entire decision turns on the difference between them.

Copy link to headingWhat is RAG

Retrieval-augmented generation changes what a model knows, not how it reasons. You take your own data, the docs, the policies, the pricing, the records that the base model was never trained on, embed it, store it, and pull the relevant pieces into the prompt at query time. The model's behavior is untouched. You're handing it a better set of facts to answer from.

This is the right tool when the bottleneck is knowledge that moves. A support agent that needs your current product docs, a chatbot that has to cite today's pricing, anything where the correct answer changed last week: that's RAG's territory. Our guidance frames it as enhancing out-of-the-box outputs with your specific data, improving accuracy and relevance without the cost of training. The thing I'd underline is the failure boundary. RAG fixes ignorance. It does nothing for a model that knows the facts and still answers in the wrong shape.

Copy link to headingWhat is fine-tuning

Fine-tuning changes how a model behaves, not what it knows. You train on examples until the model internalizes a pattern: a fixed output format, a house tone, a domain-specific way of reasoning through a problem. The knowledge it draws on is still whatever it learned in pretraining. What you've moved is the behavior.

This is the right tool when the bottleneck is consistency. A model that has to return JSON in your exact schema on every call, or reason through legal clauses the way your attorneys do, has a behavior problem, and behavior is the one thing retrieval can't touch. We trained a model called vercel-autofixer-01 for exactly this kind of job, using reinforcement fine-tuning with Fireworks AI to make one narrow behavior, catching and fixing streaming errors, reliable. The tradeoff is the part teams underestimate. A fine-tuned model's knowledge is frozen at training time, so the day your facts or your target behavior move, you're retraining. RAG you update by changing a row. Fine-tuning you update by running the training job again.

Copy link to headingFine-tuning vs RAG: a side-by-side comparison

Before the nuance, here is how I lay the two out for teams deciding where to start. Read down the column that matches the problem you actually have, not the approach you find more interesting to build.

The table makes the cleanest version of the point obvious. These are not two answers to one question. They are answers to two different questions, and the rows where they diverge hardest, data freshness and how you update, are the rows that decide which one you'll regret picking.

Copy link to headingThe real question is knowledge vs behavior, not RAG vs fine-tuning

When we analyze how teams build on our platform, the ones that stall made the same move: they treated a behavior problem as a knowledge problem, or the reverse. The most common framing puts RAG on fresh data and fine-tuning on everything else, and it misses the actual dividing line. Fine-tuning changes behavior. RAG changes knowledge. Diagnose which one is broken and the architecture chooses itself.

The classic failure mode is fine-tuning a model on facts that change monthly. It looks reasonable in month one. The model returns your data in your format and everyone is happy. Then the data shifts, and because the facts are baked into the weights, the only way to update them is another training run. By the time you've done that three times, you've built a retraining treadmill to solve a problem retrieval would have handled with a database write. The reverse failure is quieter. A team bolts retrieval onto a model that already knows the facts but keeps answering in the wrong shape, and no amount of better context fixes a behavior problem.

Here is the diagnostic I'd run before writing any code. Match the symptom to the underlying problem first, then pick the tool.

The teams that get this distinction right ship faster, and it isn't because they're smarter about models. It's because they spent ten minutes naming the problem before they spent ten weeks building the wrong solution to it.

Copy link to headingWhen to use RAG

Reach for RAG when the answer depends on something that wasn't true when the model was trained. In my experience, the strongest signal is temporal: if you can finish the sentence "the right answer changed when..." then you have a knowledge problem, and retrieval is the tool.

The cases where RAG is clearly correct share a shape:

• The answer depends on data that changes weekly or faster, like pricing, inventory, or account state.

• The model has to cite a source, and the source is yours, not something in pretraining.

• You need an audit trail showing which document an answer came from.

• The knowledge base is large enough that stuffing all of it into every prompt is wasteful.

The textbook case is a support assistant answering from this week's pricing and policy docs: the model's behavior is fine, the facts just move under it. The honest tradeoff is that RAG moves your hardest problem from the model to the pipeline. You stop worrying about what the model knows and start worrying about whether retrieval surfaced the right context, which is its own discipline and its own failure mode. More on that below, because it's the part most teams underinvest in until it burns them.

Copy link to headingWhen to use fine-tuning

Reach for fine-tuning when the model already has the knowledge and keeps getting the behavior wrong. We reached for it with vercel-autofixer-01 for precisely that reason: the frontier models in v0 had the capability, but error-correction needed to be consistent and predictable in a way prompting alone couldn't guarantee at scale.

Fine-tuning earns its complexity when:

• You need the same output format on every single call, with no drift.

• The domain has reasoning conventions a general model won't follow reliably from a prompt.

• A specific behavior has to be fast and cheap at high volume, and you can't afford the latency of a long instructional prompt every time.

• The behavior is stable enough that you won't be retraining constantly to keep up with it.

The textbook case is structured extraction or classification at volume, where the model has to emit your exact schema on every call and a long instructional prompt each time is too slow to run. The cost is the freeze. A fine-tuned model's knowledge stops at its training cutoff, and its behavior can drift relative to what you actually want as your product evolves. You pay for the win up front in data curation and training, and you pay again every time the target moves. That's a fine trade for a narrow, durable behavior. It's a bad trade for anything you'd otherwise fix with fresher context.

Copy link to headingWhy "always start with RAG" is aging out

Here is the contrarian position I'll defend: the "start with RAG" default came from a period when context windows were small enough that retrieval was the only practical way to get external documents in front of a model. That constraint is loosening, and the advice built on it is loosening with it.

Two things changed underneath the old advice. Base models got dramatically better at the behaviors teams used to fine-tune for, and context windows got large enough to make retrieval optional for static document sets. GPT-5 scores 96.7% on Tau 2-bench telecom, a tool-calling benchmark where no previous model scored above 49%. If you were planning to fine-tune for reliable tool calling, that plan may already be obsolete at zero training cost. On the prompting side, a Towards Data Science analysis found prompt optimization techniques like DSPy beating fine-tuning by 6 to 19 points on some benchmarks while using 35x fewer rollouts. And RAGFlow's 2025 review documents teams dropping retrieval entirely for contract review and fixed-format report analysis by stuffing the documents straight into long context, a shift Sebastian Raschka expects to slowly retire classical RAG as the default for document queries.

This doesn't mean RAG is dead. It means the first question changed. Instead of "RAG or fine-tuning," start with "do I need external retrieval at all, given current context sizes?" For data that changes daily, the answer is still yes. For a fixed set of 50 product docs, a long context window is often simpler, cheaper, and faster than a retrieval pipeline you have to build and maintain.

Cost follows the same logic, and it's mostly a function of volume. Below a certain scale the per-query overhead of embedding plus retrieval plus longer context isn't worth amortizing. Above it, fine-tuning's fixed training cost spreads thin enough to win.

The numbers behind those bands come from practitioner frameworks worth reading directly. A Towards Data Science analysis puts prompt engineering ahead of fine-tuning below 100K queries, and the tianpan.co production decision framework estimates fine-tuning turns cost-favorable above 10 million queries per month. In its worked example, a fine-tuned GPT-4o-mini query at roughly 500 output tokens runs about $0.0006, against roughly $0.014 for a full RAG pipeline with Sonnet and retrieved context, using OpenAI pricing and Anthropic pricing. That's about 20x at the extreme. Between those bounds, RAG with proper chunking and hybrid search stays the practical default, especially on infrastructure that doesn't bill you for I/O wait.

Copy link to headingWhat v0's composite architecture shows

Most practitioner guidance frames fine-tuning and RAG as rivals. We treat them as layers, and v0 is the proof that "pick one" is a false constraint. The v0 composite model family combines RAG for knowledge retrieval, reasoning from frontier LLMs, and the custom fine-tuned vercel-autofixer-01 for error correction. The fine-tuned model doesn't replace retrieval. It owns one behavioral problem, fixing streaming errors that frontier models produce, where consistent behavior matters more than broad knowledge.

That pattern isn't unique to us. A medical LLM review found fine-tuning combined with RAG outperforming either approach alone in domain-specific settings. The mechanism is the same one the diagnosis table implies: each layer solves a distinct problem instead of duplicating effort. Retrieval handles knowledge, fine-tuning handles behavior, and the frontier model handles the reasoning in between.

I want to be honest about the cost of this, because the takeaway is not "build a composite system." Hybrid architectures carry real complexity, and most applications don't need it. v0 justifies it because it operates at very high volume and the error-fixing behavior has to be reliable at that scale. For most teams, starting with one approach and adding the second only when you hit a wall you genuinely can't solve otherwise is the faster route to production. The layering is an option you grow into, not a starting posture.

Copy link to headingBuilding both on Vercel

Choosing an architecture is one decision. Making the retrieval pipeline survive production is a different one, and the failure modes we watched across the platform shaped how we built the AI SDK's primitives and how the underlying compute is billed.

Copy link to headingFluid compute changes the cost math for RAG

On most platforms a RAG request is expensive for a boring reason: you pay for wall time, and a RAG request spends most of its life waiting. Waiting on the embedding API, waiting on the vector database, waiting on the LLM to generate. On wall-time billing you pay for every one of those idle seconds.

Fluid compute bills active CPU time and excludes I/O wait. The most expensive parts of a RAG pipeline, vector retrieval and LLM latency, aren't billed at the compute layer at all, which is why the same pipeline costs less here than on wall-time infrastructure. Vercel Functions running the Edge Runtime are billed in units of 50ms of CPU time per invocation, where CPU time excludes time spent waiting for data fetches. Fine-tuning changes the shape of this rather than making compute free. A provider-hosted fine-tuned model is still an external call, so that wait is excluded too, and the savings come from removing retrieval work and shortening prompts. Run inference yourself and that compute is billed wherever it runs.

Copy link to headingAI SDK retrieval primitives

The AI SDK provides embed and embedMany for generating embeddings, cosineSimilarity for comparing vectors, and tool-based retrieval that lets the model decide when to search. Our official RAG chatbot guide wires this up with Next.js and the App Router, Drizzle ORM, and PostgreSQL with pgvector. The flow is exactly as plain as it sounds: chunk the source, embed with embedMany, store in Postgres, then at query time retrieve by vector similarity and keep only matches above a 0.7 threshold.

import { embedMany, cosineSimilarity } from 'ai'
const { embeddings } = await embedMany({
  model: embeddingModel,
  values: chunks,
})

// at query time, keep only the chunks that clear the bar

const matches = stored.filter(
  (row) => cosineSimilarity(queryEmbedding, row.embedding) > 0.7,
)

Retrieval itself runs through tool calling. A getInformation tool takes a query, calls findRelevantContent, and returns matching chunks, and stopWhen: stepCountIs(5) keeps the model from looping through retrievals forever.

const result = streamText({
  model,
  stopWhen: stepCountIs(5),
  tools: {
    getInformation: tool({
      description: 'look up information to answer the question',
      inputSchema: z.object({ query: z.string() }),
      execute: async ({ query }) => findRelevantContent(query),
    }),
  },
});

AI SDK 4.1 introduced createDataStreamResponse, which streams retrieved context to the client before generation starts, so users see their sources immediately instead of staring at a blank screen. AI SDK 6 lets you control retrieval per request through its agent call options: route simple queries to smaller models, swap document sets by user context, or skip retrieval entirely when a query doesn't need it.

Copy link to headingAI Gateway for comparing model behavior against RAG

The decision between a fine-tuned model and a base model with RAG shouldn't be made on benchmarks. It should be made on your traffic. AI Gateway makes that practical with one API key, hundreds of models, and zero markup on tokens. Switching models becomes a one-line change instead of a multi-file edit. Put your candidates behind the same interface, run them against a base-plus-RAG setup, and decide on production behavior instead of theory.

Copy link to headingRAG fails silently, and that's the real risk

The hardest part of RAG was never building the pipeline. It's knowing when the pipeline is quietly wrong, and I've watched this one catch experienced teams off guard more than any other failure in this space.

Here's the trap. A Towards Data Science case study showed naive RAG returning flatly incorrect answers, wrong dollar amounts, wrong policies, wrong rate limits, with confidence scores of 78 to 81%, identical to the scores on correct answers. As the study put it, "confidence was never the signal that something had gone wrong." When a retrieved chunk is semantically close to the query but factually irrelevant, the model produces a fluent, confident, wrong answer, and nothing in the output tells you it happened.

The fixes are concrete, and they all push effort into retrieval quality rather than model quality:

• Hybrid search that pairs vector similarity with BM25 keyword matching, which catches the cases where semantic search alone retrieves plausible-but-wrong documents. Nitor Infotech's RAG maintenance guidance names semantic-only search as a common pitfall.

• A similarity threshold, like the AI SDK guide's 0.7 floor, that rejects weak matches instead of feeding them to the model.

• A retrieval step cap, like stopWhen: stepCountIs(5), so the model can't spiral through irrelevant lookups.

• Self-contained chunks. Our RAG guide recommends each chunk carry enough context to stand alone, including product names and section headings.

Fine-tuned models fail differently, and more visibly in one sense: they don't retrieve the wrong document because they don't retrieve at all. Their failure is staleness and behavioral drift, and the fix is retraining. The practical takeaway across both is the same. A MEGA-RAG study reported more than a 40% reduction in hallucination rate, but only because it returned the right supporting context. Invest in retrieval quality before you invest in a better model. The model is rarely your bottleneck.

Copy link to headingThe faster path

We built v0 on a hybrid architecture because it genuinely needed both: broad knowledge retrieval and precise error-correction behavior. That complexity is justified at v0's scale. It is not justified as a starting point for most teams. The faster path is to begin with the AI SDK's retrieval primitives, run the pipeline on Fluid compute where I/O wait is excluded from billing, and add fine-tuning only when you hit a behavior problem prompting can't fix.

If you scan back through the decisions in this piece, fine-tuning vs RAG was never really the question. The question is which failure mode you're solving for. Stale knowledge is a retrieval problem. Inconsistent behavior is a training problem. And an increasing share of what teams used to customize for is now just a base-model capability you can test for free. The teams I've watched ship fastest test the base model first, add retrieval when knowledge is the bottleneck, and fine-tune only when behavior is the thing that's broken. They don't pick a side. They diagnose, then they layer.

Copy link to headingFAQs

Copy link to headingWhat's the core difference between RAG and fine-tuning?

RAG changes what a model knows by retrieving external data into the prompt at query time. Fine-tuning changes how a model behaves by training it on examples. If your problem is stale or missing facts, that's knowledge, and RAG is the tool. If your problem is inconsistent format, tone, or reasoning, that's behavior, and fine-tuning is the tool.

Copy link to headingIs fine-tuning better than RAG?

Neither is better in the abstract; they solve different problems. Fine-tuning wins when the issue is behavior: format, tone, or domain reasoning. RAG wins when the issue is knowledge: facts that change or live in your private data. Asking which is "better" usually means the underlying problem hasn't been diagnosed yet, and for many production systems the strongest answer is both, layered.

Copy link to headingCan you use both RAG and fine-tuning together?

Yes, and for some systems you should. v0 runs a composite architecture that combines RAG for knowledge, frontier model reasoning, and a fine-tuned model (vercel-autofixer-01) for a specific behavior. A medical LLM review found the combination outperforming either approach alone in domain-specific settings. The caveat is that hybrid systems carry real complexity, so most teams should start with one and add the other only when they hit a wall.

Copy link to headingIs fine-tuning cheaper than RAG?

Only at high volume. Practitioner frameworks put prompt engineering ahead of fine-tuning below roughly 100K queries and estimate fine-tuning turning cost-favorable above roughly 10 million queries per month, where a fine-tuned query can cost around 20x less than a full RAG pipeline. Between those bounds, RAG is usually cheaper, especially on infrastructure like Fluid compute that excludes I/O wait from billing.

Copy link to headingDo long-context models make RAG obsolete?

For static document sets, increasingly yes. Teams are dropping retrieval for fixed tasks like contract review by stuffing documents directly into long context, and some practitioners expect classical RAG to fade as the default for document queries. For data that changes daily, RAG is still the right answer. The better first question is whether you need external retrieval at all given current context sizes.

Copy link to headingDo you still need to customize models at all with GPT-5-class models?

Less often than you used to. GPT-5 scores 96.7% on the Tau 2-bench telecom tool-calling benchmark, where no previous model topped 49%, which can eliminate the need to fine-tune for reliable tool calling at zero training cost. Test base model capabilities first. Add retrieval when knowledge is the bottleneck, and fine-tune only when behavior is the problem prompting can't solve.