Most model decisions are reversible with a string change. Embedding models are not. Vectors from two different models occupy incompatible spaces, so switching means re-embedding an entire corpus, and dimension count is baked into every vector already stored. That makes the choice load-bearing on data rather than on code, which is why it deserves more than a leaderboard glance. This guide covers the six criteria that decide the fit and how to test one before committing.
Embedding models also cover images and audio, and multimodal retrieval brings its own evaluation problems. This guide is about text retrieval, where most production retrieval-augmented generation (RAG) and semantic search systems live.
Copy link to headingKey takeaways
Task type sets the decision before any other criterion, because retrieval and classification reward different models.
Domain fit beats leaderboard rank once real query traffic depends on specialized vocabulary.
Dimension count is schema design, since changing it later means re-embedding and re-indexing the entire corpus.
An embedding swap costs a string change at the application layer and a full re-index at the corpus layer.
Benchmarks combining open and private datasets expose the generalization gap that averaged leaderboard scores hide.
Copy link to headingWhat are embedding models?
An embedding model is a neural network, typically a transformer encoder, that converts text into a dense numerical vector. Each vector is a list of float values that places its input as a point in high-dimensional space, where location carries semantic meaning and the distance between two points measures how related their inputs are.
A pooling layer compresses token-level vectors into one fixed-size vector for indexing, and that fixed size is the dimension count. The model that produces those vectors defines the geometry retrieval, which depends on the geometry being fixed the moment the corpus is embedded.
Copy link to headingHow embedding models differ from the chat models teams already run
Most teams meet embedding models after they have already shipped something with a chat model, and they carry over the wrong mental model of what a swap costs. Changing a chat model changes what happens on the next request. Changing an embedding model invalidates everything already stored.
The comparison worth studying is where each decision leaves residue:
The last two rows drive the rest of this guide. A weak embedding model fails quietly, and a general benchmark score is a weak predictor of how it will fail on a specific corpus.
Copy link to headingWhy embedding model choice matters for production teams
Model choice sets five properties of the retrieval system that are hard to change once traffic is running, and each traces back to that fixed geometry:
Domain geometry: A general-purpose model interprets specialized vocabulary through a broad training distribution, which changes how internal enterprise terms or biomedical terms cluster relative to each other.
Vector space incompatibility: Queries and documents have to pass through the same model, because two models place the same text in unrelated coordinate systems.
Exact-match behavior: Dense embeddings handle identifiers such as
SKU-7821-Bpoorly, so hybrid retrieval needs to stay available to keep exact operational phrases reachable.Schema lock-in: Dimension count is written into every stored vector, so changing it means regenerating and reloading the whole corpus.
Grounding quality: Retrieval that reliably surfaces the right source gives the generation step a stronger grounding layer, so an unmeasured default hides a production tradeoff rather than avoiding one.
None of these properties announce themselves during a prototype with 500 documents. Measured retrieval on a real corpus turns all five into engineering decisions with visible costs, which is what explicit criteria are for.
Copy link to headingSix criteria for embedding model selection
Model lists date quickly while selection criteria don't. What follows applies to today's models and to ones released after this article.
Before any of the six, settle what the model is for. Every embedding leaderboard sorts by task, because a model tuned for classification and one tuned for retrieval are different decisions with different winners. RAG and semantic search are retrieval problems, so the retrieval score is the one that carries information and the headline average is the one that hides it. Teams that skip this step compare models on a number that partly measures clustering and summarization performance they will never use.
Copy link to heading1. Dimension count
Dimension count behaves like schema. It belongs in the design conversation before the first ingest run, not in a round of tuning afterward.
Each float32 dimension costs 4 bytes, and storage scales linearly from there. Ten million vectors need roughly 14.3 GiB at 384 dimensions, 28.6 GiB at 768, 57.2 GiB at 1,536, and 114.4 GiB at 3,072. Index memory tends to grow alongside the raw vectors, so the practical gap between a compact model and a large one widens as the corpus does. Smaller vectors also make each similarity computation cheaper, though how much recall that costs is a question only the target corpus answers.
Three levers control storage cost, and they combine:
Dimension count at selection time: The base decision, and the one that hardens into schema.
Matryoshka representation learning (MRL) truncation: Models trained with nested representations tolerate having their vectors shortened after generation, degrading gracefully rather than collapsing as dimensions are dropped.
Quantization: Reducing the precision of each value rather than the number of values. Moving from
float32toint8cuts storage 4x, and binary quantization to one bit per dimension cuts it 32x, with a rescoring pass over the top candidates recovering most of the lost recall.
Quantization is a post-processing step on the vectors, so a team can revisit it later in a way they cannot revisit the dimension count. Any change to a quantization scheme still needs a fresh eval on the target corpus, because the recall cost is corpus-dependent. A mid-sized vector from an MRL-capable model is a defensible starting point, since it preserves both remaining levers.
Copy link to heading2. Benchmark rank
The Massive Text Embedding Benchmark (MTEB) works as a shortlist filter and fails as a decision procedure. It shows which models are competitive in general, and the corpus decides which one wins.
Two structural problems limit what a headline rank can convey. The current multilingual edition, MMTEB, spans over 500 evaluation tasks across 250-plus languages and 10 task categories, and averaging across categories dilutes the retrieval signal that predicts RAG performance. The retrieval suite also leans on general web corpora such as MS MARCO, which resemble a specialized enterprise corpus only loosely.
Benchmark overfitting is the harder problem. When training data overlaps with evaluation data, scores inflate without improving generalization, and the leaderboard rewards models for memorizing the test set. The Retrieval Embedding Benchmark (RTEB) was built to detect exactly that, using a hybrid of open datasets anyone can reproduce and private datasets scored by the maintainers.
A model that scores well on the open half and drops on the private half directly reveals its generalization gap. For a team whose corpus the model has never seen, that gap carries more information than the rank above it. RTEB covers law, healthcare, code, and finance across 20 languages, and reports normalized discounted cumulative gain at rank 10 (nDCG@10) rather than a blended average.
RTEB is in beta and names its own limitations, and one of them deserves attention. Around half of its retrieval datasets are repurposed from question-answering sets, which can create lexical overlap between query and document and quietly favor keyword matching over semantic understanding. Use it as a stronger filter than a general average, not as a verdict. The multimodal equivalent for teams whose retrieval spans images and video is the Massive Multimodal Embedding Benchmark (MMEB).
Both benchmarks narrow the field. Neither replaces an eval set built from a team's own documents, and a top-ranked general model losing to a lower-ranked one on an in-domain set is a routine result rather than a surprising one.
Copy link to heading3. Domain and language fit
Specialized models win when the corpus uses specialized language, and the effect is large enough to show up in the foundational out-of-domain research.
The BEIR benchmark evaluated 10 retrieval systems across 18 datasets and found that BM25, a bag-of-words lexical baseline, remains a strong zero-shot performer, while dense and sparse retrieval models trained on MS MARCO can substantially underperform it on data they were not trained for. Vendors have responded to that gap commercially, and domain-tuned embedding models now ship as products for legal, financial, and code retrieval alongside general-purpose ones. In production the mismatch is rarely subtle. It looks like a model that fails to resolve MI as myocardial infarction, and it surfaces as a retrieval miss rather than an error.
Language fit follows the same logic, and English results do not transfer. Multilingual traffic needs the languages it actually serves tested directly, using queries in those languages against documents in those languages.
A domain-specific model for the corpus belongs on the shortlist by default. If its in-domain gain sits inside eval noise, the general model wins on lower migration risk and broader provider support, which is a real advantage rather than a consolation.
Copy link to heading4. Cost and latency at volume
Per-token pricing looks negligible until it runs across a few million documents. The first full ingest is expensive at any rate, and every re-index pays it again.
At query time, latency is the binding constraint. Real-time search runs on a tight end-to-end budget, and the embedding call has to fit inside it, so a cross-region call that eats most of that budget before retrieval even starts is a real problem. Caching frequent query vectors and dropping to a smaller model are the two levers, and a retrieval metric only means something read next to a latency measurement taken from the same region.
Copy link to heading5. Open weights versus hosted API
Model lifecycle risk runs higher for embeddings than for chat, because replacement vectors don't line up with the vectors already in the index. When a provider retires an embedding model, the corpus needs re-embedding before the replacement can serve the same index, and the deprecation clock is set by someone else.
Open-weight models such as Nomic Embed reduce that exposure. A team can pin a checkpoint indefinitely, and the text never leaves its own infrastructure, which matters when regulated workloads require that control.
Privacy is a sharper consideration than data residency alone. Research on embedding inversion has shown that vectors are not a safe one-way transformation of their inputs. A multi-step method that iteratively corrects and re-embeds candidate text recovered 92% of 32-token inputs exactly, and recovered full names from a dataset of clinical notes. Treat a vector index holding sensitive text with the access controls a team would apply to the text itself, whoever hosts the model.
At low volume, hosted APIs win on cost and operational overhead, and self-hosting an encoder to save on a small ingest is work without a return. At high sustained volume with utilization to match and MLOps capacity already in place, self-hosting becomes economically rational. The decision turns on sustained utilization rather than peak throughput.
Copy link to heading6. Input length limits
Input caps vary by more than an order of magnitude across current models, and that spread sets the chunking strategy rather than the reverse.
Compact sentence-transformer and BGE-style models commonly cap input at 512 tokens. Newer hosted models accept far more, with current gateway-addressable models ranging from 32,000 tokens up to 128,000. Over-limit behavior is provider-specific, and the difference matters. Some providers truncate silently and return a vector built from partial content, while others reject the request outright. Silent truncation is the more dangerous of the two, because the pipeline reports success and the dropped tokens never reach the index.
General benchmarks give no warning here, since much of the retrieval suite uses short passages and a model that degrades on long inputs still scores well. Measuring the token-length distribution of real documents comes first, because chunk size cannot exceed the model's maximum. Each of these six criteria is only as good as the eval set behind it.
Copy link to headingFour practices for evaluating embedding models before committing
All four run before the first production index exists, while changing course still costs a rerun rather than a migration. Running them early is what lets a team weigh quality against cost while the schema is still soft.
Copy link to heading1. Build a domain-specific eval set first
Benchmark scores rarely transfer cleanly to a specific corpus, and retrieval quality is where the gap shows. A working eval set pairs real queries with the documents that should answer them, and even a small, carefully curated set beats none.
Pick metrics that match what the team needs to know:
Recall@k: Whether relevant documents appear at all in the first k results, for binary relevance.NDCG: Graded relevance, which rewards relevant documents more when they rank higher.MRR(mean reciprocal rank): How high the first correct result lands, which is what matters when users read one answer.MAP(mean average precision): Precision across the full ranked list, for cases where several documents are relevant.
Add a robustness pass over edge cases and deliberately diverse inputs, including short queries, long queries, identifiers, and mixed-language text. Aggregate scores hide the inputs where a model collapses, and those inputs tend to be the ones users send.
Copy link to heading2. Test with real production queries, not sample text
An eval set built by generating questions from the chunks they retrieve is misleading, because those questions sit unnaturally close to their source and make retrieval look easier than it will be in production. Queries pulled from real logs, search history, and support tickets are harder, and they are the honest test.
Set the threshold before testing rather than after seeing results. If a candidate misses an nDCG@10 or Recall@k target, a cross-encoder reranker is the cheaper next move, since it improves ranking without touching stored vectors. Fine-tuning comes after that, and it adds a model the team then owns.
Copy link to heading3. Price the re-embedding before locking a dimension count
Dimension count locks in on the first ingest, and changing it means regenerating and reloading every vector. The token bill is usually the smallest line in that estimate. Engineering time, index rebuild windows, and the risk carried by a cutover cost more.
An MRL-capable model keeps truncation available, which is the option that matters when the first index succeeds and the corpus grows faster than the storage plan assumed. Write the estimate down before committing, because a number produced under deadline pressure later will be optimistic.
Copy link to heading4. Plan the model migration before it becomes urgent
Vector database vendors make a reasonable case that model choice doesn't have to be permanent: Qdrant notes that its architecture makes migrating between models relatively straightforward, and Weaviate recommends starting with a lightweight model and swapping later. That advice is accurate about the layer it describes, the storage layer, which is worth reconciling with the position here rather than treating as a contradiction.
Both are true at different layers. A swap is close to free at the application layer, where the model reference is a string in a config file. It is expensive at the corpus layer, where every vector has to be regenerated and re-indexed before the new model can answer a single query. The teams that get surprised are the ones who priced the application-layer change and scheduled the corpus-layer one for a sprint that turned out to be a quarter.
Two patterns keep the corpus-layer cost manageable:
Versioned metadata on every embedding: Store the model name, version, dimension count, distance metric, and any required prefix format alongside each vector. The cost is negligible and it makes the eventual swap traceable instead of archaeological.
Multiple vectors per record: Vector stores that support several named vectors on one record, each with its own dimension count and distance metric, allow a second model to be added to an existing collection without recreating it. Queries can then route to one model or run hybrid retrieval across both during a coexistence period, which replaces a cutover with a gradual shift.
A blue-green pattern covers the case where coexistence isn't practical, validating the new model and index against the same eval set before any traffic moves, with the old index live until the new one proves out. Either way, the migration is a planned project with a schedule rather than an emergency.
Copy link to headingHow Vercel helps engineering teams put an embedding model into production
Selection criteria and migration plans both assume reversibility, and reversibility is largely a property of the infrastructure the calls run through. The AI Gateway and surrounding primitives are built so provider choice doesn't harden into application code.
Copy link to headingSwap embedding models with a config change through AI Gateway
Committing to one provider's SDK makes the swap larger than the model decision that triggered it, and the rewrite lands on whichever team owns the ingest pipeline. AI Gateway exposes one API across providers, so moving between models and providers doesn't require rewriting application code.
Embedding models are addressed as creator/model-name strings, which puts dozens of embedding models from different providers behind one interface and one API key. Moving to the shortlist winner becomes a string change at the application layer, leaving the eval set and the migration plan to handle the corpus-layer work that actually carries risk.
Copy link to headingBatch and single embedding calls with the AI SDK
Hand-rolled provider calls make ingest pipeline behavior harder to replace later, because retry logic, batching, and error handling end up specific to one vendor's client. AI SDK gives TypeScript teams one interface for both call shapes, with embed for single values and embedMany for batches.
A batch ingest call looks like this:
import { embedMany } from 'ai';
const { embeddings } = await embedMany({ model: 'openai/text-embedding-3-small', values: chunks,});
Keeping batch size and concurrency visible next to token usage and the model string is what makes ingest throughput and query latency measurable while the eval set decides which model serves production traffic.
Copy link to headingKeep ingest pipelines running through provider outages
A document is not searchable until its embedding call finishes, so a provider outage midway through a large ingest leaves the corpus half-indexed and a queue to reconcile by hand. AI Gateway routes across providers through one endpoint and retries automatically when a provider fails.
Teams can also configure model fallbacks that are tried in sequence until one succeeds. Ordering is worth thinking about for embeddings specifically, since a fallback to a different model produces vectors that don't match the index. Falling back across providers hosting the same model keeps vectors comparable, while falling back to a different model is a decision to make deliberately rather than by default.
Copy link to headingRun full-corpus re-embedding as a durable workflow
Re-embedding a corpus does not fit inside a request, and the first attempt often reveals that by timing out halfway through. Incremental ingest and request-sized embedding work fit Fluid compute, where function duration defaults to 300s and reaches 800s on Pro and Enterprise, with an extended maximum of 1,800s currently in beta for per-function configuration.
Full-corpus migrations need orchestration that survives a deploy and resumes from a checkpoint instead of starting over. Vercel Workflows does this, holding state for minutes to months with no duration limit and surviving crashes or deployments mid-migration. Keeping that orchestration outside the user request path, then cutting traffic over only after the eval set confirms retrieval quality on the new index, is what turns a re-embedding run into a routine operation.
Copy link to headingChoose and route embedding models on Vercel
Choosing an embedding model looks like a model decision and is really a decision about which layer absorbs the next change. Dimension count, task type, and domain fit set retrieval quality, and all three are reversible only through a full corpus rebuild. What the platform underneath decides is whether the application-layer half of a swap costs a config edit or a rewrite. That is why model evaluation and platform choice belong in the same conversation.
Vercel gives teams the primitives that keep an embedding choice reversible:
AI Gateway: Embedding providers addressed as
creator/model-namestrings behind one API key, so the shortlist winner is a config change rather than a rewrite.Model fallbacks: An ordered provider list that keeps an ingest pipeline running when one embedding provider degrades.
AI SDK: One interface for
embedandembedManyacross providers, so ingest and query paths don't carry vendor-specific client code.Vercel Workflows: Durable execution with checkpoints and retries for full-corpus re-embedding, with no duration limit and state that survives deploys.
AI Gateway observability: Token counts and latency by model, so ingest throughput stays visible while a migration runs.
To put a shortlisted embedding model into production, start a new project and keep the model path replaceable from the first commit, or browse Vercel templates for a working starting point.
Copy link to headingFrequently asked questions about embedding models
Copy link to headingCan teams switch embedding models without re-embedding the corpus?
No. Embeddings from different models, and from different versions of one model, occupy geometrically incompatible spaces, so every switch requires re-embedding the corpus. A blue-green cutover that validates the new index before routing queries to it keeps the swap reversible.
Copy link to headingAre more embedding dimensions always better?
Not automatically. The right dimension count is workload-specific, so test candidates against the target corpus rather than defaulting to the maximum. An MRL-capable model leaves room to shorten vectors later without regenerating them, and quantization reduces storage further without changing dimension count.
Copy link to headingHow much notice do providers give before deprecating embedding models?
Notice varies by provider and by where a model sits in its lifecycle. Because replacement vectors don't align with existing ones, plan for full re-embedding and re-indexing whenever a provider retires an embedding model, and prefer pinnable open weights when that timeline has to stay under a team's control.
Copy link to headingDo open-source embedding models need special query prefixes?
Often, yes. Several open model families require task prefixes, such as distinct markers for queries and documents, and omitting a required prefix degrades retrieval quality without raising an error. Check the model card, then store the prefix format in the same versioned metadata as the model name and dimensions.