Cohere Rerank API with AI Gateway

AI Gateway provides Cohere-compatible Rerank API endpoints, so you can use the Cohere SDK or plain HTTP requests to rerank documents through a unified gateway with only a URL change.

Reranking reorders a list of documents by their relevance to a query. It is commonly used to improve results in retrieval-augmented generation (RAG) pipelines after an initial retrieval step. To see which models support reranking, use the Reranking filter on the models page.

The Rerank API implements the same specification as the Cohere Rerank API.

The Rerank API is available at the following base URL:

The Cohere SDK appends the version path (/v1/rerank or /v2/rerank) to this base URL, so configure the SDK with the bare host.

The Rerank API supports the same authentication methods as the main AI Gateway:

• API key: Use your AI Gateway API key with the Authorization: Bearer <token> header
• OIDC token: Use your Vercel OIDC token with the Authorization: Bearer <token> header

You only need one of these. If an API key is specified it takes precedence over any OIDC token, even if the API key is invalid.

The AI Gateway supports both versions of the Cohere Rerank API:

• POST /v2/rerank - The current Cohere Rerank API dialect. This is what the Cohere v2 SDK clients use.
• POST /v1/rerank - The earlier Cohere Rerank API dialect. It additionally supports object documents, rank_fields, and return_documents.

Both endpoints accept the same model slugs and return the same response shape.

• model (string): The reranking model to use (for example, cohere/rerank-v3.5)
• query (string): The search query to rank documents against
• documents (array): The documents to rank. On /v2/rerank these are strings. On /v1/rerank they can be strings or objects.

• top_n (integer): The number of top results to return. Defaults to returning all documents.
• max_tokens_per_doc (integer): Long documents are truncated to this many tokens.
• return_documents (boolean, /v1/rerank): When true, each result includes the original document. Defaults to false.
• rank_fields (array of strings, /v1/rerank): For object documents, the fields to rank on, in order. Defaults to the text field.

The response contains a ranked list of results, ordered by relevance:

{
  "id": "gen_01J...",
  "results": [
    { "index": 1, "relevance_score": 0.999071 },
    { "index": 0, "relevance_score": 0.32713068 }
  ],
  "meta": {
    "api_version": { "version": "2" },
    "billed_units": { "search_units": 1 }
  }
}

• results[].index (integer): The position of the document in the original documents array
• results[].relevance_score (number): A relevance score normalized to the range [0, 1]. The scale is not linear: a score of 0.9 does not mean a document is twice as relevant as one scored 0.45.
• results[].document (object): The original document, included only when return_documents is true on /v1/rerank
• meta.billed_units.search_units (number): The number of billed search units. One search unit covers up to 100 documents.

import { CohereClient } from 'cohere-ai';
 
const cohere = new CohereClient({
  token: process.env.AI_GATEWAY_API_KEY,
  environment: 'https://ai-gateway.vercel.sh',
});
 
const response = await cohere.v2.rerank({
  model: 'cohere/rerank-v3.5',
  query: 'What is the capital of the United States?',
  documents: [
    'Carson City is the capital city of the American state of Nevada.',
    'Washington, D.C. is the capital of the United States.',
    'Capital punishment has existed in the United States since before it was a country.',
  ],
  topN: 2,
});
 
console.log(response.results);

import os
import cohere
 
co = cohere.ClientV2(
    api_key=os.getenv('AI_GATEWAY_API_KEY'),
    base_url='https://ai-gateway.vercel.sh',
)
 
response = co.rerank(
    model='cohere/rerank-v3.5',
    query='What is the capital of the United States?',
    documents=[
        'Carson City is the capital city of the American state of Nevada.',
        'Washington, D.C. is the capital of the United States.',
        'Capital punishment has existed in the United States since before it was a country.',
    ],
    top_n=2,
)
 
print(response.results)

curl https://ai-gateway.vercel.sh/v2/rerank \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cohere/rerank-v3.5",
    "query": "What is the capital of the United States?",
    "documents": [
      "Carson City is the capital city of the American state of Nevada.",
      "Washington, D.C. is the capital of the United States.",
      "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 2
  }'

The /v1/rerank endpoint accepts object documents. Use rank_fields to choose which fields to rank on, and return_documents to echo the original documents back in the response:

curl https://ai-gateway.vercel.sh/v1/rerank \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cohere/rerank-v3.5",
    "query": "What is the capital of the United States?",
    "documents": [
      { "title": "Nevada", "text": "Carson City is the capital city of Nevada." },
      { "title": "USA", "text": "Washington, D.C. is the capital of the United States." }
    ],
    "rank_fields": ["title", "text"],
    "return_documents": true,
    "top_n": 2
  }'

You can control which provider serves the request with providerOptions.gateway, the same way as other AI Gateway APIs. For example, restrict the request to a specific provider:

{
  "model": "cohere/rerank-v3.5",
  "query": "What is the capital of the United States?",
  "documents": ["..."],
  "providerOptions": {
    "gateway": {
      "only": ["bedrock"]
    }
  }
}

See provider options for the full list, including provider filtering, ordering, and model fallbacks.

If you use the AI SDK, call reranking models with the rerank function instead of the REST API. See Reranking for details.

The API returns standard HTTP status codes. Errors follow the Cohere error format:

{
  "message": "invalid request: query: too_small"
}

• 400 Bad Request: Invalid request parameters
• 401 Unauthorized: Invalid or missing authentication
• 403 Forbidden: Insufficient permissions
• 404 Not Found: Model or endpoint not found
• 429 Too Many Requests: Rate limit exceeded
• 500 Internal Server Error: Server error