--- title: Make your documentation readable by AI agents description: Serve markdown to AI agents using content negotiation, .md endpoints, agent auto-detection, llms.txt, sitemap.md, and MCP. Includes a checklist, implementation patterns, and a compliance framework for measuring agent readiness across content platforms. url: /kb/guide/make-your-documentation-readable-by-ai-agents canonical_url: "https://vercel.com/kb/guide/make-your-documentation-readable-by-ai-agents" published: 2026-03-27 last_updated: 2026-07-16 authors: Rich Haines, Timothy Jordan related: - /docs/mcp - /kb/guide/agent-readability-spec install_vercel_plugin: npx plugins add vercel/vercel-plugin --- AI agents (Claude Code, ChatGPT, Cursor, Copilot) are a primary consumer of developer documentation. They don't need navigation chrome, dark mode toggles, or animated code blocks. They need: - **Discoverable content**: Where the docs are and what they cover - **Clean retrieval**: Markdown, not a DOM tree - **Structured metadata**: Version, last updated, and canonical URL - **Tool access**: Search and fetch via protocol, not scraping Most content platforms serve agents the same HTML page they serve humans. The agent then spends tokens stripping tags, guessing at content boundaries, and hoping the important information survives extraction. The result: hallucinated APIs, outdated code examples, and missed context. ## Three layers of agent readiness Agents interact with content in three layers. They discover what exists, retrieve clean content, optionally index at scale, and use tools for precision queries. ### Layer 1: Discovery Help agents find what exists before they fetch anything. | Requirement | Implementation | Purpose | | ----------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `/llms.txt` | Curated markdown index at the site root with section headings and links | Entry point for agents and humans pasting docs into an IDE | | `sitemap.xml` | Standard sitemap with accurate `lastmod` dates | Freshness tracking for agents that monitor content changes | | `sitemap.md` | Semantic sitemap served as markdown with section headings, categories, and page descriptions | High-level orientation for LLM-assisted navigation and contributor onboarding | | `robots.txt` | Documented stance on agent access (which bots can crawl, which pages are off-limits) | Crawl control and access transparency | | JSON-LD structured data | Title, description, canonical URL, and breadcrumbs on every HTML page | Agents that parse HTML can understand page type and relationships without traversing the DOM | Example JSON-LD for a docs page: ```json { "@context": "", "@type": "TechArticle", "headline": "Vercel Functions", "description": "Deploy server-side code on Vercel.", "url": "", "breadcrumb": { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Docs", "item": "" }, { "@type": "ListItem", "position": 2, "name": "Functions", "item": "" } ] } } ``` ### Layer 2: Retrieval This is the highest-impact layer. When an agent fetches a docs page, it should get markdown, not HTML with embedded scripts. | Mechanism | How it works | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Content negotiation | Agents that send `Accept: text/markdown` receive markdown with `Content-Type: text/markdown; charset=utf-8` and a `Vary: Accept` header. Claude Code sends this header natively. | | Agent auto-rewrite | Detected AI agents receive markdown automatically, even without an `Accept: text/markdown` header. Detection uses user-agent matching, RFC 9421 Signature-Agent headers, and a heuristic fallback for unknown agents. | | `.md` endpoints | Append `.md` to any docs URL to get the markdown version directly. Works without custom headers. Useful for pasting docs into a chat or IDE. | | Rich frontmatter | Every markdown response includes metadata for accurate citations. | | HTML alternate link | Adding `` in the HTML `` tells agents a markdown version exists. | Example request and response: ```bash curl -H "Accept: text/markdown" ``` ```yaml --- title: Functions description: Deploy server-side code on Vercel. canonical_url: md_url: last_updated: 2026-01-15T12:34:56.000Z --- # Functions Deploy server-side code on Vercel... ``` ## Related pages > **For AI agents:** Follow these links to understand how this page connects to the rest of the Vercel ecosystem. For the full cross-link map (inbound, outbound, prerequisites, and semantic neighbors), see the .graph.md link below. - [Markdown & Agent Discovery](https://vercel.com/docs/agent-resources/markdown-access?from=related) — Access Vercel documentation as markdown using .md endpoints or the copy button. - [Agent Resources](https://vercel.com/docs/agent-resources?from=related) — Resources for building with AI on Vercel, including documentation access, MCP servers, and agent skills. - [AI Coding Agents](https://nextjs.org/docs/app/guides/ai-agents?from=related) — Learn how to configure your Next.js project so AI coding agents use up-to-date documentation instead of outdated trainin - [Ship It](https://eve.dev/docs/tutorial/ship-it?from=related) — Part 9 of the Build an Agent tutorial. Put a web dashboard on the agent with useEveAgent, replace placeholderAuth, and d - [Coding Agents](https://vercel.com/docs/ai-gateway/coding-agents?from=related) — Configure popular AI coding agents to use the AI Gateway for unified model access and spend monitoring. - [Vercel Plugin](https://vercel.com/docs/agent-resources/vercel-plugin?from=related) — Install the Vercel plugin to give supported AI coding tools Vercel context, skills, specialist agents, slash commands, a - [vercel agent](https://vercel.com/docs/cli/agent?from=related) — Generate an AGENTS.md file with Vercel deployment best practices using the vercel agent CLI command. - [How to serve documentation for agents](https://vercel.com/kb/guide/how-to-serve-documentation-for-agents?from=related) — Learn how to serve markdown to agents and HTML for humans from the same URL - [How to build AI Agents with Vercel and the AI SDK](https://vercel.com/kb/guide/how-to-build-ai-agents-with-vercel-and-the-ai-sdk?from=related) — Learn how to build, deploy, and scale AI agents on Vercel using the AI SDK. This guide covers calling LLMs, defining too - [Draft content in your voice from Slack with eve](https://vercel.com/kb/guide/eve-content-agent?from=related) — Deploy the eve content agent template, a Slack bot that drafts blog posts, LinkedIn posts, release notes, and newsletter - [Draft documentation fixes from reader feedback with Sanity and eve](https://vercel.com/kb/guide/sanity-eve-agent?from=related) — Build an autonomous agent that drafts fixes in Sanity, stages them for human review, and calls the model through AI Gate - [AI Agents on Vercel](https://vercel.com/kb/guide/ai-agents?from=related) — This guide provides an overview of how to build and deploy AI agents on Vercel. Full cross-link map for this page: [/kb/guide/make-your-documentation-readable-by-ai-agents.graph.md](/kb/guide/make-your-documentation-readable-by-ai-agents.graph.md) Without `canonical_url` and `last_updated`, agents can't link back to the source or judge whether their information is stale. The `Vary: Accept` header tells CDNs to cache HTML and markdown responses separately for the same URL. Without it, an agent's markdown response could be served to a browser, or vice versa. #### **Always serve markdown to agents** Content negotiation and `.md` endpoints cover agents that explicitly request markdown. Many agents don't. They send a standard `GET` with no special headers. If your platform can detect that the request comes from an AI agent, serve markdown anyway. Vercel uses a three-layer detection approach: 1. **User-agent matching**: Check against a maintained list of known AI agent strings (Claude, ChatGPT, GPTBot, Cursor, Copilot, and others). This is the most reliable signal. 2. **Signature-Agent header**: The RFC 9421 standard header, used by ChatGPT's agent. Validate against known AI service domains. 3. **Heuristic fallback**: If the request is missing the `sec-fetch-mode` header (which real browsers always send) and the user-agent matches a bot-like pattern, treat it as an agent. This catches unknown agents at the cost of occasional false positives, which are low-harm since serving markdown to a non-AI bot has no negative effect. This means agents get markdown on every valid docs request, regardless of how they make it. #### **Handle 404s with markdown, not HTML** When an agent requests a page that doesn't exist, an HTML 404 page is useless. The agent can't parse it, and the conversation stalls. Instead, return a markdown response with actionable content: 1. Search your docs index for pages similar to the requested path. 2. If a result scores above a high-confidence threshold (0.99+), redirect the agent to the correct page. 3. Otherwise, return markdown-formatted suggestions listing the closest matches as links. 4. Return a 200 status, not 404. Agents need content they can act on. 5. Append a sitemap footer to every markdown response so agents can browse the full index if they hit a dead end. ### Layer 3: Tool access The most sophisticated agents don't scrape at all. They call tools. | Tool | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | MCP servers | Search docs, fetch specific pages, and list available content through a standard protocol. The [Vercel MCP server](https://vercel.com/docs/mcp) covers `vercel.com/docs`, `nextjs.org`, and the AI SDK docs. | | Search APIs | Return structured JSON results with canonical URLs, snippets, and freshness metadata. | | AI Chat | Conversational interface on docs sites backed by doc-aware tools: `search_docs`, `get_doc_page`, and `list_docs`. Available to both humans and agents. | ## Checklist > For the full scoring rubric (0–100) and detailed verification steps, see the [Agent-Readability Spec](https://vercel.com/kb/guide/agent-readability-spec). **Discovery** - Serve `/llms.txt` with a curated H1 + H2 index of your content - Publish `sitemap.xml` with accurate `lastmod` dates - Serve `/sitemap.md` with a semantic, markdown-formatted sitemap describing docs sections and pages - Document agent access policy in `robots.txt` - Add JSON-LD structured data (title, description, canonical URL, and breadcrumbs) to every page **Retrieval** - Return markdown for `Accept: text/markdown` with a `Vary: Accept` header - Generate `.md` endpoints for all content pages - Include frontmatter metadata (`title`, `canonical_url`, `last_updated`) in every markdown response - Add `` to HTML pages - Detect AI agents (user-agent matching, Signature-Agent header, heuristic fallback) and serve markdown automatically - Verify by appending `.md` to any page URL and confirming you get clean markdown with frontmatter **Tool access** - Expose search via MCP server or search API - Add a `SKILLS.md` or `AGENTS.md` file with install, config, and usage instructions for coding agents To get started, run through the [checklist](#checklist) above, then score your site against the [Agent-Readability Spec](https://vercel.com/kb/guide/agent-readability-spec). ## Further reading - [Agent-Readability Spec](https://vercel.com/kb/guide/agent-readability-spec): Full checklist and 0–100 scoring rubric - [Vercel MCP server](https://vercel.com/docs/mcp): Query Vercel, Next.js, and AI SDK docs via MCP - [llms.txt specification](https://llmstxt.org/): The llms.txt standard