---
title: Make your site readable by AI agents
description: A practical guide to making your website discoverable, readable, and actionable for AI agents, grounded in over 1,000 real agent runs and a controlled experiment with the ora.ai research lab.
url: /kb/guide/make-your-site-readable-by-ai-agents
canonical_url: "https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents"
published: 2026-08-28
last_updated: 2026-08-28
authors: Vercel, Ora
related: []
install_vercel_plugin: npx plugins add vercel/vercel-plugin
---

AI agents visit websites on users' behalf to answer questions, integrate APIs, and complete tasks. Together with the [ora.ai](https://ora.ai/) research lab, we observed 1,033 agent runs across 25 sites and ran 190 controlled fetch probes. The trace study shows where agents went and which fetched pages grounded their answers. The controlled experiment tests which page configurations allowed two fetch clients to retrieve a planted answer.

In that controlled experiment, two kinds of change prevented answer retrieval. One hid the answer behind JavaScript from a fetch-only client. The other returned `403` to both clients. Agents must find and read a site's content before they can act on it.

> Numbers in this guide are tagged by source. \[STUDIES\] refers to research from the ora.ai research lab, covering [1,033 derived run summaries across 25 sites](https://github.com/agentready-org/standard/blob/main/data/traces.csv) and [190 controlled fetch probes across 19 site configurations](https://github.com/agentready-org/standard/blob/main/data/fetchability.csv). The underlying runs were traced turn by turn. Statements without a tag are recommendations rather than measured findings.

These results describe this sample rather than the web as a whole. The trace results are observational and run-weighted. Run counts vary by site, model, and harness. Six sites account for 691 of 1,033 runs, or 66.9%. [See the data](#site-weighting). The controlled probes test whether an answer is present in the returned content, not whether an agent understands it or completes a task.

This guide reflects data collected from June through August 2026. Ora continues to expand the datasets with more sites, models, harnesses, and probes. Future guide versions may update the figures and recommendations as the sample grows.

Every recommendation depends on the agent reaching the page and finding the answer in the returned HTML.

## Be reachable and citable

To reliably cite current first-party information, agents need a discoverable, fetchable page. They may receive or recall a URL, follow a link, or find a page through a crawl-built search index. Access controls can stop an agent from fetching a page while answering a user.

| Practice                                          | What it does                                                                            | What the runs show                                                                                                                                                                                          |
| ------------------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Publish the answer as first-party docs**        | Provides a canonical, current source that agents can fetch, ground answers in, and cite | \[STUDIES\] Of the 1,023 runs with an answer-provenance result, 844, or 82.5%, traced to a fetched page. Of those grounded answers, 394, or 47%, traced to a docs page. [See the data](#answer-provenance). |
| **Keep public pages reachable**                   | Lets agents fetch current information without an access challenge                       | \[STUDIES\] In the controlled probes, neither client could retrieve the answer when the server returned `403`. [See the data](#fetchability:-answer-retrieval).                                             |
| **Use robots.txt only for deliberate exclusions** | Tells compliant crawlers which paths not to fetch                                       | Not evaluated in these studies. Public sites do not need a `robots.txt` file to permit crawling. Add rules only for deliberate exclusions and monitor their effect.                                         |
| **List your pages in a sitemap.xml**              | Maps every page you publish for indexing                                                | \[STUDIES\] Agents fetched `sitemap.xml` directly in 4% of these runs. Its main role remains helping crawlers and search systems enumerate the site's canonical pages. [See the data](#sitemap-reach).      |

Publish each answer on one canonical, current first-party docs page. In these runs, docs were the most common grounding source. Of the 844 grounded answers, 394, or 47%, traced to a docs page.

Public sites do not need a `robots.txt` file to permit crawling. If your site already uses one for deliberate exclusions, audit it separately rather than adding agent-specific rules by default.

Then fetch a page with an agent user-agent string to inspect the response:

```bash
curl -iL -A 'Claude-User/1.0' https://example.com/your-page
```

The request should end with a 200 response. Intermediate 301 or 308 redirects are acceptable. The returned HTML should contain the answer. A 403, empty app shell, or JavaScript-only answer prevents fetch-only agents from reading the page.

This `curl` request cannot confirm whether a real agent is blocked. Bot blockers weigh where a request comes from more than its user-agent string, so your laptop may pass where the real agent is blocked, or the reverse. Server logs are the best operational check. OpenAI and Anthropic publish crawler IP ranges. Find requests from those ranges and confirm that they reach the intended page and receive the expected response. A `200` confirms access, but not that the returned content contains the answer or that the agent interpreted it correctly.

Agents can use `llms.txt` only after discovering and reaching the site.

## Put the answer where agents read

After reaching a page, an agent must be able to extract the answer from the returned content. In the studies, agents used the same homepage and docs pages as human visitors. Agent-specific resources such as `llms.txt`, Markdown mirrors, and JSON-LD can help agents discover or consume the same underlying information. **They do not replace accurate, fetchable source pages**.

If your site is client-rendered, making answers available in the initial HTML may require server-side rendering (SSR), prerendering, or a hybrid approach so agents that do not run JavaScript can read them. In the controlled probes, adding `llms.txt`, JSON-LD, sitemaps, Markdown mirrors, and similar features did not change whether the clients could retrieve an answer already present in server-rendered HTML.

### Agents read the same pages people do

Where agents go on your site, and what lets them pull the answer once there:

| Practice                            | What it does                                                                                                    | What the runs show                                                                                                                                                                                     |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Homepage in raw HTML**            | The front door, with real links to docs, product, and pricing                                                   | \[STUDIES\] Reached in 69% of runs, and it was the agent's first step in 92% of them. In 59% of the runs that reached it, the next hop was a docs page. [See the data](#homepage-reach-and-first-hop). |
| **Docs pages that answer**          | The surface agents actually read, where most tasks get resolved                                                 | \[STUDIES\] Agents reached docs in 855 of 1,033 runs, or 83%. Among those docs-reaching runs, they fetched an average of 3.4 docs pages. [See the data](#docs-reach-and-depth).                        |
| **Fetchable without JavaScript**    | The answer is in the initial HTML, server-rendered or prerendered, not injected by a single-page app after load | \[STUDIES\] In the controlled probes, a fetch-only client could not retrieve an answer available only through JavaScript. [See the data](#fetchability:-answer-retrieval).                             |
| **Correct, copyable code examples** | Provides runnable examples in fenced, language-tagged blocks that separate code from prose                      | \[STUDIES\] Of the 119 runs whose answer included code, 73, or 61%, contained lines taken verbatim from a page fetched on the site. [See the data](#code-lift).                                        |

### Link the files that describe your site

\[STUDIES\] Depending on the file type, 86-97% of attributed fetches came through links rather than guessed paths. [See the data](#discovery-file-linkage).

Link each resource from content agents already visited, such as the footer, page metadata like `<link rel="alternate">` in the head, or a resources section.

| File             | What it does                                               | What the runs show                                                                                                                                                                                                                                                                                                                                          |
| ---------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **llms.txt**     | A curated, described index of your key pages               | \[STUDIES\] Agents reached `llms.txt` through links in 356 of 416 attributed fetches, or 86%. Of the 329 runs that reached it, 325 had a scorable steering result. Of those 325 runs, 117, or 36%, later fetched a page it listed. Across all 329 reaching runs, 120, or 36%, grounded the final answer in its content. [See the data](#llms.txt-behavior). |
| `.well-known/*`  | Protocol and metadata discovery                            | \[STUDIES\] Agents reached a `.well-known` path in 234 of 1,033 runs, or 22.7%. [See the data](#.well-known-reach).                                                                                                                                                                                                                                         |
| **openapi.json** | Machine-readable API contract                              | \[STUDIES\] On the seven sites that shipped an `openapi.json`, agents fetched it in 112 of 541 runs, or 21%. [See the data](#openapi-reach).                                                                                                                                                                                                                |
| **JSON-LD**      | Adds machine-readable entity and product facts to the page | This study did not measure an improvement in agent accuracy or task completion from JSON-LD. Repeat critical facts in visible text because some converters omit script blocks.                                                                                                                                                                              |

When shipping an llms.txt, make it a lean, described index:

```plaintext
# Acme

> Acme ships a payments API for developers.

## Docs

- [Quickstart](https://example.com/docs/quickstart): first charge in one request
- [Authentication](https://example.com/docs/auth): API keys, scopes, rotation

## API

- [OpenAPI](https://example.com/openapi.json): full request and response contract
```

Add JSON-LD where it accurately represents the page, including the homepage when it contains organization or product facts. Keep critical information in visible text as well. This is a structured-data recommendation rather than a measured outcome from these studies.

### Agents frequently requested Markdown in the study harnesses

\[STUDIES\] Across the study harnesses, 2,259 of 3,470 web fetches, or 65.1%, requested Markdown. When a fetch specified any format, it selected Markdown in 2,259 of 2,361 cases, or 95.7%. In the controlled experiment, every non-blocking configuration exposed the planted answer to the applicable fetch client. [See the data](#markdown-requests). Make the HTML work before adding a Markdown mirror.

```html
<!-- docs/quickstart.html -->
<!-- Advertise the mirror so agents can discover it. -->
<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">
```

You can also provide the same relation in an HTTP `Link` header, allowing a CDN to advertise the mirror without changing page markup.

### Return accurate HTTP status responses

Return `404` for a missing path or a valid redirect to a replacement. Do not return `200` for a missing page, even if the body suggests alternatives. A soft 404 removes the reliable HTTP signal that distinguishes missing content from a live page.

When you throttle, return `429 Too Many Requests` with a `Retry-After` header instead of a silent challenge page. This gives clients an explicit, standards-based back-off signal.

If you offer Markdown through content negotiation, keep the response metadata accurate. A request with `**Accept: text/markdown**` should receive Markdown with `**Content-Type: text/markdown; charset=utf-8**`. Include `**Vary: Accept**` so caches keep the HTML and Markdown responses separate. If Markdown is unavailable, return HTML with its correct content type.

## Enable agents to act

If agents need to operate your product, expose the required actions through an authenticated API, MCP server, SDK, or CLI, and document the complete path from authentication to the first successful action.

Prioritize authentication an agent can complete and a public API with a machine-readable OpenAPI contract. For authentication, provide self-serve API keys with documented scopes or discoverable OAuth 2.0 metadata under `/.well-known`. \[STUDIES\] On the seven sites that shipped an `openapi.json`, agents fetched it in 112 of 541 runs, or 21%. [See the data](#openapi-reach).

The studies measured OpenAPI retrieval, not successful authentication or task completion. Follow the relevant protocol specifications and test the complete workflow with the agent clients you intend to support. Use a stable endpoint such as `/mcp` for a Streamable HTTP MCP server, but do not treat that path as a protocol requirement.

## Run additional agent-readiness checks

Agent readiness includes practices beyond those covered here. The Ora scanner evaluates more checks through its public methodology, per-check breakdown, and leaderboard. Its checks and weights may change between guide versions as new agent behavior is measured.

Scan any domain at [ora.ai](https://ora.ai) or from the API:

```bash
curl -X POST 'https://ora.ai/api/scan' \
  -H 'Content-Type: application/json' \
  -d '{"url": "vercel.com"}'
```

Example response (abridged). Scores change as the site is rescanned:

```json
{
  "score": 88,
  "grade": "A",
  "analysisStatus": "complete",
  "essentials": {
    "score": 87,
    "label": "Strong technical baseline"
  }
}
```

## References

- [What agents actually reach](https://ora.ai/blog/what-agents-actually-reach): an earlier field report from the same research program. The published dataset below is the source of the figures in this guide.
  
- [ora research](https://ora.ai/research): the ongoing research program behind the protocol.
  
- [Derived agent-run summaries](https://github.com/agentready-org/standard/blob/main/data/traces.csv) and [controlled fetch probes](https://github.com/agentready-org/standard/blob/main/data/fetchability.csv): the data behind every `[STUDIES]` number in this guide.    ## Changelog - **v1.0** · August 2026 · first public version.    ## Provenance ### Answer provenance Backs: "Of runs with an answer-provenance result, 82.5% traced to a fetched page. Among grounded answers, 47% traced to a docs page." Of the 1,023 runs with an answer-provenance result, where did the agent's final answer come from? Ten additional runs were marked not applicable. | Result                                    | Runs | Share          | | ----------------------------------------- | ---- | -------------- | | Answer traced to a page the agent fetched | 844  | 82.5% of 1,023 | | Not traceable to a fetched page           | 179  | 17.5% of 1,023 | | Not applicable                            | 10   | N/A            | Of the 844 grounded answers, the page types the wording traced to (top 3): | Page type       | Runs | Share | | --------------- | ---- | ----- | | Docs page       | 394  | 47%   | | llms.txt        | 120  | 14%   | | Other site page | 116  | 14%   | Docs lead every other type. The full 14-way split is in [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv).

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv). Among rows where `answer_grounded` is set, use `answer_grounded == 1` to identify grounded answers. Use `answer_ground_kind` for the page type.

### Site weighting

Backs: "Six sites account for 691 of 1,033 runs, or 66.9%."

The six sites with the most runs were:

| Site              | Runs    | Share     |
| ----------------- | ------- | --------- |
| zapier.com        | 124     | 12.0%     |
| monday.com        | 124     | 12.0%     |
| telnyx.com        | 122     | 11.8%     |
| attio.com         | 122     | 11.8%     |
| ora.ai            | 119     | 11.5%     |
| stripe.com        | 80      | 7.7%      |
| **Top six total** | **691** | **66.9%** |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): group rows by `domain`, sort run counts in descending order, and sum the first six.

### Sitemap reach

Backs: "Fetched directly in 4% of runs."

| Result              | Runs | Share |
| ------------------- | ---- | ----- |
| Fetched sitemap.xml | 43   | 4%    |
| Did not             | 990  | 96%   |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): `sitemap_reached == 1`.

### Homepage reach and first hop

Backs: "Reached in 69% of runs, and it was the agent's first step in 92% of them. In 59% of the runs that reached it, the next hop was a docs page."

The homepage funnel across 1,033 runs:

| Result                                  | Runs | Share        |
| --------------------------------------- | ---- | ------------ |
| Reached the homepage                    | 708  | 69% of 1,033 |
| Homepage was the first fetch            | 648  | 92% of 708   |
| Next hop after homepage was a docs page | 416  | 59% of 708   |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): `homepage_reached`, `homepage_first_turn`, and `next_hop_docs`, each rated against `homepage_reached == 1`.

### Docs reach and depth

Backs: "Reached in 83% of runs. Among docs-reaching runs, agents fetched an average of 3.4 docs pages."

| Metric                                  | Value              |
| --------------------------------------- | ------------------ |
| Runs that reached a docs page           | 855 (83% of 1,033) |
| Docs pages fetched, total               | 2,870              |
| Docs pages per docs-reaching run (mean) | 3.4                |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv). Select rows where `docs_reached == 1`, then calculate the mean of `docs_pages_fetched` over those runs.

### Fetchability: answer retrieval

Backs: "In the controlled probes, a fetch-only client could not retrieve an answer available only through JavaScript. Neither client could retrieve the answer when the server returned `403`."

One docs site was served under 19 configurations and probed across 5 topics by two clients: a plain non-JavaScript fetch and a JavaScript-executing one (190 probes). Only three configurations prevented at least one client from retrieving the answer, and they represent two kinds of change:

| Configuration                        | plain-fetch | js-fetch   |
| ------------------------------------ | ----------- | ---------- |
| js-only (answer inside a JS payload) | 0/5 passed  | 5/5 passed |
| bot-block (403 to agent UAs)         | 0/5 passed  | 0/5 passed |
| hostile (403 to agent UAs)           | 0/5 passed  | 0/5 passed |
| each of the other 16 configurations  | 5/5 passed  | 5/5 passed |

A JavaScript-hidden answer is unavailable to the fetch-only client. Blocking prevents both clients from retrieving the answer.

Reproduce over [fetchability.csv](https://github.com/agentready-org/standard/blob/main/data/fetchability.csv): `success` grouped by `variant` and `client`.

### Code lift

Backs: "When an agent's answer included code, 61% of the time it contained lines taken verbatim from a page it fetched on the site."

Among the 119 runs whose answer contained code:

| Result                                              | Runs | Share |
| --------------------------------------------------- | ---- | ----- |
| Included a line lifted verbatim from a fetched page | 73   | 61%   |
| Code, but not traceably lifted                      | 46   | 39%   |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): among rows where `answer_code_matched_fetch` is set, share `== 1`.

### Discovery-file linkage

Backs: "Depending on the file type, 86-97% of attributed fetches came through links rather than guessed paths."

For each discovery file, the share of fetches that arrived through a link rather than a guessed path:

| File           | Via link | Fetches |
| -------------- | -------- | ------- |
| openapi.json   | 97%      | 122     |
| .well-known/\* | 93%      | 383     |
| llms.txt       | 86%      | 416     |

These values count fetches, not unique runs. A run can fetch the same file more than once, so the 122 `openapi.json` fetches above correspond to 112 runs in the openapi reach section.

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): per file, `{file}_linked / {file}_fetches`.

### llms.txt behavior

Backs: "Agents reached llms.txt through links in 356 of 416 attributed fetches, or 86%. The steering and answer-grounding measures use different denominators."

How agents reached llms.txt (416 fetches):

| Path                   | Fetches | Share |
| ---------------------- | ------- | ----- |
| Arrived through a link | 356     | 86%   |
| Guessed the path       | 60      | 14%   |

Of the 329 runs that reached `llms.txt`, 325 had a scorable steering result. The answer-grounding measure is available across all 329 runs:

| Result                                | Runs | Share                        |
| ------------------------------------- | ---- | ---------------------------- |
| Fetched a page listed by `llms.txt`   | 117  | 36% of 325 scorable runs     |
| Drew the final answer from `llms.txt` | 120  | 36% of all 329 reaching runs |
| Steering result not applicable        | 4    | N/A                          |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv). Calculate via-link as `llms_txt_linked / llms_txt_fetches`. Calculate steering from `llms_followed_listing` where `llms_txt_reached == 1` and `llms_followed_listing` is set. Calculate answer-from-llms as `answer_ground_kind == 'llms_txt'` over `llms_txt_reached == 1`.

### .well-known reach

Backs: "Agents reached a .well-known path in 234 of 1,033 runs, or 22.7%."

| Result                     | Runs | Share |
| -------------------------- | ---- | ----- |
| Reached a .well-known path | 234  | 22.7% |
| Did not                    | 799  | 77.3% |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): `well_known_reached == 1`.

### openapi reach

Backs: "On the seven sites that shipped an openapi.json, agents fetched it in 112 of 541 runs, or 21%."

Restricted to the 7 sites that ship openapi.json (541 runs):

| Result               | Runs | Share |
| -------------------- | ---- | ----- |
| Fetched openapi.json | 112  | 21%   |
| Did not              | 429  | 79%   |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv): `openapi_reached == 1` over `site_ships_openapi == 1`.

### Markdown requests

Backs: "Agents requested Markdown in 2,259 of 3,470 web fetches, or 65.1%. When they specified a format, they chose Markdown in 2,259 of 2,361 fetches, or 95.7%."

| Request                                    | Calls | Share          |
| ------------------------------------------ | ----- | -------------- |
| Web fetches requesting Markdown            | 2,259 | 65.1% of 3,470 |
| Of fetches that named any format, Markdown | 2,259 | 95.7% of 2,361 |

Reproduce over [traces.csv](https://github.com/agentready-org/standard/blob/main/data/traces.csv). Calculate `markdown_calls / web_fetch_calls` and `markdown_calls / format_set_calls`.