--- title: Making your catalog discoverable to AI agents description: Agentic commerce readiness starts with the initial HTML response. Server-render your product catalog, ship JSON-LD, and verify AI agents can read it. url: "https://vercel.com/kb/guide/agentic-commerce-readiness" published: 2026-10-01 last_updated: 2026-10-01 authors: Vercel install_vercel_plugin: npx plugins add vercel/vercel-plugin --- AI shopping agents read your product pages before any JavaScript runs. The crawlers behind them fetch your HTML and never execute it, so a storefront that assembles its catalog in the browser returns a shell with no product name, price, or availability in it. Agentic commerce readiness means the first HTML response already states what the product is, what it costs, whether it's in stock, and who sells it. Here's how to get there, starting with the rendering decision everything else depends on. ## **What agentic commerce readiness means for your product catalog** For a product catalog, readiness is decided by the first HTTP response. Only one of the two kinds of AI traffic hitting your storefront is limited to it. Background crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot fetch HTML without executing it. Browser-based agents run a real browser, so they see what a person sees. Build for the crawlers, because anything they can read, the browser-based agents can read too. Across Vercel's network in December 2024, GPTBot generated [569 million requests](https://vercel.com/blog/the-rise-of-the-ai-crawler) and Anthropic's Claude crawlers generated 370 million. Combined with AppleBot and PerplexityBot, those crawlers accounted for roughly 28% of Googlebot's volume over the same period. Once a crawler reaches the page, it looks for structure in three places: - [**Schema.org**](http://Schema.org) **Product markup:** A JSON-LD (JavaScript Object Notation for Linked Data) script tag in the initial HTML that names the product, price, availability, and seller. - **Open Graph tags:** Metadata that agents and platforms extract to build previews. - **A product feed:** Catalog data you push to a shopping surface rather than something the crawler finds on your site. An `llms.txt` file is a lower priority than all three. Across [137,000 sites](https://ahrefs.com/blog/llmstxt-study), 97% of `llms.txt` files were never fetched at all. Agents that did reach one got there through links from other pages in [86% of attributed fetches](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). Ship the HTML and schema layers first, then add a linked `llms.txt` if you want it. ## **Why rendering decides whether AI agents can read your catalog** Structured data only reaches a crawler if it's in the response body. What a non-rendering crawler receives depends on your rendering strategy. Here's what each strategy sends a crawler: | Strategy | What the crawler receives | Agent-readable? | | ----------------------------------------------------------- | ------------------------------------------------------ | --------------- | | Static generation and Incremental Static Regeneration (ISR) | Complete pre-generated HTML, served from the CDN cache | Yes | | Server-side rendering (SSR) | Complete server-rendered HTML, built per request | Yes | | Partial Prerendering (PPR) | A static shell, followed by streamed dynamic content | Not by default | | Client-side rendering (CSR) | An HTML shell and script tags, with no product data | No | Under PPR, Next.js [checks the user agent](https://nextjs.org/docs/app/getting-started/caching) and serves a complete document to bots it recognizes. Its default list was built for search engines and social unfurlers, so GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot don't match it and instead receive the streaming shell. To bring AI crawlers into that blocking render, extend the list in `next.config.ts`: ```tsx import type { NextConfig } from 'next' const config: NextConfig = { htmlLimitedBots: /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|[\w-]+-Google|Google-[\w-]+|Bingbot|applebot|Twitterbot|Slackbot|Discordbot|LinkedInBot|facebookexternalhit/, } export default config ``` Setting `htmlLimitedBots` replaces the default list rather than adding to it, so carry over the entries you still rely on. Because the shell is rebuilt instead of reused, a shell component that depends on build-time-only data can fail at request time, and a page that loads for a person can error for a crawler. The same rendering change affects human search traffic. Ruggable rebuilt its frontend on Next.js and Vercel and saw a [300% increase](https://vercel.com/customers/how-ruggable-saw-more-organic-clicks-by-optimizing-their-frontend) in unbranded search clicks, and Plex refactored to SSR and on-demand ISR for a [6× increase in impressions](https://vercel.com/customers/how-plex-6x-their-impressions-deploying-next-js-on-vercel). Both came from the same mechanism that feeds AI crawlers. ## **An agentic commerce readiness checklist for Next.js storefronts** With rendering settled, work through these five in order. ### **1\. Allow AI crawlers in robots.txt and confirm your firewall** A `robots.txt` file that predates AI crawlers can block them through a catch-all disallow rule. The App Router's [robots.ts convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) makes the policy explicit and reviewable in code. Declare the agents you want on your discovery routes: ```tsx import type { MetadataRoute } from 'next' export default function robots(): MetadataRoute.Robots { return { rules: [ { userAgent: ['OAI-SearchBot', 'GPTBot', 'ClaudeBot', 'PerplexityBot'], allow: ['/products/', '/collections/'], disallow: ['/checkout', '/account', '/admin'], }, ], sitemap: '', } } ``` `robots.txt` covers only one layer of access. Vercel's [AI bots managed ruleset](https://vercel.com/docs/bot-management) identifies known AI crawlers and can deny them at the firewall regardless of what `robots.txt` says. The ruleset is inactive by default, shown as **Allow** in the dashboard. Check that it isn't blocking crawlers from your catalog routes. ### **2\. Add JSON-LD structured data to every product page** The JSON-LD script tag has to ship in the initial HTML. Injecting it after load keeps it from reaching any non-rendering crawler. Render it inside the page component, following the [App Router pattern](https://nextjs.org/docs/app/guides/json-ld): ```tsx export default async function Page({ params }) { const { id } = await params const product = await getProduct(id) const jsonLd = { '@context': 'https://schema.org', '@type': 'Product', name: product.name, description: product.description, image: product.image, sku: product.sku, brand: { '@type': 'Brand', name: product.brand, }, offers: { '@type': 'Offer', price: product.price, priceCurrency: 'USD', availability: 'https://schema.org/InStock', priceValidUntil: product.priceValidUntil, seller: { '@type': 'Organization', name: 'Your Store', }, }, } return ( <>