--- title: How to block bots from OpenAI GPTBot description: Learn how to use the Vercel WAF to block, rate limit, or challenge traffic from OpenAI GPTBot. url: /kb/guide/how-to-block-bots-openai-gptbot canonical_url: "https://vercel.com/kb/guide/how-to-block-bots-openai-gptbot" published: 2025-11-03 last_updated: 2025-11-10 authors: Lee Robinson related: - /blog/the-rise-of-the-ai-crawler - /docs/concepts/functions/edge-middleware/firewall - /blog/how-google-handles-javascript-throughout-the-indexing-process install_vercel_plugin: npx plugins add vercel/vercel-plugin --- ## Related pages > **For AI agents:** Follow these links to understand how this page connects to the rest of the Vercel ecosystem. For the full cross-link map (inbound, outbound, prerequisites, and semantic neighbors), see the .graph.md link below. - [Bot Management](https://vercel.com/docs/bot-management?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) — Learn how to manage bot traffic to your site. - [Preventing infrastructure abuse with Vercel Firewall](https://vercel.com/blog/preventing-infrastructure-abuse-with-vercel-firewall?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) - [The three types of AI bot traffic and how to handle them](https://vercel.com/blog/the-three-types-of-ai-bot-traffic-and-how-to-handle-them?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) - [New one-click AI bot managed ruleset](https://vercel.com/changelog/new-one-click-ai-bot-managed-ruleset?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) - [Vercel WAF upgrade brings persistent actions, rate limiting, and API control](https://vercel.com/blog/vercel-waf-upgrade-brings-persistent-actions-rate-limiting-and-api-control?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) - [Attack Mode](https://vercel.com/docs/vercel-firewall/attack-mode?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) — Learn how to use Attack Mode to help control who has access to your site when it's under attack. - [Block, rate limit, and challenge traffic with the Vercel Firewall](https://vercel.com/changelog/block-rate-limit-and-challenge-traffic-with-the-vercel-firewall?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) - [How to protect your AI app from bots](https://vercel.com/kb/guide/how-to-protect-your-ai-app-from-bots?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=related) — Learn how to protect your AI app from bots, scrapers, and abuse using Firewall, BotID, and more. Full cross-link map for this page: [/kb/guide/how-to-block-bots-openai-gptbot.graph.md](/kb/guide/how-to-block-bots-openai-gptbot.graph.md?from=related&source_path=%2Fkb%2Fguide%2Fhow-to-block-bots-openai-gptbot&source_site=vercel-kb&relationship=graph) OpenAI has an AI-based crawler (also referred to as “GPTBot”) that fetches and indexes website content, which is then used to power the training data of models like GPT series, o series, and more. According to [our research](https://vercel.com/blog/the-rise-of-the-ai-crawler), OpenAI generates hundreds of millions of monthly requests, making it the most active AI crawler on the web. ## Good bots vs. bad bots - **Good Bots**: Examples include Googlebot, Applebot, and Bingbot. They’re typically transparent about their intentions, respect robots.txt, and help improve search engine visibility. - **Bad Bots**: These bots may scrape content without permission, inflate server usage, or perform malicious activities. Even certain “legitimate” AI crawlers can become unwanted if they exceed fair-use limits or repeatedly crawl error pages (e.g., excessive 404 requests). We'll focus on the OpenAI GPTBot here. ## What is the Vercel Firewall? [Vercel Firewall (WAF)](https://vercel.com/docs/concepts/functions/edge-middleware/firewall) is a Web Application Firewall service that lets you: - **Log or block requests** that match certain criteria (IP address, user agent, request path, geolocation, etc.). - **Challenge suspicious traffic** with an automated check (e.g., requiring the visitor’s browser to pass a JavaScript challenge). - **Rate limit** excessive or malicious requests. All configuration changes are applied globally within ~300ms, and can be rolled back instantly. ## Identifying the Bot ### User Agent OpenAI identifies itself in the `User-Agent` header. Check your Firewall traffic for entries like `Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)`. If you see repeated requests from that User-Agent, you can be confident it’s OpenAI. Note that AI crawlers occasionally use different sub-variants of user agents over time. ## Taking action with the Firewall Vercel Firewall allows a few approaches: ### Rate Limit If you don’t mind occasional indexing by OpenAI models, but want to cap how frequently it crawls your site, you can **rate limit** requests from the “GPTBot” user agent. This helps prevent resource overload while still allowing some AI-based traffic. ### Challenge If you want to slow down less-dedicated bots, you can **challenge** requests from certain user agents. The challenge forces a short browser-based security check. Most legitimate human traffic (with real browsers) will pass automatically, but a bot that can’t solve or respond to the challenge will be blocked. ### Persistent Blocking (No Charge) If you want to fully prevent GPTBot from crawling your site—and avoid incurring data transfer or function usage for these requests—you can **persistently block** it. Requests that match your block rule won’t reach your Vercel Functions or static pages, so you won’t be charged. ## Templates Vercel provides several [Firewall Templates](https://vercel.com/templates/vercel-firewall) you can clone or learn from: 1. [**Rate Limit API Requests**](https://vercel.com/templates/other/rate-limit-api-requests-firewall-rule) 2. [**Block OFAC-Sanctioned Countries**](https://vercel.com/templates/other/block-ofac-sanctioned-countries-firewall-rule) 3. [**Block WordPress URLs**](https://vercel.com/templates/other/block-wordpress-urls-firewall-rule) 4. [**Block Bad Bots**](https://vercel.com/templates/other/block-bad-bots-firewall-rule) 5. [**Block AI Bots**](https://vercel.com/templates/other/block-ai-bots-firewall-rule) To block OpenAI GPTBot specifically, you can start with the “Block AI Bots Firewall Rule” template and modify it for the defined user agent. ## Example: Creating a Custom Rule to Block OpenAI GPTBot 1. **Navigate to “Firewall”** in your Vercel project dashboard. 2. **Click “Add Rule”** (or “Create Rule” if using templates). 3. **Select the “Block AI Bots” template** (or a blank custom rule). 4. **Match Condition**: For “User Agent” contains “GPTBot”. 5. **Action**: Choose “Deny” (to block) or “Challenge” (to verify) or “Rate Limit” (to limit). 6. **Review & Publish** changes. **Note**: Once published, changes take effect globally in ~300ms. You can always roll back if you block or challenge traffic unintentionally. ## Resources - [**How Google Handles JavaScript Throughout the Indexing Process**](https://vercel.com/blog/how-google-handles-javascript-throughout-the-indexing-process) - [**The Rise of the AI Crawler**](https://vercel.com/blog/the-rise-of-the-ai-crawler) These posts provide background on how advanced bots handle JavaScript, distribution, and crawling inefficiencies (e.g., excessive 404s).