AI CRAWLERS
Should you block GPTBot and other AI crawlers?
You can block GPTBot with two robots.txt lines, User-agent: GPTBot and Disallow: /. That opts your content out of OpenAI model training without removing you from ChatGPT search, which uses a different bot, OAI-SearchBot. Allow the search bots if you want AI visibility. Digimetrics' AI Readiness Check shows which bots your site currently allows.
- Free for 7 days
- Every tool unlocked
- No card needed
Updated 23 October 2026 · Digimetrics editorial team
The short answer
- AI companies split crawlers by job: training bots (GPTBot, ClaudeBot), search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User).
- Blocking a training bot opts you out of future training. Blocking a search bot removes you from that assistant's search answers. They are independent settings.
- User-triggered fetchers may not follow robots.txt, because a person asked for the page: OpenAI and Perplexity both say so.
- Google-Extended limits Gemini training and grounding but does not affect Google Search or AI Overviews.
- Check firewalls and CDNs too: they can block bots your robots.txt allows.
AI crawler user agents: what each one does
Each major AI company documents separate user agents for training, search and user-requested fetches. Knowing which is which is the whole decision.
The table below uses each company's own documentation, checked in October 2026. The robots.txt token is what you write after User-agent:.
| Robots.txt token | Company | Job | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train its generative AI foundation models | Content signaled as not for training; ChatGPT search unaffected |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search | Not shown in ChatGPT search answers (may still appear as a navigational link) |
| ChatGPT-User | OpenAI | Visits pages for user actions in ChatGPT and Custom GPTs | User-initiated, so robots.txt rules may not apply |
| ClaudeBot | Anthropic | Collects web content that could contribute to model training | Future content excluded from training datasets |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results | May reduce visibility in Claude's search results |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | Claude cannot retrieve your page for those user queries |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for foundation-model training | Lose eligibility for Perplexity search results |
| Perplexity-User | Perplexity | Fetches pages during a user's question | Generally ignores robots.txt because a user requested it |
| Google-Extended | Controls use for Gemini training and grounding in Gemini Apps and Vertex AI | No effect on Google Search inclusion or ranking | |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | Removed from Google Search crawling, AI features included |
Should I block GPTBot?
Block GPTBot if you do not want your content used to train OpenAI's models and you accept that trade-off. Leave it allowed if broad model familiarity with your brand matters more to you. Either way, keep OAI-SearchBot allowed if you want ChatGPT search traffic.
This is a policy decision, not a technical one, and reasonable businesses land on both sides. Here is how it usually breaks down:
| Your situation | Usual choice for training bots | Usual choice for search bots |
|---|---|---|
| Publisher or creator whose content is the product | Block GPTBot, ClaudeBot, Google-Extended | Often allow, to keep referral traffic from AI search |
| Business that wants AI assistants to recommend it | Often allow | Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Site with paywalled or licensed content | Block | Decide per section with path rules |
| Internal tools, staging or private areas | Block | Block (and use authentication, not just robots.txt) |
The common mistake is blocking everything with “AI” in its name. Blocking PerplexityBot to avoid training, for example, removes you from Perplexity search even though Perplexity says that bot is not used for foundation-model training.
Robots.txt rules for AI crawlers you can copy
Write one group per user agent. The pattern below opts out of training while staying visible in AI search; adjust the Allow and Disallow lines to your policy.
Place this in robots.txt at your top-level directory. Anthropic notes you must do this for every subdomain you want to opt out. Each bullet is one line of the file; leave a blank line between groups.
- User-agent: GPTBot
- Disallow: /
- User-agent: ClaudeBot
- Disallow: /
- User-agent: Google-Extended
- Disallow: /
- User-agent: OAI-SearchBot
- Allow: /
- User-agent: Claude-SearchBot
- Allow: /
- User-agent: PerplexityBot
- Allow: /
Blocking GPTBot with Nginx or Cloudflare
Robots.txt is the documented method and the one AI companies say they honor. Server and CDN blocks are stronger but blunter, and Anthropic warns IP blocking may not reliably opt you out.
Nginx. You can match the user agent in your server block and deny the request, for example with an if condition on $http_user_agent matching GPTBot that returns a forbidden status. This stops the fetch outright but does not tell the bot your preference, and user-agent strings can be spoofed. Anthropic specifically says blocking its IP addresses may not work correctly or persistently, because it stops its bots reading your robots.txt. Use server rules as a backstop, not a replacement.
Cloudflare. Cloudflare offers AI bot policies under Security Settings with presets by behavior: Search, Agent and Training, each with block on all pages, block on pages with ads, or allow. Its documentation says that from September 15, 2026, new domains default to blocking Training and Agent bots on pages with ads while Search stays allowed, and mixed-purpose crawlers that combine search and training are blocked by all training-blocking configurations. If you want AI search visibility, check that your Search setting is Allow.
Verify real bots. OpenAI, Anthropic and Perplexity publish IP ranges for their bots. If you allow a bot at the firewall, combine the user agent with those IP ranges, as Perplexity recommends, so impostors do not get through.
Does blocking AI crawlers hurt SEO?
Blocking training bots does not affect Google rankings. Blocking Googlebot does, and blocking AI search bots removes you from those assistants' answers.
Google states that Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. So opting out of Gemini training is safe for your Google SEO. AI Overviews and AI Mode are part of Search and are controlled by Googlebot plus snippet controls like nosnippet, which we cover in how to appear in Google AI Overviews.
The real cost of over-blocking shows up outside Google. Block OAI-SearchBot and OpenAI says you will not be shown in ChatGPT search answers. Block PerplexityBot and you lose Perplexity results. That is lost visibility in exactly the places buyers now ask for recommendations; see how to rank in ChatGPT and Perplexity SEO.
What tools help you manage AI crawlers?
You need a way to see what your site currently allows, a way to edit it, and a way to see whether the change affected AI visibility.
The AI Readiness Check shows bot access, llms.txt and structured data in one score. The Full Site Audit includes AI-bot access in its key signals and tags fixes as GEO. After changes, the AI Mentions Tracker re-asks your buyer questions across GPT-4o, Claude, Gemini, Perplexity and Google AI Overview. For the rest of the AI visibility work, follow the AI SEO checklist.
Note that llms.txt does not control access: it is a guide for agents, not a permission file. Our llms.txt guide explains the difference.
HOW IT WORKS
How to block GPTBot (and keep ChatGPT search) in five steps
- 1
See what you allow today
Run the AI Readiness Check (up to a minute). It reports whether AI bots are allowed in, alongside llms.txt and structured data.
- 2
Decide your policy per bot type
Training, search and user fetchers are separate decisions. Use the table below to choose. Write the decision down so the next person who edits robots.txt or the CDN knows why each bot is allowed or blocked.
- 3
Edit robots.txt at your site root
Add a group per bot, for example User-agent: GPTBot then Disallow: /, and a separate User-agent: OAI-SearchBot group with Allow: /. Repeat for each subdomain you want covered.
- 4
Match your CDN and firewall to the policy
Make sure Cloudflare or your WAF is not blocking the search bots you allowed, or allowing what you meant to block.
- 5
Allow time, then re-check
OpenAI says robots.txt changes take about 24 hours to reach its search systems; Perplexity says up to 24 hours. Re-run the check, then track AI mentions with the AI Mentions Tracker.
QUESTIONS PEOPLE ASK
Frequently asked questions
Still unsure? Email support@digimetrics.ai and a person will reply.
How do I block GPTBot in robots.txt?
Add two lines to robots.txt at your site root: User-agent: GPTBot and Disallow: /. To block only part of the site, use a path, such as Disallow: /blog/. OpenAI says GPTBot respects this as a signal that content should not be used for training.
Will blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search. OpenAI says GPTBot and OAI-SearchBot are independent; search uses OAI-SearchBot. Blocking GPTBot affects training use only.
What is the GPTBot user agent string?
OpenAI gives an example string ending in compatible; GPTBot followed by a version and the URL openai.com/gptbot, and notes the version number may change. Use the token GPTBot in robots.txt rather than the full string.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. Use nosnippet or noindex to limit what Search shows.
Can AI bots ignore robots.txt?
User-triggered fetchers may. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person requested the page. The automatic crawlers each company documents say they follow robots.txt.
Can I slow AI crawlers down instead of blocking them?
Anthropic supports the non-standard Crawl-delay directive for its bots, for example Crawl-delay: 1 under User-agent: ClaudeBot. Support varies by company, so check each bot's documentation.
See which AI bots can read your site right now.
Run the AI Readiness Check free for 7 days, no credit card.