GPTBot is OpenAI's crawler for training its AI models. It's one of many AI bots that now visit business websites. Some feed AI search answers, and some only collect training data.
The difference matters. Block the wrong bot, and your site may drop out of AI search answers. This guide sorts the main bots by job and shows the rules for each choice.
Reporting note: We checked this guide against each vendor's public documentation on October 28, 2026.
Key takeaways
- GPTBot gathers content that may train OpenAI's models
- OAI-SearchBot, not GPTBot, decides if you can show in ChatGPT search
- Google says Google-Extended doesn't affect Search inclusion or ranking
- Robots.txt is a request that good bots follow, not a lock
- CDN and firewall settings can block bots your robots.txt allows
What is GPTBot, and what does it do?
OpenAI runs several bots, and each one has its own job. Its crawler documentation says GPTBot collects content that may train its foundation models. Blocking GPTBot tells OpenAI not to use your content for that training.
GPTBot doesn't decide whether you show up in ChatGPT search. A separate bot called OAI-SearchBot handles that. OpenAI says each setting is independent of the others.
Here's how OpenAI describes its four bots:
- GPTBot gathers content that may train OpenAI's models
- OAI-SearchBot finds sites to show in ChatGPT search answers
- ChatGPT-User visits a page when a person asks ChatGPT to
- OAI-AdsBot checks pages that advertisers submit as ChatGPT ads
OpenAI says sites that block OAI-SearchBot won't show in ChatGPT search answers. They may still appear as plain navigation links. So that's the bot to think hardest about.
Which AI crawlers affect AI search?
Most AI companies now split their bots by job. Search bots fetch pages so an AI tool can show and link them. Training bots gather content to build future models.
Scroll sideways to read the full table.
| Company | Search or answers | Model training |
|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot |
| Anthropic | Claude-SearchBot | ClaudeBot |
| Perplexity | PerplexityBot | None listed; PerplexityBot isn't used for training |
| Googlebot | Google-Extended |
Anthropic's crawler help page describes the same split. Claude-SearchBot indexes content for search. ClaudeBot gathers content that could help train its models.
Perplexity's crawler docs say PerplexityBot links sites in its answers. It isn't used for model training.
How Google handles AI crawling
Google works a little differently. Googlebot crawls for Search, and Google's AI features use that same index.
Its AI features guide sets one bar for supporting links. A page must be indexed and eligible to show a snippet.
Google-Extended is a separate token, not a separate bot. Google's common crawlers page says it doesn't affect inclusion in Google Search. It isn't used as a ranking signal either.
Google-Extended controls whether your content may train Gemini models. It also covers grounding in Gemini Apps and Vertex AI. Grounding is when a model pulls in Search content to answer a prompt.
Should a small business allow or block AI crawlers?
There's no single right answer. It depends on what you sell and how you feel about AI training. These questions can help you decide:
- Do you want a chance to appear in AI search answers?
- Are you comfortable with your pages training future models?
- Is your content the product, like paid courses or research?
- Does your host report heavy crawl load from any bot?
For most local and service businesses, the search bots are worth allowing. They give AI tools a way to find and cite your pages. That aims to put your business in front of people asking for what you do.
Training bots are a personal call. Blocking them doesn't remove you from the search side, based on each vendor's docs. Many owners allow search bots and block training bots.
Robots.txt rules for each choice
Robots.txt is a plain text file at the root of your site. It tells crawlers which pages they may visit. You write a group of rules for each bot by name.
Scroll sideways to read the full table.
| Your goal | Rules to add |
|---|---|
| Block OpenAI training | User-agent: GPTBot, then Disallow: / |
| Block Anthropic training | User-agent: ClaudeBot, then Disallow: / |
| Block Gemini training and grounding | User-agent: Google-Extended, then Disallow: / |
| Allow ChatGPT search | User-agent: OAI-SearchBot, then Allow: / |
| Allow Perplexity search | User-agent: PerplexityBot, then Allow: / |
A bot without its own group follows your catch-all rules, if you have any. So check that a rule for every bot doesn't block a search bot.
Changes aren't instant. OpenAI and Perplexity both say updates can take about 24 hours to take effect.
What robots.txt can't do
Robots.txt is a request, not a lock. Google's robots.txt guide says it's up to each crawler to obey the rules. Reputable crawlers do, but others might not.
It also won't keep a page out of search results. Google says a blocked page can still be indexed if other sites link to it. Use noindex or a password for that.
Some fetchers work differently, too. Perplexity says Perplexity-User generally ignores robots.txt, since a person asked for the page. OpenAI says robots.txt rules may not apply to ChatGPT-User for the same reason.
Check your CDN and firewall settings
Your robots.txt might say yes while your firewall says no. Many hosts and CDNs now offer AI bot controls. A blocked search bot can't read your pages, whatever robots.txt says.
Cloudflare is one example. Its July 1, 2026 changelog post split AI bot controls into Search, Agent, and Training. Each can be allowed, blocked, or blocked only on pages with ads.
New Cloudflare domains got new defaults on September 15, 2026. The defaults block Training and Agent bots on pages that show ads. Search bots stay allowed.
How to confirm the right bots can get in
- Open your robots.txt file in a browser
- Find the group for each bot you care about
- Check catch-all rules that apply to every bot
- Review AI bot settings in your CDN or firewall
- Search your server logs for each bot's name
- Compare visiting addresses with the vendor's published IP list
- Recheck a day after any change
OpenAI and Perplexity publish IP address lists for their bots. A visitor that uses a bot's name from another address may not be that bot.
AI companies add and rename bots over time. So check each vendor's bot page every few months. Update your file when a new name appears.
Crawler access is only the first step. Our AI search optimization work aims to help AI tools find and cite your pages.
Our ChatGPT SEO guide covers how to become a source customers may see cited. Our AEO audit guide shows how to check whether AI answers mention you.