Guides / Product data and technical SEO
Should you block GPTBot? AI crawlers and robots.txt explained
SHORT ANSWER
Block GPTBot only if you do not want your content used to train OpenAI's models. Blocking it does not remove you from ChatGPT search, because that is OAI-SearchBot's job, and OpenAI says each setting is independent. If you want to be found, allow OAI-SearchBot and PerplexityBot, and decide about training crawlers separately.
Many websites block every AI bot in robots.txt without knowing that the bots do different things. Blocking the wrong one can remove you from AI search answers. Blocking the right one can keep your content out of model training. This guide separates them, using what OpenAI and Perplexity say in their own documentation.
OpenAI: three agents, three jobs
- OAI-SearchBot is for search. OpenAI says it is used to surface websites in ChatGPT's search features, and a site that opts out will not be shown in ChatGPT search answers.
- GPTBot crawls content that may be used to train OpenAI's generative models. Disallowing it says your content should not be used for training.
- ChatGPT-User visits a page when a person asks ChatGPT something. OpenAI says it is not used to crawl the web automatically, and that robots.txt rules may not apply because a user started the visit.
OpenAI also says each setting is independent of the others, and that it can take about 24 hours for a robots.txt change to take effect in search.
Perplexity: two agents
- PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it if you want to appear.
- Perplexity-User fetches pages for a person's question. Because a user asked for it, Perplexity says it generally ignores robots.txt rules.
Google's guidance on its AI features says no special optimization is needed beyond foundational SEO. It lets you control what is shown with robots.txt directives for Googlebot and with snippet controls such as nosnippet, data-nosnippet and max-snippet.
A sensible default for a business that wants to be found
- Allow search crawlers (OAI-SearchBot, PerplexityBot, Googlebot) so your pages can appear in answers.
- Decide separately about training crawlers such as GPTBot. That is a business choice about your content, not a visibility switch.
- Check that your firewall or CDN is not blocking these bots even though robots.txt allows them. Our free scan tests robots.txt rules for 11 search and AI crawlers.
- Test after every change and wait a day before judging the result.
Names and rules change, so check each company's current documentation before you edit your file.
Common questions
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. OpenAI says OAI-SearchBot is what surfaces sites in ChatGPT search, and a site that blocks OAI-SearchBot will not be shown in search answers.
How do I see which bots my site allows?
Open yourdomain.com/robots.txt and look for User-agent lines naming each bot. Our free scan reads the file for you and reports the rules for 11 search and AI crawlers.
Sources
- Overview of OpenAI Crawlers, OpenAI
- Perplexity Crawlers, Perplexity
- AI features and your website, Google Search Central
Free tools: AI crawler checker · llms.txt generator · robots.txt AI rules generator