Free growth diagnostic

Free tools / AI crawler checker

AI crawler checker

IN SHORT

Enter your address to see whether Googlebot, Bingbot, ChatGPT's OAI-SearchBot, PerplexityBot, GPTBot and 6 more crawlers are allowed by your robots.txt, plus whether you have a sitemap, an llms.txt file and valid structured data.

Free, about 20 seconds, no login. We read your public robots.txt, sitemap, llms.txt and one page, as a crawler would.

What it checks

Each crawler does a different job. Blocking a search crawler removes you from that service's answers; blocking a training crawler only affects whether your content may be used to train models.

  • Googlebot (Google): Google Search, including AI Overviews and AI Mode.
  • Bingbot (Microsoft): Bing search, which Copilot also uses.
  • OAI-SearchBot (OpenAI): Shows websites in ChatGPT search answers.
  • PerplexityBot (Perplexity): Shows and links websites in Perplexity answers.
  • Claude-SearchBot (Anthropic): Improves search results for Claude users.
  • ChatGPT-User (OpenAI): Visits a page when a ChatGPT user asks about it.
  • GPTBot (OpenAI): Collects content that may train OpenAI models.
  • ClaudeBot (Anthropic): Collects content that may train Anthropic models.
  • Google-Extended (Google): Controls whether your content helps train and improve Gemini.
  • Applebot-Extended (Apple): Controls use of your content for Apple's AI models.
  • CCBot (Common Crawl): Open web archive that many AI models are trained on.

It also checks for a sitemap and an llms.txt file, whether your structured data parses, and the content basics a machine uses to understand the page.

What to do with the result

If a search crawler you want is blocked, remove that rule from robots.txt. To block training crawlers while staying in AI search, use our robots.txt AI rules generator. On Shopify, follow our robots.txt.liquid guide.

Common questions

How do I check if GPTBot is blocked on my site?

Open yourdomain.com/robots.txt and look for a group that starts with User-agent: GPTBot followed by Disallow: /. Or enter your address above: the checker reads the file and tells you the result for GPTBot and 10 other crawlers.

If I block GPTBot, will my site disappear from ChatGPT?

Not from ChatGPT search. OpenAI says OAI-SearchBot is the crawler that surfaces sites in ChatGPT search, and that each setting is independent. GPTBot is about training.

Can a site be blocked even if robots.txt allows the crawler?

Yes. A firewall, CDN or security plugin can block a crawler before it ever reads robots.txt. The checker reads your robots.txt rules; if it says allowed but you still never appear, check your firewall or CDN bot settings too.

Is the checker free?

Yes. There is no login. The full breakdown of failed checks is emailed to you if you want it.