Free growth diagnostic

Free tools / robots.txt AI rules generator

robots.txt rules generator for AI crawlers

IN SHORT

Pick which AI crawlers to block and copy the lines into your robots.txt. On Shopify, choose Shopify to get a complete robots.txt.liquid that keeps Shopify's default rules and adds yours.

1. Start from a preset
2. Tick the crawlers to block
3. Your platform
Add to the end of robots.txt
# OpenAI: Collects content that may train OpenAI models
User-agent: GPTBot
Disallow: /

# Anthropic: Collects content that may train Anthropic models
User-agent: ClaudeBot
Disallow: /

# Google: Controls whether your content helps train and improve Gemini
User-agent: Google-Extended
Disallow: /

# Apple: Controls use of your content for Apple's AI models
User-agent: Applebot-Extended
Disallow: /

# Common Crawl: Open web archive that many AI models are trained on
User-agent: CCBot
Disallow: /

Keep your existing rules and paste these lines at the end of your robots.txt file. Then open yourdomain.com/robots.txt to check it is live.

How to use it

  1. Start from a preset. Most businesses that want customers should pick the first one.
  2. Untick or tick individual crawlers. Search crawlers are marked, so you can see what blocking them costs.
  3. Copy the output. On most sites, add it to the end of robots.txt. On Shopify, create robots.txt.liquid as described under the output.
  4. Run the AI crawler checker to confirm the result.

Want the reasoning behind each choice? Read Should you block GPTBot?

Common questions

How do I block AI training but stay in ChatGPT search?

Block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave OAI-SearchBot, PerplexityBot, Googlebot and Bingbot alone. The first preset above does exactly that.

Why doesn't the generator write Allow rules?

A crawler that has its own User-agent group follows only that group and ignores the general User-agent: * rules. An Allow group for one bot could let it into your cart, checkout or admin pages. Crawlers you do not block simply follow your normal rules.

Does blocking Google-Extended affect my Google rankings?

Google says Google-Extended does not affect a site's inclusion or ranking in Google Search. It controls whether content helps improve Gemini.

How long does a robots.txt change take to work?

Crawlers re-read robots.txt on their own schedule. OpenAI says a change can take about 24 hours to take effect in ChatGPT search.