Should you block AI bots from your site?

Whether to block AI bots depends on which ones, and most advice lumps three quite different things together. Some bots collect pages to train AI models. Some decide whether an assistant can find your site and cite it. And some only turn up because a person asked an assistant to look at your page. The companies say that blocking the first kind won't take you out of AI search. Blocking the second kind can: OpenAI says you won't be shown in ChatGPT search answers, and Anthropic says it may reduce your visibility in Claude's. The third kind may take no notice at all of your robots.txt file, the instructions your site gives to bots.

Our interest, stated first. We sell AI search visibility work, so we have an interest in sites being findable. Our own site allows every AI crawler named below, training crawlers included. That's a choice, not a tactic: nothing published says allowing training crawlers improves recommendations. If you're not sure what your site is blocking, ask us and we'll check.

This is part of our series checking what small and medium businesses are told about marketing, in which every claim is tested against what the company running the system publishes.

The three kinds, company by company

You need a bot's exact name to let it in or keep it out, and each company publishes its own. These are taken from their documentation as it stood on 27 September 2026: OpenAI, Anthropic, Perplexity, Google's common crawlers and user-triggered fetchers, and Apple.

Google and Apple don't run separate training crawlers at all. Google-Extended and Applebot-Extended are names you put in robots.txt to say how the content their ordinary crawlers collect may be used. Google says Google-Extended "doesn't have a separate HTTP request user agent string", and Apple says Applebot-Extended "does not crawl webpages".

Blocking training crawlers is your call, and it won't hide you

This is the kind most people mean when they say "block AI bots", and the companies all say it's separate from search.

• OpenAI says each of its settings "is independent of the others", so a site can allow OAI-SearchBot "in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training".
• Anthropic says that when a site restricts ClaudeBot, "it signals that the site's future materials should be excluded from our AI model training datasets".
• Apple says pages that disallow Applebot-Extended "can still be included in search results".

What none of them says is whether blocking training changes how their assistants describe or recommend your business, and nobody else has published anything on it either. So blocking training crawlers is a perfectly reasonable decision about how your own content gets used. As far as anything published shows, it won't make you more visible, or less.

What Google-Extended does, and doesn't do

Google-Extended is where the details matter most, because people often describe blocking it as "blocking Google's AI", and it isn't quite that. Google says it controls whether the pages Google collects can be used to train future versions of its Gemini AI models, and to help those models answer with up-to-date information from Google's index. Two things follow from that.

It doesn't touch Search. Google says: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

It won't take you out of AI Overviews or AI Mode, and it reaches beyond training. AI Overviews and AI Mode are part of Search, which Googlebot crawls. Google says your robots.txt rules for Googlebot are what control Search, and points to Google-Extended only "to limit AI training and grounding in some of Google's other systems". Blocking Google-Extended also stops your pages being used to answer questions in the Gemini app, so it isn't purely an opt-out from training.

Since 31 August 2026, every site has had a separate switch for Search. The Search generative AI control in Search Console lets a site keep its links and content out of AI Overviews, AI Mode and the generative AI features in Discover. Google says that excluding yourself means "You won't receive any traffic or impressions from these features", that the control "isn't used as a ranking or inclusion signal affecting other parts of Search", and that it "doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended". Unless you change it, you're included.

Blocking search crawlers is the one that costs you

This is the block that really does cost you visibility, and the companies say so, some more firmly than others.

• OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• Anthropic: disabling Claude-SearchBot "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results".
• Perplexity: "To ensure your site appears in search results, we recommend allowing PerplexityBot."
• Google and Bing: neither has a separate AI search crawler to block. Blocking Googlebot or Bingbot takes you out of their ordinary search results as well as their AI answers. Bing lists "Blocking Bingbot in your robots.txt file" among the things to avoid, and says the NOARCHIVE setting "prevents content from being used in Copilot responses and grounding results".

If you want to be found, these are the bots to leave alone. Letting them in is only the first step, though, and plenty else decides whether ChatGPT recommends you or your competitors.

When a person asks: user-triggered fetches

When somebody pastes your web address into an assistant, or asks it to check your prices, the assistant may go and read the page there and then. The companies treat that differently from crawling, and most of them say robots.txt may not stop it.

• OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply".
• Perplexity says of Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
• Google says its user-triggered fetchers, which include Google-Agent for "agents hosted on Google infrastructure", "generally ignore robots.txt rules" because "the fetch was requested by a user".
• Anthropic is the exception. It says its bots honour robots.txt, and that "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search".

So robots.txt isn't a reliable way to stop these fetches. A firewall rule can stop them, but think about who sets them off. It's usually a person asking about your business: a potential customer checking your opening hours, your prices or whether you cover their area. Block the fetch and the assistant can't read your page for them, so it answers from whatever else it can find. That's an odd way to greet someone who was about to become a customer.

You may already be blocking some of them

Plenty of sites block AI bots without anybody having decided to.

• Your robots.txt file. Anyone can read it by adding /robots.txt to the end of your domain. Look for any of the names in the table above with Disallow: / underneath, and for a User-agent: * group that disallows everything.
• Your website's security service, the service that sits in front of your site, such as Cloudflare, and can block bots before they ever reach it. Cloudflare's documentation said that from 15 September 2026 it would change its defaults for new domains, so that "bots classified as Training or as Agent" are blocked on pages that display ads, while "Search will remain allowed". Its Agent category covers "automated activity acting in real time on a person's behalf, such as chat fetch bots". Other services have their own settings, so check whichever one sits in front of yours.
• Security plugins and your hosting. Perplexity notes that a security service of this kind may need to be told explicitly to allow its bots.

If your site is missing from AI answers, a block like this is one of the first things to rule out, along with the usual reasons a website isn't showing up on Google.

How to decide

For most small and medium businesses that want to be found, the published evidence points one way:

• Allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
  What good looks like

  Your live robots.txt has no rule that blocks OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot from the pages you want found, including through a rule aimed at every crawler.

  Takes ten minutes. Covered in full above.

• Don't block the user-triggered fetchers unless you have a specific reason, such as load on your server. They are mostly acting for people asking about you.
  What good looks like

  ChatGPT-User, Claude-User and Perplexity-User are not blocked, unless you have written down a specific reason such as load on the server.

  Takes five minutes. Covered in full above.

• Decide about the training crawlers, GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, as a question of how your content gets used rather than one of visibility.
  What good looks like

  You have made a deliberate choice about GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, based on how you want your content used, and robots.txt reflects it.

  Takes twenty minutes to decide and edit. Covered in full above.

• If you block Google-Extended, remember that it also stops your pages being used to answer questions in the Gemini app.
  What good looks like

  If Google-Extended is blocked, you have done it knowing it also stops your pages grounding answers in the Gemini app, not only training.

  Takes five minutes. Covered in full above.

• If you want out of AI Overviews and AI Mode, use the Search generative AI control in Search Console, and expect to lose that traffic.
  What good looks like

  If you have used the Search generative AI control in Search Console, you did so knowing it removes your content from those features, and you are watching the traffic from them.

  Takes ten minutes, plus a monthly look afterwards. Covered in full above.

• Check your website's security service settings, not just robots.txt.
  What good looks like

  Your hosting, firewall or network service is not blocking or challenging the crawlers you allowed in robots.txt, and you have checked its bot settings yourself.

  Takes twenty minutes. Covered in full above.

• Afterwards, check what the platforms report: Bing's AI Performance report and Search Console's Generative AI performance report.
  What good looks like

  A month after any change, you have looked at Bing's AI Performance report and Search Console's generative AI performance report for a change in citations or impressions.

  Takes twenty minutes a month. Covered in Bing's AI Performance report: how to see where Copilot cites your website.

If your content is the product, as it is for a news site or paid research, you may weigh training quite differently, and that's a perfectly legitimate business decision. It's simply a different decision from the one facing a local business that wants to be found.

What nobody has published yet

• No AI company publishes whether blocking its training crawler changes how its assistant describes or recommends a business.
• OpenAI says robots.txt rules "may not apply" to ChatGPT-User, without saying when they do.
• We haven't checked the default bot settings of every hosting, security or network service, and only Cloudflare's are cited here.

So, should you block them?

It depends which ones. Blocking training crawlers is a reasonable choice about how your content is used, and the companies say it won't remove you from their search. Blocking search crawlers costs you visibility in AI answers, and blocking Googlebot or Bingbot costs you ordinary search as well. User-triggered fetches may ignore robots.txt, and blocking them at a firewall stops an assistant reading your page for the very person who asked. For a business that wants to be found, the published evidence says let the search crawlers in.