Should You Allow AI Training Crawlers—or Block Them?

The three jobs an AI bot can do

The allow-or-block question gets muddled because "AI crawler" covers three different behaviors, and each one has different consequences.

OpenAI shows this split most clearly. OAI-SearchBot controls inclusion in ChatGPT search, GPTBot controls training use, and ChatGPT-User handles fetches that users request. According to OpenAI's crawler documentation, sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." Blocking GPTBot therefore does not keep you out of ChatGPT answers, and allowing it does not put you in. Our guide on how to get mentioned by ChatGPT goes deeper on OpenAI's bots.

Anthropic and Perplexity run their own crawlers too. Cloudflare classifies Anthropic's training crawler as training-only, so blocking it has no effect on search (Cloudflare, 2026). That classification does not describe how either operator separates search from user-requested fetches, so don't assume their split mirrors OpenAI's.

Why "block all AI bots" is the expensive default

The real tradeoff comes from mixed-use crawlers. Google, Apple, and Microsoft each run a single crawler (Googlebot, Applebot, Bingbot) that serves both search and AI training. Refusing that crawler means refusing search. For these operators, the training opt-out works through a no-training preference rather than by blocking the crawler. Google and Apple offer dedicated tokens for this: Google-Extended and Applebot-Extended.

Google positions Google-Extended as separate from Search ranking. Whether it affects AI Overviews and AI Mode is unsettled, which the FAQ below covers.

Cloudflare users should pay close attention to a change that took effect on September 15, 2026. Cloudflare's Training control now offers four settings: Allow, Disallow AI Training, Block on pages with ads, and Block. Block and Block on pages with ads now apply to Googlebot, Bingbot, and Applebot, so choosing either setting to stop AI will also remove you from those search engines (Cloudflare, 2026).

Disallow AI Training is the granular option. It publishes a no-training preference in robots.txt and keeps "Accountable" mixed-use crawlers allowed for search. It blocks every other training crawler, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI, and blocking those has no effect on search. Apple, Google, and Microsoft either honor the setting or have committed to honor it within a specified time frame, so behavior may differ by operator until those commitments take effect. "Accountable" is Cloudflare's own label, not a formal standard.

If you had the legacy "Block AI" setting, it migrated automatically to Search Allow, Training Disallow AI Training, and Agent Block on pages with ads. Check your dashboard to confirm that this matches what you intend.

Site owners already treat search and training differently. Fewer than 1% of Cloudflare sites block search bots, while 17% use some mechanism to block training (Cloudflare, 2026). Search access is close to universal. The real decisions are about training and agents.

Why "allow everything" isn't automatically right either

Blocking training is a mainstream choice. Among a curated set of reputable news sites, the share disallowing at least one AI agent rose from 23% in September 2023 to nearly 60% by May 2025 (Steinacker-Olsztyn, Gosain & Dao, 2025).

The deciding factor is the business model. Cloudflare's presets for new domains are a reasonable anchor:

• Sites without ads: Search Allow, Training Allow, Agent Allow.
• Ad-monetized sites: Search Allow, Training Disallow AI Training, Agent Block on pages with ads.

Cloudflare's rationale is that ad revenue depends on a human seeing the page. In its view, AI answers built on training data can substitute for visits, and agents fetch pages with nobody present to see the ads (Cloudflare, 2026). A publisher with rights or licensing priorities can reasonably disallow training. A SaaS or B2B brand that wants AI to explain its product accurately usually has much less to protect.

Decision table: goal to settings

This grid is an editorial synthesis built on the controls described above.

The last row is rarely the right choice. If you pick it, do so deliberately, not as the side effect of a single click.

What an opt-out can and can't do

Two limits apply to any opt-out, whichever row of the table you pick.

1. It is voluntary. robots.txt, standardized in RFC 9309, states a preference. It cannot enforce one. When researchers tested crawlers on their own sites, most large AI companies respected robots.txt, but several AI apps and crawlers did not, including crawlers from ByteDance (Liu et al., 2025). If you need real enforcement, block at the network edge.
2. It applies going forward, not backward. Measurement research has found LLMs reproducing content memorized from domains that restrict LLM bots (Cui et al., 2025). An opt-out controls future crawling. It does not remove what a model has already learned.

Agents are the grey zone

Agent controls lag behind search and training controls. Cloudflare offers no Disallow setting for agents because "the Internet does not yet have a well-established directive" for expressing that preference, and it plans to revisit the question as the IETF's AI preferences work matures. It also notes that agent fetches don't carry the same search-discoverability tradeoff, which makes blocking them lower-risk than blocking search bots (Cloudflare, 2026).

robots.txt may not reliably govern ChatGPT-User. Peer-reviewed measurement found cases where ChatGPT-User ignored robots.txt and accessed restricted content (Cui et al., 2025). In practice, your options are to accept agent fetches or block them at the network edge. Don't count on a polite opt-out.

What flipping a switch actually changes

Allowing a retrieval bot makes you eligible to appear in AI answers. It does not mean you will be indexed, mentioned, cited, or recommended. OpenAI's own wording is that allowing OAI-SearchBot "helps ensure" you can appear, not that you will.

Once you are eligible, other factors decide whether you get cited. Lee (2026) found that pages ranking in Google's top 3 were 7.82 times more likely to be cited than mid-ranked pages. The study is observational, so it shows correlation, not cause. Earned third-party coverage also appears to matter. For a deeper look at how classic search signals carry over, see Is AEO Really a New Discipline?. There is no evidence that allowing training crawlers, on its own, increases citations.

OpenAI's documentation includes three details that help you confirm a change actually took effect:

• Allow about 24 hours for a robots.txt change to affect ChatGPT search.
• Check your WAF or CDN. A bot-blocking rule can silently override a correct robots.txt. OpenAI recommends allowing its published IP ranges.
• Use noindex to hide a page, not a robots block. Blocking OAI-SearchBot can still leave a title and link visible if OpenAI learns the URL from a third-party search provider. Keep the page crawlable so the noindex tag can be read.

How to measure before and after

Bot hits in your logs do not tell you whether a change helped. A crawler visit is a discovery signal, not proof of indexing, citation, or conversion. User-agent strings can also be spoofed, so treat unverified hits with caution.

Measuring answers is harder than it looks because AI answers are unstable. In a SparkToro test, ChatGPT and Google's AI returned the same brand list less than 1% of the time across repeated prompts. A single screenshot proves very little. Our analysis of third-party listicles and GEO covers what that volatility means for tracking. A practical protocol:

1. Fix a prompt set of 20–50 real buyer questions, such as "best [category] tool for [use case]" and "[you] vs [competitor]."
2. Run each prompt several times per engine across ChatGPT, Gemini, Perplexity, Google AI Mode, and Google AI Overviews.
3. Record three signals separately as appearance rates: Mentioned (the brand is named), Cited (your URL is a source), and Rank (your position among recommended brands). These signals move independently. See AI mentions, citations, and recommendations for why the distinction matters.
4. Pair them with AI-referral traffic and conversions from your analytics.
5. Change one bot setting, wait at least a few days, and rerun the same prompts. A drop in the Cited rate after blocking a retrieval bot points to lost retrieval access. A flat result after allowing training is the expected outcome, not a failure.

Where Tideflow fits

The protocol above is hard to run by hand across five engines, and harder still once you connect answer visibility to crawler activity and revenue. Tideflow AI, currently in early access, tracks Mentioned, Cited, and Rank separately on fixed prompts across those engines. It logs AI crawler visits, labels them verified or unverified using verified bot metadata, and ties AI-referral traffic to conversions (Tideflow AI).

The measurements feed the rest of the workflow. Visibility gaps become targeted content, which is published through MCP into your own agent and site stack after you approve it. Tideflow then checks whether bots picked up the new pages and whether AI-referred visitors converted.

Two limitations matter here. Tideflow's collectors cannot see requests blocked upstream by your WAF, so a bot you have blocked will not appear in its logs. And Tideflow, like any tool, cannot guarantee that an engine will cite you.

Frequently Asked Questions

What if my site isn't on Cloudflare?

Write per-token rules in robots.txt. Allow the search bots, such as OAI-SearchBot, Googlebot, Bingbot, and Applebot. Disallow the training tokens you want to refuse, such as GPTBot, Google-Extended, Applebot-Extended, and Anthropic's training crawler. Because Cloudflare classifies Anthropic's training crawler as training-only, refusing it does not cost you search visibility. Then confirm that no WAF or hosting-level bot rule contradicts those rules. For agents or non-compliant crawlers, use network-level blocking, because robots.txt is only a request.

Does disallowing training affect AI Overviews or AI Mode?

For Google, this is unsettled. Google positions Google-Extended as separate from Search ranking, but there is no evidence that it removes content from AI Overviews or AI Mode. Cloudflare's Accountable criteria treat AI summaries as a separate opt-out that is still being built, with a stated goal of per-site control over summary inclusion by early next year (Cloudflare, 2026). For now, opting out of training and opting out of summaries are two different things.

Will allowing GPTBot help me get recommended by ChatGPT?

Allowing GPTBot has no demonstrated effect on ChatGPT citations. Inclusion in ChatGPT search is governed by OAI-SearchBot. Beyond eligibility, citations correlate with search ranking and appear to benefit from third-party corroboration.

If your audit confirms that retrieval bots can reach your site and your baseline still shows few citations, the next step is content and corroboration, not crawler settings. How to Get Mentioned by ChatGPT walks through that process.

Sources

• Tideflow AI, How to Get Mentioned by ChatGPT: A Tested, Step-by-Step Playbook; Is AEO Really a New Discipline?; Detect AI Bots; Are Third-Party Listicles Good GEO or Just AI-Era Link Building?; AI Visibility & Content Platform
• Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training (September 2026)
• Nicolas Steinacker-Olsztyn, Devashish Gosain, Ha Dao, Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web (October 2025)
• M.B.M. de Koster, G. Illyes, H. Zeller, L. Sassman, Robots Exclusion Protocol (September 2022)
• Enze Liu, Elisa Luo, Shawn Shan, Geoffrey M. Voelker, Ben Y. Zhao, Stefan Savage, Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers (October 2025)
• Jian Cui, Mingming Zha, XiaoFeng Wang, Xiaojing Liao, The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services (November 2025)