What happened
Cloudflare announced a new setting called Disallow AI Training, built to solve a specific problem: some crawlers, run by Google, Apple, and Microsoft, do double duty, indexing a site for search results and feeding AI training at the same time. Until now, a site owner who wanted to block AI training from one of these "mixed-use" crawlers had no way to do it without also losing that crawler's search indexing entirely. Refuse one use, and you refused both. The new setting separates the two. A site can now tell Cloudflare to keep a mixed-use crawler active for search while refusing it permission to train on the site's content, and Cloudflare publishes that preference into the site's robots.txt file automatically. Cloudflare created a label, "Accountable," for crawler operators willing to support this separation, and says Applebot and Googlebot already qualify today, each already offering a training opt-out that the companies state does not affect search ranking. Microsoft's Bingbot carries the same Accountable label, but the actual robots.txt control it needs is not built yet; Cloudflare says Microsoft is targeting early 2027, and until then Bing relies on a different, older mechanism. The part that matters most for a site owner who has already set something up: before September 15, choosing "Block" or "Block on pages with ads" for AI training did not stop Applebot, Bingbot, or Googlebot from crawling. As of September 15, it does. Those settings now block those crawlers outright, including for search, unless the site is moved to the new Disallow AI Training option instead. Cloudflare says it is migrating existing customer settings to the new system, but a business that manages its own Cloudflare configuration should not assume that migration already reflects what it actually wants.
Why it matters for business owners
Most businesses do not think about crawler settings at all until something breaks. This is exactly the kind of change that can quietly cost a business its search visibility without any error message, because the setting did not change on its own from the business's point of view. The infrastructure underneath it changed what the setting does. Cloudflare is not a niche tool. By its own account, it manages traffic in front of roughly a fifth of the web, including a meaningful share of small and medium-size business websites that use it for basic security and performance. If a business, or the agency or developer who set up its website years ago, ever turned on any form of AI blocking, that choice is worth revisiting now, not because AI blocking was a bad idea, but because the setting behind it works differently than it did a week ago. There is a second, larger point underneath the specific setting. Cloudflare's own CEO said this week that automated bot and AI traffic passed human traffic on the web back in May 2026, and that his company's data now shows close to 60% of website requests coming from bots rather than people. Whatever a business decides about training opt-outs specifically, the underlying fact is that a growing share of the audience for any website is no longer a person, and the tools that manage that audience are actively being rebuilt in real time.
What owners should not misunderstand
This is not evidence that Google, Apple, or Microsoft have started ignoring website owners' preferences, and it is not a reason to block every AI crawler by default. Google and Apple both state, and Cloudflare confirms, that opting out of AI training does not affect a site's search ranking. Blocking search crawlers entirely, by contrast, has an obvious and immediate cost: the site disappears from search results. It is also not evidence that this problem is solved. Cloudflare's own description of Apple, Google, and Microsoft's compliance is careful: it says they "honor or have committed (in a specified time frame) to honor" the new setting, and independent reporting on the announcement points out that two of the three companies' commitments carry no published deadline, and Microsoft's core capability is roughly a year and a half from actually existing. "Accountable" is a label Cloudflare created and grants, not an independently audited certification. Finally, this does not apply to every business. It applies specifically to websites that use Cloudflare for DNS, CDN, or bot management, and specifically to the mixed-use crawlers this announcement covers. A business on a different infrastructure provider, or one with no AI-blocking configuration at all, has nothing new to check here, though the same underlying question, whether and how AI systems should be allowed to train on the business's own published content, is worth having an answer to regardless of which infrastructure provider a site runs on.
The operational lesson
A setting a business turned on once, for a specific reason, does not necessarily keep doing the same thing forever. The infrastructure it depends on can change what that setting does, silently, without the business changing anything on its end. That is true of AI-crawler controls this week, and it is a pattern worth watching for in any AI-adjacent setting a business has configured and then stopped thinking about: privacy toggles, data-sharing preferences, model opt-outs, retention settings. The practical habit this points to is periodic verification, not one-time setup. A setting configured correctly a year ago is not guaranteed to still do what the business thinks it does today, especially anywhere AI policy or AI infrastructure is involved, because that is one of the fastest-moving parts of the technology stack right now.
What a serious business should do next
If your website uses Cloudflare, check your current AI bot settings this week. Look specifically at whether "Block" or "Block on pages with ads" is selected for AI training, and confirm whether that is still doing what you originally intended now that those settings also apply to Googlebot, Applebot, and Bingbot. If keeping search visibility matters to your business, and it does for almost every business that relies on organic traffic, switch to the Disallow AI Training setting rather than a blanket Block, so you get the training opt-out without risking your search indexing. After changing anything, verify the actual result. Fetch your site's live robots.txt file and confirm it reflects what you intended, rather than trusting a dashboard label alone. Independent analysis of this same announcement makes the same point: a setting name is not proof of the outcome. Decide, on purpose, what your business's actual position is on AI training on your own content. That is a business decision, not just a technical checkbox, and it is easier to make clearly before a setting changes again than to react to it each time. If your website is not on Cloudflare, ask whoever manages your hosting or CDN provider whether a comparable AI-training opt-out exists, and whether it can be set without affecting search visibility.
The Atlacis view
Atlacis helps business owners see past the assumption that a setting configured once will keep working the same way indefinitely. This week's Cloudflare change is a small, concrete example of a much larger pattern: the infrastructure and policies governing how AI systems interact with a business, its website, its data, and its workflows, are being rewritten while businesses are using them, not before. Atlacis helps owners map where their business actually depends on AI-adjacent infrastructure, whether that is a website's crawler settings, a vendor's data-handling terms, or an internal tool's access permissions, and build the habit of checking those dependencies on a real schedule instead of assuming a setup from a year ago still means what it used to.
The short version
- On September 15, 2026, Cloudflare launched a Disallow AI Training setting that lets a website opt out of AI training while staying indexed for search, confirmed directly by Cloudflare and independently corroborated by Search Engine Journal and other outlets.
- Before September 15, Cloudflare's Block and Block on pages with ads settings for AI training did not affect Googlebot, Applebot, or Bingbot. As of September 15, those same settings now block those crawlers entirely, including for search, unless a site switches to the new Disallow AI Training setting.
- Apple and Google already support a robots.txt training opt-out that they state does not affect search ranking. Microsoft's equivalent robots.txt control is not built yet; Cloudflare says Microsoft targets early 2027.
- Cloudflare's own wording is that Apple, Google, and Microsoft "honor or have committed (in a specified time frame) to honor" the new setting, not a flat, independently verified guarantee, and two of the three commitments carry no published deadline.
- This applies to websites that use Cloudflare, which the company says manages traffic for roughly a fifth of the web. A separate Fortune interview published today has Cloudflare's CEO stating automated bot and AI traffic passed human web traffic in May 2026.
- If your business's website is on Cloudflare, check your AI bot settings this week, switch to Disallow AI Training if you want to keep search visibility, and verify the change in your live robots.txt file rather than trusting the dashboard label alone.
Where ATLACIS can help
Sources
- Cloudflare: Have it both ways, stay discoverable in search while disallowing AI training (September 15, 2026)
- Cloudflare press release: Cloudflare Helps End the Search-or-AI-Training Tradeoff (September 15, 2026)
- Search Engine Journal: Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot (Matt G. Southern, September 15, 2026)
- Fortune: Bots now outnumber human traffic on the web. Cloudflare's CEO wants AI companies to pay for it (Diane Brady, September 18, 2026)