Block AI crawlers with AI Crawl Control or the AI bot policies in Security Settings — both are on every plan, including Free. A managed robots.txt only states a preference and does not enforce anything.
The confusion is worth naming up front, because it is the reason so many sites believe they are protected and are not. Cloudflare ships three separate features that all get filed under “blocking AI bots”, and they sit at different points on a spectrum from asking politely to actually refusing.
Cloudflare is unusually blunt about the first one. Its robots.txt documentation says compliance is voluntary — the file expresses your preferences but does not prevent crawlers from accessing your content at a technical level, and some operators will disregard directives like Disallow: / and crawl anyway. The docs then point straight at AI Crawl Control for enforcement, and note the two are designed to be used together: state the preference, then enforce it.
If you do turn the managed file on, it behaves considerately. Cloudflare detects whether your origin already serves a robots.txt — verified by an HTTP 200 — and prepends its managed content to yours rather than replacing it, so your existing Disallow lines and sitemap reference survive.
What changes on September 15, 2026
This is the part with a deadline attached, so it goes before the how-to. Cloudflare’s Block AI Bots documentation sets out three changes landing on that date:
- New domains get new defaults. Bots classified as Training or as Agent will be blocked on pages that display ads. Search remains allowed.
- Mixed-purpose crawlers stop slipping through. A crawler used for both Search and Training will be blocked by every configuration that blocks AI training — including the legacy option. Today that legacy setting explicitly excludes them.
- The old switch is deprecated. “Block AI bots” is marked as deprecating on the same date, superseded by the behaviour presets below.
Cloudflare notes that before September 15 all customers can opt out of the new defaults from Security Settings. If you run an ad-supported site and have deliberately left AI access open — because you have an agreement, or because the referral traffic is worth more to you than the crawl — that opt-out is the thing to handle this week rather than next month.
The three behaviours you can set separately
Rather than one on/off switch, Cloudflare now splits AI activity by what the crawler is doing with your content. All customers get these presets, and each is configured independently:
| Behaviour | What it covers | Typical call |
|---|---|---|
| Search | Crawlers collecting or indexing your content to answer questions about it later | Usually allow — this is where citations come from |
| Agent | Real-time activity on a person’s behalf, such as chat fetch bots and browser-use agents | Depends on whether those visits convert |
| Training | Crawlers taking content to train or fine-tune a model, including mixed-purpose crawlers | The one most publishers block |
Each of the three takes one of three mitigations: Block (on all pages), which applies across the whole zone; Block on pages with ads, which uses Cloudflare’s automated detection of ad-bearing pages and blocks only there; or Allow, which adds no blocking. Each blocking option covers verified bots in that class plus unverified bots that behave similarly.
Going crawler by crawler
Where the presets are broad strokes, AI Crawl Control works one operator at a time. Its Crawlers table lists each crawler with its operator, category, request volume and — the column most worth your attention — how many times it has violated your robots.txt. That number turns an abstract argument about crawler etiquette into a list of names, and it is the cleanest justification for moving a specific operator from allow to block.
There is an important asterisk on the Free plan. Cloudflare documents that free-tier detection identifies AI crawlers by their user agent string, which catches well-known, self-identifying crawlers and nothing else. Thorough detection using the Bot Management detection ID requires a paid plan. So on Free you can reliably block the crawlers that announce themselves honestly — which is a real result, just not the same as blocking everything.
Two adjacent features round it out. Cloudflare tracks robots.txt health and violations so you can see who is ignoring you, and pay per crawl — currently a closed beta — adds charging as a third option beside allow and block.
The layer that does not block anything
AI Labyrinth deserves its own note because it is easy to misfile as a blocking tool. It adds invisible, nofollow-tagged links to your pages; crawlers that ignore your no-crawl instructions follow them into a maze of never-ending links, and what they reveal about themselves is fed back into Cloudflare’s detection for every customer who blocks AI bots. Cloudflare states the links do not affect your SEO or your site’s appearance and are only seen by bots, and that well-behaved crawlers ignore the honeypot safely.
For anything the presets do not express — a single path, a specific user agent, a rate you want to cap rather than refuse — the general WAF custom rules are still the escape hatch, and a block there produces the same 1020 page a human would see. If your objection is contractual as much as technical, Cloudflare also publishes sample terms of service language for restricting AI scraping, which is the piece that gives the technical block something to stand on.
FAQ
How do I block AI crawlers on Cloudflare?
Use AI Crawl Control or the AI bot policies in Security Settings, both available on every plan including Free. Setting a crawler to Block refuses its requests at the edge. Managed robots.txt is a separate setting that only states a preference, so on its own it stops nothing.
Does robots.txt actually stop AI crawlers?
No. Cloudflare states plainly that robots.txt compliance is voluntary: the file expresses your preferences but does not prevent crawlers from accessing your content at a technical level, and some operators disregard it. Use it to declare intent, and use AI Crawl Control to enforce that intent.
What changes for AI bots on September 15, 2026?
Cloudflare is setting updated defaults for new domains: bots classified as Training or Agent will be blocked on pages that display ads, while Search stays allowed. Mixed-purpose crawlers that do both Search and Training will be blocked by every configuration that blocks AI training, including the legacy Block AI bots option, which is being deprecated on the same date.
Can I block AI training but still allow AI search?
Yes, and that is the point of the three presets. Search, Agent and Training are configured separately, so you can leave Search allowed to keep earning citations and referrals while blocking Training. Note that a crawler doing both Search and Training counts as mixed-purpose and will be blocked by the training setting.
Does blocking AI bots hurt my SEO?
Blocking Training and Agent does not affect a conventional search crawler, because those are separate categories from Search. AI Labyrinth is also safe in this respect: Cloudflare states its invisible links do not impact your SEO or your site's appearance and are only seen by bots.