The three AI crawler families to distinguish
Cloudflare AI Audit, updated on 1 July 2026, classifies AI bots into three categories with very different intentions: search crawlers (GPTBot, PerplexityBot, Amazonbot) feed AI answers and generative search engines; training crawlers (CCBot, Common Crawl, ClaudeBot in scraping mode) collect content to build or fine-tune models; autonomous agents drive browsers without a human session to perform tasks on behalf of a third party.
Why differentiating these three families matters
- Visibility vs. exposure — allowing search crawlers increases chances of being cited in AI answers; blocking training crawlers protects high-value content.
- Granular control — robots.txt per User-agent allows a per-family policy without cutting classic SEO indexation.
- Agent attack surface — an autonomous agent can scrape a form or trigger actions; blocking them at WAF level is independent of the rest.
- Editorial intent — some content (paid guides, tutorials) is reserved for human readers and should not feed a training corpus without agreement.
- Server load — training crawlers are often aggressive and do not honour Crawl-delay; identifying them allows throttling or network-level blocking.
- Auditability — Cloudflare AI Audit traces traffic per category, making the decision visible and reversible.
robots.txt: the first line of defence
The robots.txt file is the most readable declaration of intent. Well-configured crawlers (GPTBot, PerplexityBot, Amazonbot) honour it. Less scrupulous training crawlers (CCBot) respect it variably. Autonomous agents generally ignore it. This is why robots.txt must be complemented by active rules at the network level.
Audit and configuration step by step
Audit traffic with Cloudflare AI Audit
In the Cloudflare dashboard, open Security > Bots > AI Audit. The interface displays volume per category (search, training, agents) and per user-agent. Identify the bots present and their actual weight before configuring anything.
Declare policy per family in robots.txt
Add targeted blocks: User-agent: GPTBot / Allow: / for search crawlers to keep; User-agent: CCBot / Disallow: / for training crawlers to block; User-agent: * to maintain the default SEO rule.
Create a Cloudflare WAF rule for autonomous agents
In Security > WAF > Custom Rules, create a rule targeting cf.bot_management.js_check_failed combined with a low bot score. Choose Block or Challenge depending on your client's tolerance.
Block unruly training crawlers at the WAF
For CCBot and bots with unreliable user-agents, add a WAF rule on http.user_agent contains "CCBot" with action Block.
Verify classic SEO indexation is preserved
After deployment, check in Cloudflare Analytics that Googlebot, Bingbot and usual SEO crawlers are not impacted.
Do not block everything: search crawlers have value
GPTBot (OpenAI), PerplexityBot and Amazonbot feed the answers of generative search engines. A site blocked for these crawlers does not appear in AI summaries from ChatGPT or Perplexity. The recommended strategy: allow search crawlers, block training crawlers, filter autonomous agents at the WAF.
Client sites without Cloudflare: what to do
For sites hosted without Cloudflare as proxy, robots.txt remains the only declarative lever. Complement it with User-agent directives for each known bot. On a VPS with root access, fail2ban can read access logs and ban by IP the crawlers that ignore robots.txt.
