🌐 Trang này chưa được dịch sang tiếng Việt — đang hiển thị nội dung tiếng Anh.

Module 7d — The Agentic Internet: AI Crawler & Bot Control

Goal: Control how AI crawlers and bots access your own public content — see who's crawling, allow/block them by purpose, enforce robots.txt, and (optionally) monetize access with Pay Per Crawl.

🧭 A different direction. Every other module in this guide secures traffic going out — your people and agents reaching the internet, apps, and AI models. This module is the opposite direction: AI crawlers and agents coming in to your public websites. As the internet becomes "agentic," that inbound flow is now the majority of traffic — so controlling it is its own discipline.

Why this matters now (Cloudflare's 2026 agentic-internet report)

• AI is being adopted ~2× faster than smartphones — 2.5 billion users (30% of humanity) in 3.5 years.
• More than 50% of internet traffic is now non-human — bots and AI agents crossed that historic threshold.
• 52% of crawler requests are for AI training (June 2026), up from 22% in spring 2025.
• For every hour people spend searching online, only ~15 minutes is on the open web — AI answers increasingly replace clicks ("Google Zero").
• The old exchange — content for referral traffic — is breaking down, so a content-licensing economy is emerging (50+ publisher–AI agreements since 2023).

Translation for you: any organization that publishes content online now needs to decide, deliberately, which AI systems may access it, for what purpose, and on what terms. Cloudflare's AI Crawl Control is where you do that.

Part A — See who's crawling you (AI Audit)

Every site on Cloudflare has access to AI Audit inside AI Crawl Control.

1. 👉 In the Cloudflare dashboard, select your account and domain, then go to AI Crawl Control.
2. 👉 Open the Crawlers tab.
3. 📺 What you'll see: a table of AI crawlers hitting your site, with:
   • Crawler and the operator that owns it (GPTBot/OpenAI, ClaudeBot/Anthropic, Bytespider/ByteDance…)
   • Category (AI Crawler, AI Assistant, AI Search, Archiver…)
   • Requests (allowed + unsuccessful, with a trend)
   • Robots.txt violations
4. 👉 Open the Metrics tab to chart activity over time — group by Crawler / Category / Operator / Host / Status code, and (on paid plans) see referrer analytics showing which AI operators send you traffic.

✅ Checkpoint: you can see which AI crawlers access your content and how heavily.

💡 Detection quality: on the Free plan, crawlers are identified by user-agent (catches well-known, self-identifying bots). Paid plans add Cloudflare's Bot Management detection for crawlers that don't self-identify.

Part B — Control each crawler (allow / block / charge)

Decide, per crawler, how it may access your content.

1. 👉 In the Crawlers tab, find a crawler and use the Action column:
   • Allow — let it scrape (good for crawlers that send citations/referrals, or where you have an agreement).
   • Block — stop it entirely. You can configure the block response returned.
   • Charge — require payment per request (Pay Per Crawl — Part D).
2. 👉 Optionally toggle Enforce robots.txt so Cloudflare upholds your robots.txt rules for that crawler automatically.

Set policy by purpose (recommended)

Rather than one crawler at a time, set behavior-based policy:

1. 👉 Go to Security → Settings → Configure AI bot policies.
2. 👉 Choose a mitigation for each behavior preset:

   Behavior	What it is	Common choice
   Search	Indexes your content to answer questions later	Allow (keeps you discoverable)
   Agent	Real-time actions on a person's behalf (chat-fetch, browser-use)	Allow or Block on ad pages
   Training	Takes content to train/fine-tune models (incl. mixed Search+Training)	Block if you don't want to feed model training

3. 👉 Each preset offers Block (all pages), Block on pages with ads, or Allow (do not block).

⚠️ New defaults from 15 Sep 2026: for new domains, crawlers classified Training or Agent are blocked on pages that display ads, while Search stays allowed; mixed-purpose (Search+Training) crawlers are blocked by any training-block setting. The legacy one-click "Block AI bots" toggle is being replaced by these behavior policies. Review your setting so it matches your intent.

✅ Checkpoint: each crawler purpose has a deliberate action; a blocked crawler receives your configured response.

💡 Need finer control? Use Cloudflare WAF custom rules (per-path, per-crawler) or Redirect Rules to steer crawlers to specific areas — e.g. allow crawling of /blog but block /pricing.

Part C — Enforce robots.txt

robots.txt is your stated policy; AI Crawl Control makes it visible and enforceable.

1. 👉 In AI Crawl Control → Robots.txt, review file health, crawler violations, and Agent Readiness.
2. 👉 Turn on managed robots.txt / enforcement so Cloudflare can uphold your directives with an automatic WAF rule (rather than relying on crawlers to voluntarily obey).
3. 💡 Consider the Content Signals Policy (contentsignals.org) to declare how your content may be used after access (e.g. search yes, AI training no) directly in robots.txt.

✅ Checkpoint: violations are visible, and non-compliant crawlers can be blocked from the Crawlers tab or via WAF.

Part D — Monetize access (Pay Per Crawl)

If your content has training value, you can charge AI crawlers instead of just allowing or blocking — turning scarcity into revenue.

🔒 Pay Per Crawl is in private/closed beta. Join at cloudflare.com/paypercrawl-signup or ask your Cloudflare account executive.

How it works (site-owner side):

1. 👉 Set your price and select which crawlers to charge.
2. 👉 Connect Stripe for payments and watch delivery analytics.
3. 📺 When a crawler requests paid content without agreeing to pay, Cloudflare returns HTTP 402 Payment Required. Crawlers that opt in send payment intent via signed HTTP headers (crawler-max-price / crawler-exact-price, via Web Bot Auth) and are charged per successful crawl.
4. 👉 Keep discovery pages free: some paths are always free (/robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, /crawlers.json), and you can exempt more via Rules → Configuration Rules → Disable Pay Per Crawl on a URI pattern.

✅ Checkpoint: chosen crawlers are charged (or receive 402) while your free/discovery paths stay open.

Part E — Monitor the trend

• 👉 Cloudflare Radar (radar.cloudflare.com) publishes bot & AI-crawler insights across the internet — useful context for how crawler behavior is shifting industry-wide.
• 👉 Revisit AI Crawl Control → Metrics regularly; crawler mix changes fast (the training-vs-search split shifted dramatically in a single year).

How this fits the AI security picture

Together these cover the full "agentic internet": you govern how your organization consumes AI and how the AI ecosystem consumes you.

✅ Module 7d complete!

You now have:

• ✅ Visibility into which AI crawlers access your sites (AI Audit)
• ✅ A deliberate allow / block / charge action per crawler and per purpose (Search / Agent / Training)
• ✅ robots.txt made visible and enforceable
• ✅ (Beta) Pay Per Crawl to monetize AI access
• ✅ Awareness of the Sept 2026 default changes and where to tune them

Quick troubleshooting

👉 Next: Module 8 — Cloudflare WAN

Connect whole offices and data centers to Cloudflare's network.