Cloudflare will block AI crawlers that hide their purpose

Cloudflare is changing its default settings so that "mixed-use" web crawlers get blocked on pages that carry advertising, starting September 15. A mixed-use crawler is one that blends search indexing, AI agent browsing, and model training into a single tool without telling the site which it is doing. Search-only crawlers such as Google Search stay allowed; the crawlers that refuse to separate their purposes are the ones that get stopped unless the site owner opts back in. The new default applies to new Cloudflare customers, new sites made by existing customers, and all existing free-tier customers.

The point is to force AI companies to run distinct crawlers for distinct jobs, so a publisher can say yes to search and no to training. Alongside the block, Cloudflare is turning its "Pay Per Crawl" experiment into a broader "Pay Per Use" marketplace where publishers can charge AI companies when their content produces value, with Ceramic.ai and You.com among the first partners. CEO Matthew Prince framed the urgency plainly, saying the company "must go further and act faster so that a sustainable ecosystem can emerge." Cloudflare adds a bandwidth argument too: it says more than half of AI crawler traffic is re-fetching pages that have not changed. You can read TechCrunch's report for the rollout details.

Because Cloudflare sits in front of a large share of the web, a default flip like this moves the whole market at once rather than one site at a time.

Why it matters

If you run an AI crawler, plan now to split search, agent, and training traffic into labeled crawlers or expect to be blocked on ad-supported sites and asked to pay. If you publish, this gives you a switch you did not have before: allow the crawling you want and charge for the rest.