Cloudflare to Block AI Crawlers and Introduce "Pay Per Use" Content Model by September 2026

Cloudflare will implement a new default policy to block "mixed-use" AI crawlers from ad-displaying pages, effective September 15, 2026, aiming to reshape the economics of AI crawling by introducing a "Pay Per Use" model for content. This policy, applying to new customers, new sites from existing customers, and all free-tier users, comes as Cloudflare CEO Matthew Prince noted that non-human internet traffic has surpassed human traffic for the first time. For broader context, explore our AI News.

Understanding the New Default Policy

The core of Cloudflare's initiative is a default-block policy targeting "mixed-use" AI crawlers. These are defined as crawlers that combine traditional search indexing with data collection for AI training and agent development. By implementing this block, Cloudflare, which serves as a content delivery network and security layer for millions of websites, provides publishers with increased use over how their content is utilized by AI systems.

This move comes as Cloudflare CEO Matthew Prince noted that non-human internet traffic has surpassed human traffic for the first time, highlighting the growing impact of automated systems on web infrastructure. The policy is set to take effect on September 15, 2026, giving the industry time to adapt to the new framework.

Transition to a "Pay Per Use" Model

In conjunction with the blocking policy, Cloudflare is evolving its existing "Pay Per Crawl" marketplace into a "Pay Per Use" model. This new model is designed to allow publishers to charge AI companies directly when their content creates value for AI applications. This represents a fundamental change from traditional web crawling, where content is often accessed without direct compensation to the publisher for its use in AI training.

Initial launch partners for this "Pay Per Use" model include Ceramic.ai and You.com. These partnerships indicate the early adoption and potential for this new monetization strategy within the AI ecosystem. The model aims to establish a clearer economic relationship between content creators and AI developers, ensuring that the value derived from published content is appropriately compensated.

Implications for the AI and Publishing Landscape

Cloudflare's policy is an aggressive step to alter the financial dynamics of AI crawling. The company highlighted that the "world's largest search engine," Google, currently has significantly more information access due to its bundled AI crawling and search indexing capabilities. Cloudflare estimates Google has access to roughly twice as much information as its competitors, a disparity that this new policy seeks to address by creating a more level playing field for content access and monetization.

By setting a default block for "mixed-use" crawlers, Cloudflare is positioning itself to empower publishers, giving them greater control over their digital assets. This could lead to a re-evaluation of how AI models are trained and how content is sourced, potentially fostering a more equitable distribution of value in the digital economy. Publishers using Cloudflare's services will have the option to permit or deny access to these specific types of AI crawlers, or to participate in the "Pay Per Use" model.

Key Takeaways for Publishers and AI Developers

• Cloudflare will implement a default block on "mixed-use" AI crawlers from ad-displaying pages starting September 15, 2026.
• This policy applies to new customers, new sites from existing customers, and all free-tier users.
• "Mixed-use" crawlers combine traditional search crawling with AI training and agent data collection.
• Cloudflare is transitioning to a "Pay Per Use" model, allowing publishers to charge for content used by AI.
• Initial partners for the "Pay Per Use" model include Ceramic.ai and You.com.

Conclusion

Cloudflare's new policy and the introduction of its "Pay Per Use" model represent a significant shift in the relationship between content publishers and AI developers. By establishing a default block on certain AI crawlers and creating a framework for direct compensation, Cloudflare aims to rebalance the economics of content utilization in the AI era. This development, effective September 15, 2026, will likely prompt both publishers and AI companies to re-evaluate their strategies for content access and monetization.

Sources

• https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/