Cloudflare's New Policy Pushes AI Companies to Pay for Publishers' Content

Cloudflare, a leading internet infrastructure and security company, has announced a pivotal new policy designed to reshape how artificial intelligence (AI) companies access and utilize web content. Effective September 15, 2026, Cloudflare will implement default settings that block "mixed-use" web crawlers – those used for a combination of traditional search, AI agent functions, and model training – from accessing pages that host advertisements. This significant move mandates a clear separation between crawlers intended solely for search engine indexing and those employed for AI purposes, impacting new Cloudflare customers, new sites from existing customers, and all current free users. The policy change is set to profoundly influence how AI model providers gather data for training and power their agentic services. Cloudflare emphasizes that while website owners generally desire discoverability through both search and AI, they also demand robust protections for their intellectual property against free exploitation. The company subtly targeted "the world's largest search engine," implying Google, for making it challenging for publishers to remain visible without their content being leveraged for AI. Google, in response, has previously highlighted its "Google Extended" bot, which allows site owners to opt out of AI training use, though its flagship Googlebot still crawls for search, including AI features like AI Overviews. Matthew Prince, Cloudflare co-founder and CEO, underscored the urgency of this shift, noting that non-human traffic, primarily bots, recently surpassed human internet traffic for the first time – a milestone anticipated for the following year. Prince stated, "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." He expressed hope that these default changes would encourage mixed-use crawlers to differentiate between search, agent use, and training, fostering a more transparent and equitable digital environment. Beyond these new default settings, Cloudflare has been actively developing tools to empower publishers in the AI era. This includes the evolution of its "Pay Per Crawl" marketplace into "Pay Per Use," enabling publishers to charge AI companies not just for fetching content, but specifically when that content generates value. This innovative approach also promises to conserve publishers' bandwidth and compute resources, as Cloudflare's data indicates that over 50% of AI crawler traffic is spent re-fetching unchanged pages, a wasteful practice the new system aims to mitigate. To kickstart the "Pay Per Use" model, Cloudflare is collaborating with initial partners Ceramic.ai and You.com. Under these partnerships, publishers who opt in will receive compensation when their content appears in Ceramic's AI search results or when You.com accesses their premium content. Cloudflare has indicated that other AI companies will have the flexibility to customize this model to align with their specific operational frameworks, signaling a broader industry shift towards more equitable compensation for content creators in the age of artificial intelligence.