As time passes, simply rejecting the use of AI seems increasingly pointless. Nonetheless, Cloudflare has made public several new tools for your anti-AI toolkit, concentrating some control back into the hands of users. Cloudflare provides a variety of internet services, such as domain registration, content delivery, and DDoS mitigation. As part of its offerings, Cloudflare is launching a novel initiative—Pay Per Crawl, aiming to financially impact AI companies that scrape online content.
Last year, Cloudflare introduced a feature that enables users to block AI bots with a single click. This tool, called AI Audit, also allows users to select specific bots to block and provides insight into which language models (LLMs) and generative AIs are reproducing their content. Currently, Cloudflare has announced that AI crawlers will be barred by default, which they describe as a pioneering step toward a sustainable future for both content creators and AI innovators.
Users who prefer their content to remain accessible for free scraping can opt out of this default setting. However, the second announcement from Cloudflare poses a significant question: why would you choose to permit that?
Cloudflare’s Pay Per Crawl program, which began beta testing last year, allows content creators to charge AI companies wishing to scrape their data. Although the identity of the participants in this program has not been fully disclosed, ProRata, the startup running the AI search engine Gist.AI, has been confirmed.
To further support their initiative, Bill Gross commented: “Our whole mission is to protect and elevate human creativity in the AI era. We’re proud to be one of the initial AI companies to support Cloudflare’s goal of establishing a new permission-based internet model, ensuring that creators receive fair compensation for their contributions.”
It is plausible that larger AI companies will be drawn to this initiative. For instance, OpenAI has already established content partnerships with several media organizations. Pay Per Crawl represents an attempt to create a fair playing field, allowing smaller creators who may lack the resources to negotiate complex partnership terms to be heard.
Cloudflare emphasizes that while some web crawlers play a vital role in preserving internet history, many AI crawlers engage in unapproved scraping that can overload servers, similar to a DDoS attack. Cloudflare has shared that it currently blocks an extraordinary number of cyber attacks daily. With its new AI blocking tools, they employ a mix of machine learning and behavioral analysis to identify these disruptive crawlers, leveraging various strategies to protect user content effectively.
As they pursue a stronger Permission-Based Model for the Internet, we wonder if this move could inspire cautious optimism concerning the future shape of digital rights and web governance.
