With the rapid rise of large AI models and the huge demand for public content on the Internet during training, website operators have also paid more attention to their own content rights. In response to this trend, Cloudflare recently announced the launch ofPay per Crawl marketplace services, allowing website owners to decide whether to authorize AI crawlers to crawl content according to their own wishes, and even set rates for each crawl, opening up a new business model for content management.
Cloudflare announced that the service has entered its beta testing phase. Participating AI companies and website publishers will need to verify their identities and set up authorization through their Cloudflare accounts. In the future, AI crawlers will no longer be able to "snatch" content silently, but will instead require legal authorization through payment.
Cloudflare will act as an intermediary, assisting both parties in completing the transaction process and distributing the proceeds to content providers.
More importantly, Cloudflare also announced that starting today, all newly registered websites protected by Cloudflare will be completely blocked from AI crawlers by default. Unless the website owner proactively grants permission, no AI-trained crawlers will be able to access content. This adjustment is clearly aimed at AI companies, including Google, OpenAI, and Anthropic, that have been known to scrape large amounts of website content without authorization in recent years, giving content creators more autonomy.
Major media outlets including Condé Nast, Time, The Atlantic, Fortune, and the Associated Press have signed contracts with Cloudflare, enabling their websites to block access to content by AI crawlers by default. This reflects the proactive efforts of traditional publishers to preserve the value of their content and develop new revenue streams in the AI era.
Cloudflare CEO Matthew Prince pointed out that the potential of the pay-per-view market is not limited to the news publishing industry. As "agentic AI" becomes more prevalent in the future, it is likely to become a vital source of data for purposes such as training models and providing real-time responses.
Data released by Cloudflare shows that Google currently crawls web content 14 times for every search recommendation it contributes, but OpenAI crawls 1700 times, and Anthropic crawls as many as 73000 times. There is a clear imbalance in the traffic and content crawling numbers between the two parties, and there is clearly room for the market to readjust profit sharing.
Interestingly, Cloudflare has revealed that it is open to using stablecoins or its own digital currency to process these transactions in the future, making cross-border transactions more efficient. However, the marketplace currently does not utilize any digital currency mechanisms and still primarily relies on traditional settlement methods.
Overall, Cloudflare's move is not only a technological upgrade, but also has the potential to open up new business opportunities for website operators in the AI era, shifting from "passively providing content" to "actively setting authorization and charging standards," further reshaping the value distribution logic of the online content economy.
