Cloudflare (NET) Opinions on AI Scraper Blocking Policy July 06, 2025 — 01:00 pm EDT Written by Quiver DiscussionTracker for Quiver Quantitative -> ...

Cloudflare, the internet infrastructure company responsible for routing about 20% of global web traffic, has announced it will begin blocking artificial intelligence (AI) crawlers by default. The ...

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies ...

If you’re a site operator on Cloudflare’s network, you will now need to actively allow AI bots to index your content. If you don’t, they get blocked. Subscribe to Media CoPilot.

Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow or block AI crawlers. At least 16% of the world's internet traffic gets routed through ...

Cloudflare claims its clients can now allow or disallow crawling for each stage of the AI life cycle (in particular, training, fine-tuning, and inference) and white-list specific verified crawlers.

Pay Per Crawl. Cloudflare also announced a new compensation initiative to work out a method for AI companies to pay to crawl your content, it is called Pay Per Crawl.

So far, Cloudflare’s head of AI control, privacy, and media products, Will Allen, tells WIRED, over 1 million customer websites have activated its older AI-bot-blocking tools.

Global giants like Asana, Atlassian, Block, PayPal, Sentry, and Stripe are already working with Cloudflare to create powerful AI experiences.

Every new domain customer that signs up with Cloudflare to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether.