The era of unrestricted AI crawling appears to be ending - well, at least for the fifth of the internet that flows through Cloudflare's pipes.

Cloudflare reshapes the AI landscape by blocking Big Tech AI bot crawlers by default, allowing websites to demand payment for content access.

Cloudflare is blocking AI crawlers from scraping websites by default, starting July 2, 2025. With the new “Pay Per Crawl” initiative, website owners can now charge AI companies to access their ...

Cloudflare, the internet infrastructure company responsible for routing about 20% of global web traffic, has announced it will begin blocking artificial intelligence (AI) crawlers by default. The ...

Cloudflare (NET) Opinions on AI Scraper Blocking Policy July 06, 2025 — 01:00 pm EDT Written by Quiver DiscussionTracker for Quiver Quantitative -> ...

Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow or block AI crawlers. At least 16% of the world's internet traffic gets routed through ...

Tech giants are rewriting the rules on web scraping, blaming unnamed third parties for disregarding robots.txt, and seemingly claiming the right to reuse anything posted anywhere for AI. Now ...