Cloudflare is tightening the screws on AI crawlers again. The company, through which about 20% of all web traffic flows, announced that starting from September 15, 2026, "mixed-use" crawlers will be blocked by default on advertising and monetized pages for all new clients and all new sites on existing accounts. At the same time, the Pay Per Crawl model is being expanded, where bot access to content becomes a paid service. For anyone involved in data collection, this is not an abstract news story about publishers — it is a direct signal that the "gray area" of the open web is tightening another notch.
What exactly did Cloudflare announce
The key innovation is the concept of a "mixed-use crawler." This is how Cloudflare refers to bots that collect data for two purposes simultaneously: search indexing and training AI models. Previously, such crawlers operated under the banner of a "search" bot and gained access to content in exchange for traffic. Now, Cloudflare is breaking this deal: if a site earns money from advertising, the mixed-use crawler will be blocked by default from monetized pages.
This is technically implemented through managed robots.txt rules and updated bot management heuristics. The site owner does not need to write anything manually — Cloudflare manages the lists and logic itself. The start date for new sites is September 15, 2026; existing projects will be able to enable blocking at their discretion.
The company's position was summarized by its CEO Matthew Prince: "Now that most of the traffic on the internet is non-human, we must act more decisively and quickly to create a sustainable ecosystem." The changes have been publicly supported by Associated Press, Time, The Atlantic, and Reddit — major publishers who suffer the most from the free siphoning of their content.
Numbers that explain everything
To understand why the infrastructure provider has engaged in a war with AI companies, it is enough to look at the crawl-to-referral ratio for 2025, which was provided by Cloudflare:
- Google — 14:1. For every 14 crawls of a page, the search engine sends one live visitor to the site. Tolerable: this is the old deal of "we take the content, you get the traffic."
- OpenAI — 1,700:1. For 1,700 crawls — one referral. The exchange has already ceased to be equitable.
- Anthropic — 73,000:1. Seventy-three thousand crawls for one referral. In fact, content is being siphoned off, and there is no traffic in return.
This asymmetry has broken the old model of the "open by default" web. As analysts have accurately stated: "Where the web used to be open until you closed it, now a significant part of it is closed until you open it." This is a 180-degree reversal of the default principle — and it impacts not only AI labs but also any automated collectors that fall under the same scrutiny.
Pay Per Crawl: access becomes a commodity
The second part of the announcement is the expansion of Pay Per Crawl, a marketplace that Cloudflare launched in private beta on July 1, 2025. The logic is simple: instead of a binary "allow/block," publishers now have a third option — charging for each crawl. The site owner sets the rate, and the AI company decides whether to pay it or not. Cloudflare acts as an intermediary that verifies the bot's identity and handles the transactions.
This is a tectonic shift for the market. Data that has been considered "free because public" for years is turning into a paid resource with a price tag. And the broader Cloudflare rolls out this model, the more expensive and complicated legal, "front-end" access to content becomes for those who are not willing to sign contracts and pay per crawl.
How this relates to proxies and scraping
Here lies the main practical takeaway. When major platforms close their gates to "honest" crawlers based on User-Agent, demand inevitably shifts to tools that make traffic indistinguishable from human traffic. Ben Brandidge, founder of the analytics company Synthient, states directly: "AI is definitely driving demand for residential proxies." — because companies are looking for ways to bypass blocks on their scrapers.
The scale of this shadow layer is already impressive. According to IPInfo, only six tracked providers accounted for nearly 79 million unique IPs over 90 days. Moreover, the market is heavily intertwined: about 46% of residential proxy addresses are simultaneously present in the pools of several providers, and individual IPs have appeared in the networks of more than a hundred different services. This means that a "cheap residential proxy" often turns out to be the same resold address that has already been flagged in dozens of other campaigns — and may have already been blacklisted.
We have already noted that regulatory pressure on data collection is increasing alongside technical pressure: the EU AI Act, coming into effect on August 2, 2026, gives weight to robots.txt and opt-out for training models. Cloudflare's move is the technical half of the same trend: legal and infrastructure barriers are converging, and "just sending a GET request from a data center IP" is becoming less effective.
What data collectors should do
It's too early to panic, but it is time to reassess your stack. Here are some practical guidelines for the coming months.
- Separate tasks by level of protection. For open, poorly protected sources, fast and cheap data center proxies are still sufficient. Spending residential traffic on them is a waste of money.
- For sites under Cloudflare, use residential IPs. When the target sends requests through bot management, residential proxies from real home providers save the day: their behavior is considered human by default. The key is not the absolute size of the pool, but its cleanliness and low percentage of overlaps with other services.
- Mobile IPs — for the toughest targets. Social networks and platforms with aggressive anti-bot measures are better accessed through mobile proxies: behind one operator's IP sit thousands of real subscribers, making it unprofitable for the platform to block it entirely.
- Focus on "survivability," not "volume." After Cloudflare's shift to "closed by default," the quality metric becomes not gigabytes, but the share of successfully collected pages without bans. A cheap resold pool with a 46% overlap of addresses may yield impressive traffic numbers but a dismal success rate.
- Follow the rules where they exist. If a source offers an official API or Pay Per Crawl terms, it is often cheaper and more reliable for stable production pipelines than an arms race. Proxies are a tool for public data, not for circumventing direct contractual prohibitions.
Conclusion
Cloudflare's move is not a one-time news story, but part of a systemic shift: the web is transitioning from an "open until closed" model to a "closed until opened" one. Ratios like 73,000:1 show why this is inevitable, and Pay Per Crawl turns access to data into a commodity with a price tag. For scraping, this means one thing: the era when a data center IP and a User-Agent header were sufficient is coming to an end. Those who proactively build collection on clean residential and mobile networks and focus on the share of data that actually arrives intact will win. Barriers will only continue to grow — and the quality of proxy infrastructure is becoming not just a cost item, but a decisive factor in whether you obtain data or not.