On July 1, 2026, Cloudflare held its second "Content Independence Day" and officially categorized AI agents as a separate class of bots for the first time. Now, traffic is divided into three classes — Search, Agent, and Training — and starting September 15, 2026, bots in the Agent and Training categories will be blocked by default on advertising pages. Meanwhile, courts and major retailers are already shutting their doors to "agent browsers" like Perplexity Comet and ChatGPT Atlas. Let's explore what is happening with the agent web and why autonomous AI assistants are increasingly hitting a wall without residential proxies.
What Happened: Bots Have a Third Caste
Previously, the web recognized two types of automated traffic: search crawlers that index pages and training crawlers that extract content for model training. In 2026, a third type was added — agent. Cloudflare, through whose network about 20% of all web traffic passes, provided a precise definition: it is "automated behavior that typically acts in real-time on behalf of a person to do something right now."
This definition includes fetch-bots of chat assistants (for example, ChatGPT-User, which pulls fresh information based on user requests) and full-fledged browser-use agents — Gemini or Claude, which operate real Chrome. The logic behind the new policy is simple: an advertisement signals that the website owner wanted to show the page to a person and monetize their attention. Therefore, on monetized pages, Cloudflare by default blocks bots that deprive the site of this attention — training and agent bots.
Key dates to remember: the changes take effect on September 15, 2026 and apply to new domains connecting to Cloudflare. Search bots remain permitted — websites still need traffic from search engines. However, multi-purpose crawlers like Googlebot will be evaluated based on the sum of all their roles: if the owner enables training blocking, such a mixed bot will be completely blocked.
Agent Browsers: Who Entered the Field in Two Years
Back in early 2025, the term "AI browser" sounded like an experiment. By mid-2026, it had become a crowded market of autonomous assistants that click, scroll, fill out forms, and make purchases on their own. Key players include:
- Perplexity Comet — launched in July 2025, free, available on desktop, Android, and iOS;
- ChatGPT Atlas from OpenAI — October 2025, subscription-based, with a separate "Agent Mode";
- Chrome + Gemini — integration from Google, January 2026, for AI Pro and Ultra subscribers;
- Claude for Chrome — extension for the existing browser, August 2025;
- Edge Copilot Mode — free mode from Microsoft, July 2025;
- Opera Neon — September 2025, public release in December, subscription around $19.90 per month.
They all operate similarly: the agent manages the interface through screenshots and input commands — it "sees" the page, clicks, types, and navigates links on your behalf. This human-like mechanism makes them a headache for anti-bot systems: formally, this is not a crawler pulling APIs, but a session of a real browser backed by a live user's request. Google even introduced a separate identifier Google-Agent in March 2026 so that website owners could distinguish such requests in their logs.
Courts Are Already on the Side of Websites
While Cloudflare is distributing settings, the conflict surrounding agents has reached the courts. The most notable case is Amazon vs. Perplexity. Amazon filed a lawsuit back in November 2025, accusing the startup of intentionally "hiding" the nature of its AI agents to continue scraping the retailer's site without permission.
On March 10, 2026, Judge Maxine Chesney sided with Amazon. The ruling stated that the company provided "compelling evidence": the Comet browser accessed the site at the user's direction but "without authorization" from the retailer. According to Amazon, Comet "masked" its automated nature, posing as a live shopper to slip past detection systems. Amazon even documented expenses — over $5,000 on developing tools to block Comet from accessing internal customer data. In response, Perplexity stated that it "will continue to fight for internet users' right to choose any AI they want."
Amazon is also taking action without the courts. Through the robots.txt file, the retailer has blocked access to bots ChatGPT-User and OAI-SearchBot (the SearchGPT engine), and even earlier — to the training GPT-bot. Crawlers associated with Meta, Google, and Perplexity have also been affected. The irony is that OpenAI is simultaneously building its own sales funnel: the Instant Checkout feature already allows purchases directly in ChatGPT from Walmart, Etsy, Shopify, and Target. Retailers want to sell through AI on their terms — and do not want outside agents wandering through their storefronts unchecked.
How Websites Detect Agents — and Why IP Alone Isn't Enough
Blocking an agent solely based on the user-agent name is naive: it can be easily spoofed. Therefore, the anti-bot stack of Cloudflare and similar services looks at a combination of signals, and here autonomous browsers struggle:
- Headless browser fingerprint. A headless environment presents a set of inconsistencies: screen resolution 0x0, absence of installed fonts, GPU reporting as the software renderer "SwiftShader." Real users do not appear this way — and this is caught in milliseconds.
- TLS fingerprinting. Even with perfect HTTP headers, the TLS handshake reveals a disconnect between the declared identity of the browser and the actual characteristics of the connection. We explored this vector in detail in the article about JA4 and TLS fingerprinting — today, it is one of the main detection tools.
- Behavioral analysis. A human scrolls with pauses, moves the mouse imperfectly. An agent, however, acts with "brutally robotic efficiency" — loading the page in milliseconds and instantly extracting data, which gives it away.
- IP reputation. Ranges of cloud providers — AWS, Google Cloud, Azure — have long been cataloged as "bot farms" and receive heavy penalties in reputation filters, no matter how clean the rest of the fingerprint is.
This is where proxies come into play. IP reputation is the first barrier that an agent launched from a data center address hits. Residential proxies provide IP addresses of real home devices, making the agent's traffic no longer appear as a request from a server rack. For mobile scenarios — social networks, applications with strict detection — even more reliable are mobile proxies with IPs from telecom operators. Data center addresses, however, do not disappear: for tasks that are not sensitive to reputation, data center proxies offer speed and price that residential ones cannot match.
Important Disclaimer: Proxies Are Just One Layer
A residential IP only removes one of four barriers. If the agent is still running in a headless browser with a "SwiftShader" fingerprint and behaves like a robot, a clean IP won't save it — detection will be based on the other signals. The working combination for 2026 is not "proxy or anti-detect," but all at once: a residential or mobile exit node, masking the browser and TLS fingerprint to resemble a real device, and human-like timing of actions. Scraping and automation have ultimately turned into a game of identities rather than addresses.
What This Means in Practice
For businesses and developers building automation on top of agent browsers, the conclusions are as follows:
- September 15 is not an abstract date. If your agents access sites with advertisements, be prepared for some of them to start returning placeholders instead of content. Plan your infrastructure in advance.
- Data center agents are the number one target. The first thing to do is to shift agent traffic from cloud ranges to residential or mobile IPs. This is cheaper than rewriting logic after mass bans.
- Legal risk is real. The Amazon vs. Perplexity case shows that "masking" an agent as a human is not a gray area, but an argument for the court. Respect robots.txt where it is critical, and do not present automation as a live user on sites that explicitly prohibit it.
- Official channels are growing. Instant Checkout and AI partnerships with retailers signal that some agent scenarios will move to legal APIs. But for now, coverage is fragmented, and autonomous agents will have to live in a world where half of the websites do not welcome them.
Conclusion
The agent web is no longer a concept from presentations, but real traffic that websites have learned to filter and want to filter. Cloudflare has given owners a lever, the courts have provided a precedent, and retailers have set an example. Autonomous AI assistants are not doomed, but it is a clear signal: for an agent to reach its goal, it must appear human at all levels — from exit IP to click timing. Residential and mobile proxies are not a luxury here, but a basic layer without which other tricks do not work. The web is not closing its doors to agents forever — it is simply no longer allowing those who come with a sign saying "I am a bot."