While some are blocking bots, others are teaching them to present identification. In 2026, Cloudflare, Google, Amazon, Akamai, and OpenAI came together around a single standard — Web Bot Auth. The idea is simple and radical: automatic traffic is no longer verified by IP address and User-Agent string, which can be easily spoofed. Instead, the agent cryptographically signs each request, and the site instantly verifies the signature. On March 2, 2026, the fifth version of the architecture draft (draft-meunier-web-bot-auth-architecture-05) was released, in May Google began live testing, and the IETF working group aimed to release the standard by August 2026. Let's explore what this mechanism is, who is behind it, and why it represents a 180-degree turn for anyone involved in scraping and multi-accounting.
What Happened: Bots Gain Cryptographic Identity
The classic "friend or foe" check on the web relied on two crutches. The first — User-Agent: a string that the client reports about itself and that any script can rewrite into a single line of code. The second — IP address lists: the site maintains a list of "official" ranges for Googlebot, Bingbot, and others, and checks reverse DNS. Both approaches fell apart in 2026.
Cloudflare explicitly states the problem in its technical documentation: the same IP can be used by many users or services simultaneously, and the ranges themselves constantly change along with the infrastructure. The User-Agent is not proof at all — it is merely self-representation. The cryptographic signature, as envisioned by the authors, provides authentication without relying on "ever-changing IP ranges" or "spoofable headers like User-Agent."
Technically, Web Bot Auth is based on the already accepted standard HTTP Message Signatures (RFC 9421). Each agent is issued a pair of Ed25519 keys. Before sending a request, the agent signs not everything indiscriminately, but the authority of the target URI — that is, the domain it is contacting (when accessing example.com, it signs example.com specifically). The request includes headers Signature-Input (a validity window with created and expires timestamps, the key identifier keyid in JWK Thumbprint format, and the tag web-bot-auth) and Signature-Agent, which points to the directory with public keys (JWKS). The site downloads the public key from this directory and verifies the signature. It is impossible to spoof it without the private key, and the validity window protects against replaying intercepted requests.
Who is Behind the Standard
This is not an experiment by a single company. Web Bot Auth is supported by Cloudflare, Amazon, Akamai, and OpenAI. The architecture draft is co-authored by Thibaut Meunier (Cloudflare) and Shandor Major (Google). AWS WAF has already added support for signature verification, and Cloudflare has integrated it into its Verified Bots program and activated it on its edge in production. A separate IETF working group for Web Bot Auth was established in 2026 and aims to submit the specification for standardization by August 2026.
Google Gets Involved — A Turning Point
On May 6, 2026, it was announced that Google is testing Web Bot Auth to verify its own bot traffic. The purpose is for the site to confirm that a request claiming to be from Google is indeed coming from Google, rather than someone who simply typed Googlebot into the User-Agent. The signed AI traffic from Google is identified as agent.bot.goog — website owners retrieve Google's public keys from this endpoint and validate the Signature and Signature-Input headers.
Why is this important right now? Precisely because of what we wrote about the blocking of agent browsers: starting September 15, 2026, Cloudflare will by default cut off "mixed" AI crawlers on advertising pages of new domains, and about one-fifth of the entire web passes through the Cloudflare network. The web is closing down. And Web Bot Auth is not about "how to block," but about "who to let in." A new type of whitelist is emerging: not by IP, but by cryptographic identity. The site will be able to implement a policy such as "show prices and product details only to verified signed agents" — and everything else will automatically remain outside.
Signed Agents: The First Cohort
A separate category is signed agents. These are bots managed by end users, not a single corporation: the platform or remote browser running the agent signs its HTTP requests, and Cloudflare validates the signatures. The first cohort includes ChatGPT agent (OpenAI), Goose (Block), Browserbase, Anchor Browser, and Cloudflare Browser Rendering. Daouwe Osinga from Block described the benefit directly: "Web Bot Auth allows sites to trust Goose while preserving what makes it unique."
The key difference: a signed agent acts on behalf of a person (a hypothetical assistant who visits a site at your request), whereas a "search" or "training" crawler operates for a company. Cloudflare is already differentiating these entities in security rules — and this distinction will determine who will be granted access in the web of 2026–2027 and who will not.
What This Means for Scraping and Proxies
At first glance, it seems that cryptographic signatures bury anonymous scraping. In practice, the situation is more nuanced — the web is splitting into two lanes, and for our audience, this changes tactics rather than abolishing them.
- Lane for signed agents. A narrow, privileged passage for major platforms and "user" assistants from the whitelist. To enter, a private key, a public JWKS directory, registration with a CDN, and compliance with policies are required. For a typical scraping project, anti-detect farm, or SMM automation, this path is closed: you are neither Google nor OpenAI, and you won't be issued a signature.
- Lane for everyone else. Here, independent scrapers, price monitoring, data collection, and multi-accounting still thrive. And it is here that the stakes for proxy quality and fingerprint reliability increase. Since "legitimate" traffic is allowed through signatures, the treatment of all unsigned traffic becomes stricter by default.
In other words, Web Bot Auth does not eliminate the need for proxies — it raises the bar for those not on the whitelist. If previously one could hope to slip through on a gray User-Agent, now an unsigned request immediately falls into the category of "prove that you are a real person." And this is proven through a combination of clean IP and plausible behavior.
Why Residential and Mobile IPs Are Becoming Even More Valuable
When a signature is unavailable, the only way to appear legitimate is to be indistinguishable from a regular user. Data center addresses do not solve this task: according to various measurements, Cloudflare only allows data center IPs through 40–60% of the time, while residential proxies provide a pass rate of 85–99%. The reason is precisely that a residential address belongs to a real internet provider and a living subscriber — by the reputation of the IP, it cannot be distinguished from a person behind the same router.
The situation is even tougher with the most secure platforms and mobile applications, where cellular networks play a role. Here, mobile proxies come to the rescue: behind one operator's IP sit thousands of living subscribers, so banning such an address en masse means cutting off real customers. This "common pot" effect makes mobile IPs the most resilient currency in a world where everything else is transparent.
But a clean IP is no longer enough. As identity verification shifts from the network to cryptography, the entire unsigned army of bots goes through behavioral and browser fingerprinting: TLS/JA4, canvas, WebGL, fonts, timings. Proxies eliminate one alarm signal (the reputation of the address), but do not negate the others. The working combination for 2026 is a residential or mobile IP plus an anti-detect environment with a consistent fingerprint. One component without the other will expose a bot at the first level of protection.
How to Act Right Now
- Separate projects by lanes. If the task is official integration on behalf of a major product, study the path of the signed agent: registration with a CDN and obtaining keys. For everything else, plan to operate in the "lane for everyone else" and establish its rules.
- Invest in IP quality, not quantity. Cheap data center pools in 2026 are increasingly a stop factor. Focus on residential and mobile addresses with rotation tailored to the task.
- Synchronize the fingerprint with the network. An IP from one country, while the browser language, time zone, and locale are from another — this is a classic failure. The fingerprint must consistently match the geolocation of the proxy.
- Keep an eye on the standard. By August 2026, the IETF plans to finalize the specification. As soon as Web Bot Auth becomes a mandatory condition for access on a specific platform, the tactics for it will need to be reconsidered in advance, not after the fact.
Conclusion
Web Bot Auth is not just another anti-bot measure, but a shift in the very logic of trust on the web: from "who are you by IP" to "prove cryptographically who you are." For giants like Google and OpenAI, it is a pass into a closing internet. For independent scrapers and multi-account projects, it is a signal that gray methods are definitively ceasing to work, and the divide between "looking like a person" and "looking like a bot" is becoming decisive. In this new reality, it will not be the one with the most IPs who wins, but the one with a clean residential or mobile address and a flawlessly consistent fingerprint. The cryptographic passport has not been issued to everyone — which means everyone else will have to be more convincing.