The parser is working. HTTP 200 responses are flowing continuously, proxies are alive, captchas are not popping up, and the queue of links is growing. But after a week, it turns out that half of the collected prices are fictitious, and gigabytes of traffic have gone to pages that do not exist on the actual site. This is not a malfunction of the parser or a poor IP pool. This is a new mode of protection: the site does not block you; it feeds you.
Over the past year and a half, the anti-bot protection industry has quietly shifted its focus. Blocking is an expensive and noticeable measure: a scraper sees a 403 error, fixes its fingerprint, changes the subnet, and returns. It is much more profitable not to interfere with its work but to make its work meaningless. Below are three mechanisms that are already operational on millions of sites, along with a set of checks that can catch them on your side.
First Mechanism: Tarpit Instead of Block
A tarpit is a generator of infinite pages. A crawler receives a valid HTML page with dozens of links, each link leading to another similarly generated page, and the crawl queue never empties.
The most well-known open tool is Nepenthes. Its settings clearly demonstrate the intent: by default, the server holds a response for 10 to 65 seconds, serves text generated from a corpus as "Markovian nonsense," and does this deterministically — the same URL always returns the same garbage, making the pages look like ordinary static files rather than traps. Data is delivered in small portions, "a few bytes at a time," to exhaust the client's timeouts. The author provides a measurement for one hour of operation: 1850 different clients, 10,015 requests, and 56,020 seconds of total delay — about fifteen hours of wasted machine time.
Iocaine is structured differently: it operates as a reverse proxy in front of the actual site, issuing a unique "poisoned" link to the bot upon the first interception and recognizing it upon return, while generating text using Markov chains — with the expectation that this text will end up in the training dataset.
Cloudflare has turned the same approach into a product — AI Labyrinth, introduced in March 2025. The bait pages are not generated on the fly: a pre-generation pipeline using AI Workers is employed, and the results are stored in R2 for quick distribution. Links to the labyrinth are embedded in regular pages through HTML transformation, and the bait pages themselves have meta-directives against indexing to protect the search results. The key here is not the bot's wasted time, but the signal: links hidden from humans and marked with nofollow are only accessed by automation, and navigating three levels deep into such a labyrinth becomes a fingerprint of a bad bot. Cloudflare estimates the scale of the problem that prompted this action at over 50 billion requests from AI crawlers per day — just under 1% of all network traffic.
How Tarpit Appears in Your Logs
A characteristic picture: a queue of several thousand URLs that keeps growing, response times consistently hovering around one and a half seconds or more, HTTP codes are all 200, and the number of extracted valid records is zero. No 403 errors, no captchas, and no way out: no matter how many pages are crawled, new links appear faster than old ones are closed.
Second Mechanism: Poisoned Content for AI Agents
If the first mechanism burns resources, the second strikes at the results. Researchers Minghao Luo and Liang Chen published a paper in June 2026 with a simulator called FORGE (Fake Online Recommendations in Generative Environments): they tested 12 major language models on 225 products across 15 categories — from clothing to electronics. The attack mechanism is simple: in the text of the page, a real brand is replaced with a fictional one.
The result — one fake page leads to up to 27% of cases where the assistant recommends a non-existent brand, and replacing all three top search results raises this share to 73.8%. The models did not just repeat the fake name — they invented merits for it, including supposed popularity in communities. The three proposed defenses (prompting skepticism, consensus on the model's internal knowledge, cross-checking between documents) either did not work or created new problems. The authors' conclusion: checks need to be done upstream — at the data collection stage, not at the reasoning stage.
The third mechanism completes the picture. In the paper "A Whole New World: Creating a Parallel-Poisoned Web Only AI-Agents Can See" (Shaked Zychlinski), cloaking aimed specifically at AI agents is described: the site recognizes the agent by browser attributes, automation framework signatures, and network characteristics, delivering a different version of the page — with hidden instructions and altered facts. A human opening the same URL sees a normal page, so a manual check of "I visited, everything is fine" proves nothing.
Why This is Primarily a Budget Issue
A tarpit is designed so that each page is cheap for the site and expensive for you. When traffic is billed by the gigabyte, an infinite page generator becomes a counter of your expenses: you pay for megabytes of Markov text that will never become a line in the database. The same arithmetic applies to failed requests — the price per gigabyte says nothing about the cost of results until you consider the cost of a single successful record, not just a request.
Hence the first practical rule: traffic limits should be set at the domain and task level, not just at the account level. A domain that has consumed more than a gigabyte without returning any records should be automatically stopped — without this, a single trap can consume the daily budget overnight.
Seven Checks That Catch Garbage
- Count yield, not response codes. The main metric of the pipeline is the share of requests that resulted in a valid record with filled mandatory fields. As long as you are looking at the share of 200s, the tarpit appears to be a perfectly healthy source.
- Canary URL. Once every N requests, request a deliberately non-existent address within the domain — with a random path segment. A normal site will respond with a 404 or a redirect, while the generator will deliver a full page with text and links. This is the cheapest and most reliable check.
- Cross-validation from another IP profile. Take the same URL via two different routes — for example, through a residential IP and through a mobile one — and compare the hash of key fields: prices, names, availability. A discrepancy with the same URL and close request times means you are being shown different versions of the page, and at least one of them is not intended for humans.
- Do not follow invisible links. Links with nofollow, zero sizes,
display:none, or those moved off-screen are bait, and following them is the very fingerprint of a bot. Filter them out at the link extraction stage, not afterward. - Strict response ceilings. Limit not only the timeout but also the maximum body size and maximum crawl depth from the entry point. Slow responses in small chunks are a typical sign of a tarpit, not a slow server.
- Look for text patterning. Markov generation reveals itself through statistics: suspiciously uniform paragraph lengths, repeating n-grams across "different" pages, dozens of outgoing links in the absence of structural elements like price, SKU, or date. A simple check for repeating shingles between adjacent pages of the domain filters out such sources in bulk.
- Check numbers for common sense. A price outside the historical corridor, a product with no matches in your own brand database, a sudden jump in assortment — these are validation rules that should be applied before recording in the database, not in a report a month later. Especially if the data later goes into a model or an automated purchasing decision.
What to Do with Already Collected Data
If suspicion arises retrospectively, sort not by dates but by sources. Group records by domain and look at three metrics: the share of pages without mandatory fields, the average number of outgoing links per page, and the variance in text length. Trap domains usually stand out immediately across all three. Then selectively recheck contentious URLs from another IP profile — if the data does not match, the entire dataset from that domain needs to be rebuilt, not filtered.
It is also worth revisiting rules for agent scenarios, where the model navigates pages and makes decisions on its own. This is where replacing a single page has the maximum effect, and there is no intermediary human to notice the oddity. The minimum safeguard is to require confirmation from two independent sources and not allow the agent to act on data obtained from a single domain.
In Brief
Bots have long surpassed humans in traffic share, and protection has responded not only with filters: today, it is cheaper to feed a machine plausible garbage than to argue with it through blocking. There are three practical consequences. Count valid records, not response statuses. Keep canary checks and traffic limits for each domain. Cross-check contentious pages from different IP profiles — discrepancies in versions of the same page are proof that you are being shown a separate, specially prepared internet for bots. Proxies in this scheme only solve one task — providing an independent second view of the page; everything else is validation on your side.