If the Wildberries or Ozon parser suddenly starts returning empty pages, timeouts, and 403 errors — the problem is not always with the platform itself. Most often, some IP addresses in the proxy pool have already "expired": they have been banned, throttled, or subjected to CAPTCHA. We will explore which metrics distinguish a working proxy from a dead one and how to automate filtering without manual control.
Why the proxy pool degrades over time
Any proxy pool — even one purchased from a reliable provider — gradually loses some working addresses over time. The reasons vary: anti-fraud systems of Wildberries, Ozon, and Avito accumulate reputation marks on IPs, some addresses get blacklisted after mass requests from other clients of the provider (if the proxy is shared), data center ranges are banned in batches by subnets, and residential IPs sometimes simply leave the pool because the real user has turned off their device.
If the pool is not cleaned, the share of "dead" proxies grows non-linearly: initially 5%, after a week of active scraping — 20-30%. The parser starts wasting resources on retries through non-working IPs, the average time to collect a single page increases, and errors accumulate in the logs, masking real problems (for example, changes in the site's layout). Without a filtering system, you either keep the pool inflated "just in case" or regularly face data collection failures.
The task is not just to record the fact of an error but to classify it: temporary server overload, ban of a specific IP, regional blocking, or a problem on the parser's side (incorrect headers, outdated cookies). The type of error determines whether to permanently remove the proxy from the pool, quarantine it temporarily, or simply retry the request.
Key error metrics for filtering
To distinguish a temporary failure from a truly dead proxy, it is necessary to collect several metrics for each IP rather than relying on a single HTTP status. Here is a basic set that should be logged with each request:
| Metric | What it signals | Action |
|---|---|---|
| HTTP 403 / 429 | Ban or rate-limit on IP | Quarantine for 30-60 minutes, if repeated — removal |
| Connection timeout | Proxy overloaded or dead | 3 consecutive timeouts — removal from the pool |
| CAPTCHA in response | IP is under suspicion of anti-bot | Reduce request frequency or replace IP |
| Response time > 5 sec | Speed degradation on the proxy side | Lower priority in the pool |
| Response size below average | Returned a placeholder/blocking page | Additional content check before acceptance |
| Success rate | Integral quality assessment of IP | Below 70% over 50 requests — removal |
It is important not to confuse site error codes with network errors. For example, a 404 is not a proxy problem, but a missing product on Wildberries. However, if the same IP regularly receives a 403 where other IPs receive a 200 — this is a clear sign of a targeted ban on that address.
Proxy scoring algorithm
Instead of a binary "working / not working," it is more convenient to calculate a score from 0 to 100 for each IP, which gradually decreases with errors and recovers with successful requests. This allows you not to discard proxies after a single random failure (the marketplace server might just have hung for a second), but quickly remove those IPs that systematically return garbage.
The logic is simple: a successful request with a 200 code and valid content adds +2 to the score (but not above 100). A timeout results in a -15. A 403/429 code results in a -25. CAPTCHA in the response results in a -20. If the score drops below a threshold (for example, 30), the proxy goes into quarantine for a fixed time, and if after returning from quarantine the score again drops below 10 — the IP is permanently removed from the pool and logged for analysis.
This approach is similar to the reputation system in anti-detect browsers like Dolphin Anty or AdsPower, where accounts have "health" based on activity. The same mechanics work well for IP addresses in scraping.
Code: ProxyPool with automatic filtering in Python
Below is a working framework for a class to manage a proxy pool considering error metrics. It can be integrated with any parser using requests, httpx, or Scrapy by replacing the proxy retrieval logic with a call to pool.get_proxy().
import time
import random
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class ProxyStats:
address: str
score: int = 100
quarantine_until: float = 0.0
total_requests: int = 0
total_success: int = 0
consecutive_timeouts: int = 0
is_dead: bool = False
@property
def success_rate(self) -> float:
if self.total_requests == 0:
return 1.0
return self.total_success / self.total_requests
class ProxyPool:
def __init__(self, proxies: list[str],
min_score: int = 30,
dead_score: int = 10,
quarantine_seconds: int = 1800):
self.stats = {p: ProxyStats(address=p) for p in proxies}
self.min_score = min_score
self.dead_score = dead_score
self.quarantine_seconds = quarantine_seconds
def _is_available(self, stats: ProxyStats) -> bool:
if stats.is_dead:
return False
if stats.quarantine_until > time.time():
return False
return True
def get_proxy(self) -> Optional[str]:
available = [s for s in self.stats.values() if self._is_available(s)]
if not available:
return None
# sort by score, take a random one from the top 30% to avoid hammering one IP
available.sort(key=lambda s: s.score, reverse=True)
top_pool = available[:max(1, len(available) // 3)]
return random.choice(top_pool).address
def report_success(self, proxy: str):
s = self.stats[proxy]
s.total_requests += 1
s.total_success += 1
s.consecutive_timeouts = 0
s.score = min(100, s.score + 2)
def report_timeout(self, proxy: str):
s = self.stats[proxy]
s.total_requests += 1
s.consecutive_timeouts += 1
s.score -= 15
if s.consecutive_timeouts >= 3:
self._quarantine(s)
self._check_dead(s)
def report_blocked(self, proxy: str, captcha: bool = False):
s = self.stats[proxy]
s.total_requests += 1
s.score -= 20 if captcha else 25
self._quarantine(s)
self._check_dead(s)
def _quarantine(self, s: ProxyStats):
if s.score < self.min_score:
s.quarantine_until = time.time() + self.quarantine_seconds
def _check_dead(self, s: ProxyStats):
if s.score <= self.dead_score:
s.is_dead = True
def stats_report(self) -> list[dict]:
return [
{
"proxy": s.address,
"score": s.score,
"success_rate": round(s.success_rate, 2),
"dead": s.is_dead,
}
for s in self.stats.values()
]
In a real parser, calls to report_success, report_timeout, and report_blocked are inserted directly into the response handler after each HTTP request. Here’s an example of integration with requests:
import requests
def fetch(url: str, pool: ProxyPool):
proxy = pool.get_proxy()
if not proxy:
raise RuntimeError("No available proxies in the pool")
proxies = {"http": proxy, "https": proxy}
try:
resp = requests.get(url, proxies=proxies, timeout=8)
except requests.exceptions.Timeout:
pool.report_timeout(proxy)
return None
if resp.status_code in (403, 429):
pool.report_blocked(proxy)
return None
if "captcha" in resp.text.lower():
pool.report_blocked(proxy, captcha=True)
return None
pool.report_success(proxy)
return resp
This approach alleviates the burden of manual monitoring: you set the thresholds once, and then the pool "recovers" itself, pushing out dead addresses and returning those that temporarily hung from quarantine.
Threshold values for banning IPs
There are no universal numbers — thresholds depend on the platform and the aggressiveness of its anti-bot. However, there are benchmarks that work well in practice when scraping Wildberries, Ozon, and Avito:
- Success rate below 70% on a sample of 30-50 requests — a reason to remove the IP, even if the current score is not yet critical.
- 3 consecutive timeouts — almost always means that the proxy is physically unavailable or overloaded, no point in keeping it in the active pool.
- CAPTCHA more than once in 20 requests — the IP is already under suspicion by the anti-bot, better to reduce the frequency of requests or replace the address.
- Average response time above 2x the median of the pool — a signal of degradation, not critical, but a reason to lower priority when selecting.
A quarantine of 30-60 minutes is usually sufficient for data center and residential proxies for the anti-bot to "forget" about temporary suspicious activity. For mobile proxies, the situation is different: there, the IP changes when the operator switches cells, so the quarantine can be shorter — 10-15 minutes, as after rotation it is physically a different address.
Healthcheck and background rotation
In addition to reactive filtering (based on actual errors in the main parsing stream), it is useful to run a background healthcheck — a separate process that pings the proxies with light requests to a neutral URL (for example, httpbin.org/ip) every few minutes and updates the pool statistics before the proxy hits a live request to Wildberries or Ozon.
import threading
def healthcheck_loop(pool: ProxyPool, interval: int = 300):
while True:
for proxy in list(pool.stats.keys()):
try:
r = requests.get(
"https://httpbin.org/ip",
proxies={"http": proxy, "https": proxy},
timeout=5
)
if r.status_code == 200:
pool.report_success(proxy)
else:
pool.report_blocked(proxy)
except requests.exceptions.RequestException:
pool.report_timeout(proxy)
time.sleep(interval)
threading.Thread(target=healthcheck_loop, args=(pool,), daemon=True).start()
Such a background process is especially important if the pool is large (hundreds of IPs) and scraping is not continuous but in batches — without healthcheck, some proxies simply manage to "expire" between runs, and the first minutes of scraping are spent on live filtering instead of immediately working with verified addresses.
Which type of proxy to choose for marketplace scraping
The choice of proxy type directly affects how often filtering will need to be run and how aggressive the thresholds will be. For stable scraping of Wildberries and Ozon, where the anti-bot analyzes behavioral patterns and subnet reputation, residential proxies perform best — their IPs belong to real home internet providers, and the anti-bot blocks them in batches by subnet much less frequently.
For simpler and less protected sources (for example, collecting public data from Avito or monitoring prices on uncomplicated storefronts), data center proxies are often sufficient — they are faster and cheaper, which is critical for large volumes of requests, but they also get banned slightly more often, so the filtering thresholds for them should be stricter.
| Proxy Type | Filtering Speed | When to Use |
|---|---|---|
| Data Center | High, subnets are banned in batches | Simple storefronts, large volumes, low protection |
| Residential | Medium, IPs are banned selectively | Wildberries, Ozon, anti-bot sites |
| Mobile | Low, IP changes automatically by the operator | Heavily protected sources, high-frequency scraping |
In practice, many teams use a hybrid pool: the majority of requests go through residential IPs, and in case of mass bans, mobile proxies are connected as an "emergency" reserve, since their IP change occurs naturally through cellular network rotation and does not require manual intervention.
Conclusion
A proxy pool without a filtering system inevitably degrades, and the longer it operates without cleaning, the more resources the parser wastes on dead IPs. A scoring model that takes into account HTTP codes, timeouts, CAPTCHA, and success rate allows for the automatic identification of problematic addresses and keeps the pool operational without manual control. The code from this article can be adapted to any stack — requests, httpx, Scrapy, or your own asynchronous parser.
If you are just building a pool for marketplace scraping and want to reduce the frequency of bans right from the start, we recommend looking into residential proxies — they provide a significantly higher success rate on protected platforms like Wildberries and Ozon compared to regular data center addresses.