A classic mistake when scaling a parser is buying proxies "by eye": you take 100 IPs, launch the parser, and get banned within half an hour. The problem is not in the number of addresses, but in the fact that no one calculates the actual load on each IP. Let's break down the formula for calculating the proxy pool based on RPS (requests per second), delays, and limits of the target site — with specific figures and Python code.
Why counting the pool by the number of IPs is a mistake
The typical approach of a beginner in parsing: "I need to collect 50,000 product cards, so let's buy 500 proxies and distribute the load." The logic seems sound, but it does not take into account the main point — sites like Wildberries and Ozon ban not based on the absolute number of requests, but on the intensity of requests from one IP over time. That is, 500 proxies, each making 20 requests per second, will get banned instantly — anti-bot systems detect a pattern similar to DDoS.
On the other hand, if you have 50 proxies, but each makes 1 request every 5-10 seconds with random pauses, mimicking human behavior, you can parse steadily for weeks without a single ban. The number of IPs is not the reason for stability, but a consequence of correctly calculated load. That is why the formula should not start from "how many IPs to buy," but from "how much RPS needs to be achieved and what is the safe load per IP."
Another nuance: different types of proxies have different "tolerance" on one IP. Datacenter proxies get banned faster at high request frequencies because their subnets are easily recognized as hosting. Residential and mobile IPs look like regular users, and they can afford a slightly higher frequency without the risk of being blocked — but that does not mean that limits can be ignored altogether.
Basic formula for calculating the proxy pool
The formula for calculating the number of proxies in the pool looks like this:
N = (RPS_target × Delay_per_ip) / Concurrency_per_ip
Where:
- N — the required number of proxies in the pool;
- RPS_target — the target parsing speed (requests per second across the system);
- Delay_per_ip — the minimum safe pause between requests from one IP (in seconds);
- Concurrency_per_ip — how many parallel threads you allow on one IP (usually 1, a maximum of 2 for residential proxies).
The logic is simple: if you want to maintain 10 requests per second in total, and the safe pause between requests from one IP is 8 seconds, then one IP can physically only deliver 1 request every 8 seconds, which is 0.125 RPS. To achieve 10 RPS in total, you need 10 / 0.125 = 80 proxies. This is the calculation that replaces the guesswork of "500 IPs just in case."
How to calculate the target RPS for parsing
Before calculating the pool, you need to determine RPS_target — how many requests per second you actually need to collect data in a reasonable time. The formula here is the inverse:
RPS_target = Total_requests / Time_budget_seconds
Example: you need to collect 100,000 product cards from Wildberries in 8 hours (28,800 seconds). If each card takes 1 request, RPS_target = 100,000 / 28,800 ≈ 3.47 requests per second. This is not as much as it seems at first glance — many overestimate the required speed and buy an excessive number of proxies.
If the task involves several types of requests (for example, first getting a list of categories, then cards, then reviews), calculate the RPS for each stage separately — they can run in parallel with different proxy pools, and the total load on the site will be distributed across different endpoints.
Limits per IP: how many requests per minute are safe
Delay_per_ip is the most important parameter in the formula, and it needs to be determined empirically for each site. General guidelines based on marketplace parsing practice:
| Platform | Safe pause between requests from 1 IP | Maximum requests/min from 1 IP |
|---|---|---|
| Wildberries (API for cards) | 4-6 sec | 10-15 |
| Ozon (product pages) | 5-8 sec | 8-12 |
| Avito (ads) | 6-10 sec | 6-10 |
| Yandex.Market | 5-7 sec | 8-12 |
These figures are a starting point, not dogma. Start with conservative values (the upper limit of the pause), monitor the percentage of 429 and 403 errors, and gradually reduce the pause if bans do not increase. A sharp increase in RPS without gradual testing is the most common way to burn out the entire proxy pool in one day.
Practical calculations: Wildberries, Ozon, Avito
Let's analyze three real scenarios with a full calculation using the formula.
Scenario 1: price monitoring on Wildberries. Need to update prices for 20,000 products every 2 hours. RPS_target = 20,000 / (2 × 3600) ≈ 2.78 RPS. With a pause of 5 sec per IP and Concurrency = 1: N = (2.78 × 5) / 1 ≈ 14 proxies. To have a buffer in case some IPs get banned, it is recommended to take a pool with a factor of 1.5-2x, that is, 21-28 proxies.
Scenario 2: one-time collection of the Ozon catalog. 500,000 cards in 24 hours. RPS_target = 500,000 / 86,400 ≈ 5.79 RPS. With a pause of 6 sec: N = (5.79 × 6) / 1 ≈ 35 proxies. With a buffer — 50-60 proxies.
Scenario 3: real-time competitor monitoring on Avito. 5,000 ads, updating every 15 minutes. RPS_target = 5,000 / 900 ≈ 5.56 RPS. With a pause of 8 sec: N = (5.56 × 8) / 1 ≈ 45 proxies. Here it is important to consider that Avito actively blocks datacenter subnets, so for this task it is wiser to immediately allocate residential or mobile IPs.
Residential, mobile, and datacenter proxies in the formula
The type of proxy directly affects Delay_per_ip and, consequently, the final N in the formula. Datacenter proxies are cheaper and faster, but require longer pauses between requests and are banned more often at increased frequencies — in fact, for the same 5 RPS, you may need 2-3 times more datacenter IPs than residential ones.
| Proxy Type | Average Delay_per_ip | When to use |
|---|---|---|
| Datacenter proxies | 8-15 sec | Open APIs, sites without strict anti-bot measures |
| Residential proxies | 4-8 sec | Wildberries, Ozon, Avito, and other marketplaces with anti-bot measures |
| Mobile proxies | 3-6 sec | Most aggressive protections, social networks, mobile APIs |
For parsing marketplaces like Wildberries and Ozon, the optimal choice is usually residential proxies — they balance price and stability, allowing for shorter pauses without a sharp increase in bans. Datacenter proxies should only be considered for sites without aggressive anti-bot measures or for one-time tasks with low RPS.
Implementing the pool in Python: code with rotation
Below is a simplified implementation of a proxy pool with request frequency control for each IP. The logic: each proxy keeps track of the last used time, and the scheduler selects only those IPs that have had enough time since the last request.
import time
import random
from dataclasses import dataclass, field
from typing import List, Optional
@dataclass
class ProxyNode:
address: str
delay_per_ip: float # safe pause in seconds
last_used: float = field(default=0.0)
class ProxyPool:
def __init__(self, proxies: List[str], delay_per_ip: float):
self.nodes = [ProxyNode(address=p, delay_per_ip=delay_per_ip) for p in proxies]
def get_available_proxy(self) -> Optional[ProxyNode]:
now = time.time()
available = [
node for node in self.nodes
if now - node.last_used >= node.delay_per_ip
]
if not available:
return None
# choose randomly from available to avoid queue patterns
node = random.choice(available)
node.last_used = now
return node
def size(self) -> int:
return len(self.nodes)
def calculate_pool_size(rps_target: float, delay_per_ip: float, concurrency: int = 1) -> int:
"""Formula: N = (RPS_target * Delay_per_ip) / Concurrency_per_ip"""
return max(1, int((rps_target * delay_per_ip) / concurrency))
# Example usage
rps_target = 5.79 # calculated from the task volume and time
delay_per_ip = 6.0 # safe pause for Ozon
n_proxies = calculate_pool_size(rps_target, delay_per_ip)
print(f"Proxies needed in the pool: {n_proxies}") # ~35, with a buffer 50-60
In a real parser, this logic needs to be supplemented with a task queue (for example, using asyncio or multiprocessing), so that workers wait for an available proxy instead of failing with an error when none are available. It is also advisable to add exponential backoff when receiving 429/403 — this will automatically increase the pause for a specific IP when signs of blocking are detected.
Monitoring and dynamic adjustment of the pool
A static calculation using the formula is a starting point, not a final solution. In real operation, you need to monitor three key metrics:
- Success rate — the percentage of successful requests (code 200) out of the total. A drop below 90-95% indicates that Delay_per_ip needs to be increased or proxies need to be added to the pool;
- Ban rate — the share of proxies that showed signs of blocking (captcha, 403, redirect to verification form) in the last hour;
- Real RPS — the actual request processing speed, which may differ from the calculated due to timeouts and retries.
A practical rule: if the ban rate exceeds 5-7% per hour, increase Delay_per_ip by 20-30% and recalculate N using the formula. If the success rate remains above 98% for several hours, you can gradually reduce the pause and decrease the pool size — this is a direct budget saving on proxies without losing stability.
A good practice is to keep a log for each IP separately: the time of the last successful request, the number of consecutive errors, the average response time. This allows for the automatic exclusion of "damaged" proxies from rotation for 30-60 minutes instead of continuing to send requests through them and getting captcha at every step.
Common mistakes in calculating the proxy pool
Even knowing the formula, it is easy to make mistakes in the details of the calculation. Here is a list of typical pitfalls:
- Ignoring the buffer for bans. Even with perfect calculation, 5-10% of proxies will be temporarily unavailable due to blocks or network issues. Always add a factor of 1.3-2x to the base N;
- Same Delay_per_ip for all endpoints. The category page and the product card page may have different limits — calculate separately;
- Concurrency greater than 1 without testing. Parallel requests from one IP sharply increase the risk of bans, especially on residential proxies — start with Concurrency = 1;
- Lack of rotation of User-Agent and headers. Even a correctly calculated proxy pool will not save you if all requests come with the same browser fingerprint;
- Fixed pause without jitter. A strictly identical interval between requests (for example, exactly 5.0 sec) is a pattern easily recognizable by anti-bot systems. Add a random deviation of ±20-30%.
Conclusion
The formula N = (RPS_target × Delay_per_ip) / Concurrency_per_ip turns the calculation of the proxy pool from guessing into an engineering task with specific figures. First, determine the target parsing speed based on the data volume and time, then empirically find the safe pause for the specific site, and only after that calculate the required number of IPs — with a buffer of 30-100% for bans and failures.
This approach saves budget: instead of buying an excessive number of IPs "just in case," you pay exactly for the volume of proxies needed to achieve the target RPS without the risk of blocks. For parsing marketplaces and other protected sites, we recommend starting with residential proxies — they provide the best balance between stability and cost when working with the anti-bot systems of Wildberries, Ozon, and Avito.