Distributed Parser on Multiple Servers: Dividing Proxy Pool Without Conflicts and Bans

When a parser moves from a single server to a cluster of multiple workers, a new problem arises — multiple machines start using the same proxy pool simultaneously. Without coordination, this leads to duplicate IPs in requests, exceeding limits for a single address, and mass bans. In this article, we will discuss how to properly shard proxies between servers, which distribution schemes work best, and how to implement this in practice with Python and Redis code.

Why Proxies Conflict in a Distributed Parser

Let's consider a typical situation: you have a pool of 500 proxies and three servers, each running 50 parallel workers for parsing Wildberries, Ozon, or Avito. If each server independently takes proxies from the common list without coordination, two workers on different machines may simultaneously send requests from the same IP to the same domain. For anti-bot systems, this looks like an abnormally high request frequency from one address — and the IP gets banned in a matter of minutes.

The second problem is uneven load distribution. If proxies are simply shared "as they come," one server may take the fastest and freshest IPs, while another gets the leftovers with already tarnished reputations. As a result, the parsing speed on the servers will differ significantly, as will the error rates.

The third problem is state loss. If a proxy is banned on one server, the other servers will not know about it and will continue to use it until they also get banned. Without centralized status tracking for proxies, each server "reinvents the wheel," wasting requests.

The solution to all three problems is to introduce a unified coordination mechanism: either through a centralized state store (Redis, PostgreSQL, Consul) or through predetermined sharding of the pool without overlapping ranges between servers.

Architecture of a Distributed Parser

Before dividing the proxies, it is necessary to determine the overall architecture of the cluster. There are two basic approaches.

Static scheme (shared-nothing). The proxy pool is divided among the servers in advance, once at startup. Each server receives its non-overlapping segment and works with it autonomously, without referring to other servers. The advantage is maximum simplicity and fault tolerance: the failure of one server does not affect the others. The downside is that if half of the proxies are banned on one server, it cannot "borrow" fresh IPs from a neighbor.

Dynamic scheme (shared-state). All servers refer to a common storage (most often Redis), which keeps track of the status of each proxy and issues it upon request with a lock. The advantage is uniform utilization of the pool and instant response to bans. The downside is that the storage becomes a single point of failure, requiring separate logic in case it becomes unavailable.

In practice, for parsing marketplaces and social networks, a hybrid approach is often chosen: a broker on Redis issues proxies with a short-term lock (TTL), and when Redis is unavailable, servers temporarily switch to a local fallback pool. This approach combines the reliability of the static scheme with the flexibility of the dynamic one.

Methods for Sharing a Proxy Pool Between Servers

Let's consider specific algorithms that can be applied for dividing the pool.

Simple modulo division (round-robin sharding). Number the proxies from 0 to N-1, and the server with number i takes all proxies for which index % number_of_servers == i. Simple, but poorly handles the addition or removal of servers — when the number of machines changes, almost the entire pool is redistributed anew.

Consistent hashing. Each server and each proxy is assigned a position on a hash ring (usually based on the hash of the IP address or server ID). The proxy is bound to the nearest server on the ring. When adding or removing a server, only a small portion of proxies are redistributed, not the entire pool — this is a key advantage for clusters that scale dynamically (for example, in Kubernetes with auto-scaling).

Centralized broker with locks. Instead of strictly binding proxies to a server, all servers request a "free" proxy from a common service (Redis, RabbitMQ), which issues it with a temporary lock. After use, the proxy is returned to the pool. This is the most flexible option, especially if the load on the servers is uneven.

Domain or task-based division. If the parser works with multiple sources simultaneously (for example, Wildberries, Ozon, and Avito in parallel), it makes sense to divide not only the proxies among servers but also the proxies among sources — so that the same IP does not appear on multiple platforms at once. This reduces the risk of the IP being blacklisted by anti-fraud systems if such platforms share data.

Method	Complexity	Scalability Resilience	When to Use
Modulo Division	Low	Poor	Fixed Number of Servers
Consistent Hashing	Medium	Good	Cluster Auto-Scaling
Centralized Broker	Medium-High	Excellent	Uneven Load, High Traffic
Domain Division	Low	Medium	Parsing Multiple Platforms Simultaneously

Implementing a Proxy Broker on Redis

The most practical option for most distributed parsers is a centralized broker on Redis. Redis is suitable because it is fast, supports atomic operations, and has TTL "out of the box." The idea is simple: all available proxies are stored in a Redis Set, and when issuing a proxy to a specific worker, it is locked for a certain period via a key with TTL.

Below is a working example of a broker in Python using the redis-py library. It implements the issuance of a free proxy with a lock, release after use, and marking the proxy as banned.

import redis
import time
import random

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)

PROXY_POOL_KEY = "proxy:pool"       # set of all working proxies
LOCK_PREFIX = "proxy:lock:"         # prefix for locking a specific proxy
BANNED_PREFIX = "proxy:banned:"     # prefix for ban marking
LOCK_TTL = 30                       # seconds to lock the proxy
BAN_TTL = 3600                      # seconds to consider the proxy banned

def load_pool(proxy_list):
    """Load the list of proxies into Redis once at cluster startup."""
    if proxy_list:
        r.sadd(PROXY_POOL_KEY, *proxy_list)

def get_free_proxy(server_id, max_attempts=20):
    """Each server calls this function to get a free proxy."""
    candidates = list(r.smembers(PROXY_POOL_KEY))
    random.shuffle(candidates)

    for proxy in candidates[:max_attempts]:
        if r.exists(BANNED_PREFIX + proxy):
            continue
        lock_key = LOCK_PREFIX + proxy
        # atomic lock setting if the proxy is free
        acquired = r.set(lock_key, server_id, nx=True, ex=LOCK_TTL)
        if acquired:
            return proxy
    return None  # no free proxies found, fallback needed

def release_proxy(proxy):
    """Release the proxy after a successful request."""
    r.delete(LOCK_PREFIX + proxy)

def mark_banned(proxy):
    """Mark the proxy as banned for BAN_TTL seconds, remove the lock."""
    r.set(BANNED_PREFIX + proxy, 1, ex=BAN_TTL)
    r.delete(LOCK_PREFIX + proxy)
    r.srem(PROXY_POOL_KEY, proxy)  # temporarily remove from the pool

def return_to_pool_if_recovered():
    """Periodic task: return proxies that have expired bans."""
    for proxy in r.scan_iter(match=BANNED_PREFIX + "*"):
        clean_proxy = proxy.replace(BANNED_PREFIX, "")
        if not r.exists(BANNED_PREFIX + clean_proxy):
            r.sadd(PROXY_POOL_KEY, clean_proxy)

The key point here is the use of SET ... NX EX in Redis. This command is atomic: if two servers try to lock the same proxy simultaneously, only one of them will succeed, the other will fail and go for the next candidate in the list. This is what eliminates conflicts between servers when accessing the common pool in parallel.

An example of using the broker in a parser worker:

import requests

def fetch_with_proxy(url, server_id):
    proxy = get_free_proxy(server_id)
    if not proxy:
        raise RuntimeError("No free proxies in the pool")

    proxies = {"http": f"http://{proxy}", "https": f"http://{proxy}"}
    try:
        response = requests.get(url, proxies=proxies, timeout=10)
        if response.status_code in (403, 429):
            mark_banned(proxy)
        else:
            release_proxy(proxy)
        return response
    except requests.exceptions.RequestException:
        mark_banned(proxy)
        raise

This scheme scales easily — you can add 10 or 50 servers, and they will all access the same Redis instance, and there will be no IP conflicts because the lock is atomic at the storage level, not at the individual server level.

Sticky Sessions and Binding Proxies to Tasks

For some tasks, simply issuing "any free proxy" is not enough. If the parser emulates user behavior — for example, sequentially browsing product cards, adding to the cart, navigating through catalog pages — it is important that all requests from a single session come from the same IP. Otherwise, the site will notice a sudden change in geo or device during the session and block it as suspicious.

To address this, the concept of a sticky session is introduced: the proxy is bound not to a single request but to a specific task or user session for its duration. In Redis, this is implemented through a separate key of the form session:{task_id} -> proxy with a TTL equal to the expected session duration.

SESSION_PREFIX = "proxy:session:"
SESSION_TTL = 600  # 10 minutes for the session

def get_sticky_proxy(task_id, server_id):
    existing = r.get(SESSION_PREFIX + task_id)
    if existing:
        return existing
    proxy = get_free_proxy(server_id)
    if proxy:
        r.set(SESSION_PREFIX + task_id, proxy, ex=SESSION_TTL)
    return proxy

For tasks with long sessions — for example, browsing the Wildberries catalog while emulating real buyer behavior — it is critical to use proxies that behave as much like a regular user as possible. Residential proxies are well-suited for this, as they are tied to real home IPs and are less likely to raise suspicion with anti-bot systems of marketplaces compared to data center addresses.

Ban Handling and Rotation in the Cluster

In a distributed system, it is important that information about a proxy ban becomes instantly available to all servers, not just the one that received the ban. In the example above, this is handled by the mark_banned function — as soon as one server records a 403 or 429 from the target site, the proxy is removed from the common pool in Redis, and the other servers automatically stop receiving it.

To determine a ban, it is worth analyzing not only the HTTP response code but also the content of the page — some sites return 200 OK, but with a captcha or placeholder instead of real content. The ban detection logic usually looks like this:

• HTTP codes 403, 429, 503 — clear signs of blocking or rate limiting.
• Redirect to a captcha or authorization page — check the URL after the request.
• Sudden decrease in response size or absence of expected CSS selectors — a sign of a placeholder.
• Connection timeout — often means that the proxy has stopped responding altogether.

After a ban is recorded, it makes sense not just to remove the proxy from the pool permanently, but to place it in "quarantine" with a TTL (as in the example — BAN_TTL), after which the proxy automatically returns to the pool. Many bans on marketplaces and social networks are temporary and are lifted after 30-60 minutes.

If bans occur too frequently even with even load distribution, this usually signals that the proxy pool itself has low quality or insufficient size for the required volume of requests. In such cases, expanding the pool with data center proxies for less sensitive tasks (e.g., checking page availability) and using more expensive residential IPs only for critical sessions helps reduce the overall ban rate without increasing costs for all traffic.

Monitoring the Health of the Proxy Pool

In a cluster of several servers, it is easy to lose the overall picture: how many proxies are currently active, how many are banned, what is the average response speed for each. Without monitoring, it is difficult to understand why parsing has slowed down — whether one server has gone down or if there are no live proxies left.

The minimum set of metrics that should be collected in Redis or a separate database includes:

• The number of active proxies in the pool at the current moment.
• The number of banned proxies and the average time until their recovery.
• The average response time for each proxy over the last N requests.
• The success rate of requests (success rate) for servers and for the pool as a whole.
• Load distribution — how many requests each server has processed over a period.

For visualization, it is convenient to aggregate these metrics into Prometheus through a simple exporter that reads data from Redis once a minute and provides it in a format understandable by Grafana. This allows for a quick view of the drop in success rate for a specific geo or type of proxy and timely addition of fresh IPs to the pool before the parser completely stops.

It is also important to monitor spikes in bans when launching new servers in the cluster — if the percentage of bans sharply increases when scaling from 3 to 6 servers, the problem is likely in sharding: new servers have started overlapping proxy ranges with the old ones. This is the first sign that it is necessary to switch from modulo division to consistent hashing.

Checklist for Launching a Distributed Parser

Before deploying the parser on multiple servers, go through the following points:

• The coordination mechanism for the pool has been chosen: static division, consistent hashing, or a centralized broker on Redis.
• Atomic locking of proxies has been implemented to prevent the use of the same IP by two servers simultaneously.
• TTL has been set for locks and bans to ensure proxies do not "hang" in a blocked state indefinitely when a worker fails.
• For tasks requiring user session emulation, sticky proxy binding has been set for the entire duration of the task.
• Ban detection has been configured not only by HTTP code but also by response content.
• Monitoring of success rates, active, and banned proxies across the entire cluster has been enabled.
• The proxy pool has been divided by task types: data center IPs for less sensitive requests, residential or mobile proxies for sessions where maximum similarity to a real user is important.
• The load on the state storage (Redis) has been checked — with a large number of servers and high request frequency, a Redis cluster may be needed instead of a single instance.

Conclusion

The main reason for conflicts when transitioning a parser to multiple servers is the lack of a single source of truth about the state of the proxy pool. Once a centralized broker with atomic locks (on Redis or a similar storage) is established, the problem of IP duplication and ban desynchronization resolves itself. The next step is to choose a sharding strategy suitable for the scale of your cluster: for a fixed number of servers, simple division is sufficient, while for a dynamically growing infrastructure, it is better to lay the groundwork for consistent hashing from the start.

If your distributed parser works with marketplaces or social networks where traffic plausibility at the individual session level is important, it is advisable to divide the pool by task type: inexpensive data center proxies are suitable for mass catalog scanning, while sessions emulating user behavior should use residential proxies with real home IPs, which significantly reduce the ban rate in systems sensitive to fraud.