When a parser or bot sends hundreds of requests per minute from a single IP, the server quickly activates the rate limit, leading to captcha or a ban. The solution is a task queue that distributes requests among workers and proxies. In this article, we will explore how such an architecture is structured in Celery, RQ, and BullMQ, and how to effectively manage the request limit for each IP address.
Why use a task queue when working with proxies
If the task is to scrape 50,000 product cards from Wildberries or check the status of 10,000 Instagram accounts, sequentially sending requests will take hours, while parallel execution without control will quickly lead to an IP ban. A task queue solves both problems: it breaks down a large workload into smaller tasks, distributes them among several workers, and allows each worker to be assigned a separate proxy.
The architecture typically looks like this: a message broker (Redis or RabbitMQ) holds the task queue, workers take tasks from the queue and execute them, with each worker using its own proxy or rotating proxies from a pool. This keeps the load on each IP within safe limits, even if the total volume of requests is huge.
This approach is used not only by parser developers but also by teams automating the verification of Facebook Ads accounts, monitoring prices on Ozon, or collecting data from dozens of websites simultaneously. Without a queue and proper distribution by IP, any such system is doomed to blockages on the very first day of operation.
The problem of request limits on a single IP
Most websites and APIs have protections against excessive load. This can manifest in various ways: the server returns HTTP 429 (Too Many Requests), shows a captcha, temporarily bans the IP for 10-60 minutes, or permanently blacklists it. Different platforms have vastly different limits — for example, the Wildberries API allows a limited number of requests per second from a single address, while Instagram reacts to suspicious activity after just 15-20 consecutive actions.
The main mistake when building a queue is launching all workers through the same IP. Even if tasks are distributed among 20 workers, if they all use one proxy or do not use one at all, the limit on that IP will be reached just as quickly as with sequential execution. Therefore, the number of workers should correlate with the number of available IP addresses in the proxy pool.
For tasks with a high frequency of requests to a single domain, it is better to use data center proxies — they provide low latency and are suitable for large-scale parallel scraping, where speed is more important than masking as a regular user. If the goal is to bypass stricter protections like anti-bot systems of marketplaces or social networks, IPs that resemble regular users, such as residential or mobile, are needed.
Celery: architecture and proxy rotation in code
Celery is the most mature and popular task queue in the Python ecosystem. It operates on top of a broker (most often Redis or RabbitMQ) and supports flexible configuration of concurrency, retries, and task priorities. For proxy rotation, it is convenient to store the pool of IPs in Redis and assign a proxy to each task when it is queued.
from celery import Celery
import requests
import redis
import random
app = Celery('tasks', broker='redis://localhost:6379/0')
redis_client = redis.Redis(host='localhost', port=6379, db=1)
PROXY_POOL = [
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
]
def get_proxy():
# Round-robin through Redis to avoid hitting one IP
idx = redis_client.incr("proxy_cursor") % len(PROXY_POOL)
return PROXY_POOL[idx]
@app.task(bind=True, max_retries=3, default_retry_delay=30)
def fetch_url(self, url):
proxy = get_proxy()
try:
response = requests.get(
url,
proxies={"http": proxy, "https": proxy},
timeout=15
)
if response.status_code == 429:
raise self.retry(exc=Exception("Rate limited"))
return response.status_code
except requests.exceptions.RequestException as exc:
raise self.retry(exc=exc)
The key point is the function get_proxy(), which uses a Redis counter to yield the next proxy from the pool in a round-robin fashion. This ensures that the load is evenly distributed among all IPs rather than concentrated on one. The max_retries parameter and handling of the 429 code allow the task to automatically retry with a different proxy if the current IP receives a limit.
To control concurrency at the worker level, Celery allows limiting the number of concurrent tasks through the --concurrency parameter when starting the worker, as well as through queues with different priorities (for example, a separate queue for slow tasks with strict rate limits).
RQ (Redis Queue): a simple alternative
RQ is a lighter library than Celery, without RabbitMQ broker and with minimal configuration. It is suitable if the project is small and does not require complex task routing, but simply needs a queue with workers and the ability to bind proxies to tasks.
from redis import Redis
from rq import Queue
import requests
import itertools
redis_conn = Redis()
q = Queue(connection=redis_conn)
PROXY_POOL = [
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
]
proxy_cycle = itertools.cycle(PROXY_POOL)
def fetch_with_proxy(url):
proxy = next(proxy_cycle)
response = requests.get(
url,
proxies={"http": proxy, "https": proxy},
timeout=15
)
if response.status_code == 429:
# Re-queue with a delay
q.enqueue_in(60, fetch_with_proxy, url)
return None
return response.status_code
# Queueing tasks
urls = ["https://example.com/page1", "https://example.com/page2"]
for url in urls:
q.enqueue(fetch_with_proxy, url)
In RQ, rotation through itertools.cycle works within the worker process — this is simpler than a Redis counter but scales poorly with many parallel workers since each worker starts the cycle from the beginning. For small projects with 1-5 workers, this is sufficient; for larger ones, it is better to revert to a centralized counter in Redis, as in the Celery example.
RQ is well-suited for developers already using Redis in their project and who do not want to set up a separate message broker. The downside is fewer built-in prioritization and monitoring mechanisms compared to Celery, so for production systems handling hundreds of thousands of tasks per day, Celery is often chosen.
BullMQ: a queue on Node.js with IP limitations
BullMQ is the standard for task queues in the Node.js ecosystem, built on Redis and has a built-in rate limiter, which is especially convenient for controlling request limits on IPs. Instead of manually counting requests, you can specify the limiter directly at the queue level.
const { Queue, Worker } = require('bullmq');
const axios = require('axios');
const connection = { host: 'localhost', port: 6379 };
const proxyPool = [
'http://user:[email protected]:8000',
'http://user:[email protected]:8000',
'http://user:[email protected]:8000',
];
const queue = new Queue('fetchQueue', { connection });
// Limiter: no more than 5 tasks per second on the queue
const worker = new Worker('fetchQueue', async job => {
const proxy = proxyPool[job.id % proxyPool.length];
try {
const response = await axios.get(job.data.url, {
proxy: false,
httpsAgent: new (require('https-proxy-agent'))(proxy),
timeout: 15000,
});
return response.status;
} catch (err) {
if (err.response && err.response.status === 429) {
throw new Error('Rate limited, retrying with another proxy');
}
throw err;
}
}, {
connection,
limiter: { max: 5, duration: 1000 },
concurrency: proxyPool.length,
});
for (let i = 0; i < 100; i++) {
queue.add('fetch', { url: `https://example.com/item/${i}` }, {
attempts: 3,
backoff: { type: 'exponential', delay: 5000 },
});
}
The limiter parameter in BullMQ is a built-in mechanism that restricts the total number of tasks processed over a period of time, which is convenient for adhering to request limits at the system level. The concurrency parameter, equal to the size of the proxy pool, ensures that exactly as many workers are running simultaneously as there are available IPs.
This approach is often used by teams writing their own bots for automating actions in TikTok or Telegram — BullMQ integrates well with the rest of the Node.js stack and does not require switching to Python for a single task.
Comparison of Celery, RQ, and BullMQ
| Parameter | Celery | RQ | BullMQ |
|---|---|---|---|
| Language | Python | Python | Node.js |
| Broker | Redis / RabbitMQ | Only Redis | Only Redis |
| Built-in rate limiter | No (manual implementation required) | No | Yes, out of the box |
| Entry threshold | Medium | Low | Medium |
| Best suited for | Large production systems | Small projects, prototypes | Node.js stack, bots, automation |
Strategies for load distribution by IP
Round-robin, as used in the examples above, is the simplest strategy but not always optimal. Here are a few options to consider depending on the task:
- Round-robin — each subsequent request goes through the next proxy in a circle. Suitable when all IPs are equal and their limits are the same.
- Sticky session — one proxy is tied to a specific account or session for the entire duration of interaction. Critical for social networks and marketplaces, where changing IPs mid-session appears suspicious.
- Weighted distribution — some requests are directed to more expensive and reliable IPs (e.g., mobile), while others go to cheaper data center ones, depending on the domain being processed.
- Adaptive throttling — the speed of sending tasks is automatically reduced if the percentage of 429 errors or captchas increases, and restored when errors disappear.
For tasks where masking as a real user is important — for example, warming up and checking accounts on Instagram or TikTok through anti-detect browsers like Dolphin Anty or AdsPower — it is better to use sticky sessions with mobile proxies: they are less likely to be flagged by anti-bot systems and behave like ordinary mobile internet users.
For mass data collection from websites, where speed and volume are important rather than algorithm trust, round-robin with residential proxies provides a balance between stability and cost: IPs appear as real users, but can be changed frequently without risking a specific session.
Monitoring errors 429 and bans
A task queue without monitoring is a black box: it is difficult to understand why some tasks fail and which IPs are already under suspicion. The minimum set of metrics to track includes:
- Percentage of responses with code 429 and 403 for each proxy separately
- Average response time — an increase in latency often precedes a ban
- Number of retries for each task
- Length of the task queue in real-time — if it is growing, workers are overwhelmed
In Celery, Flower is a convenient tool for monitoring tasks and workers in real-time. BullMQ has Bull Board with similar functionality. RQ has RQ Dashboard. All three tools show the status of tasks, the number of failed attempts, and allow quickly identifying which proxy from the pool should be temporarily excluded from rotation.
A good practice is to automatically mark proxies as "in quarantine" for 15-30 minutes after several consecutive 429 errors from that IP, and return it to the pool only after a successful test request. This prevents a banned IP from continuing to receive tasks and wasting workers' time.
Conclusion
A task queue solves the problem of scaling requests, but it does not protect against IP limits on its own — a well-thought-out proxy rotation within Celery, RQ, or BullMQ is necessary for that. The choice of tool depends on the stack: Python projects often choose Celery for production and RQ for simple tasks, while Node.js developers prefer BullMQ with its built-in rate limiter.
Regardless of the chosen queue, the result heavily depends on the quality of the proxy pool itself. If the task is fast mass scraping, consider data center proxies for speed, while for working with social networks and marketplaces, where loyalty to anti-bot systems is crucial, residential or mobile IPs are suitable — they reduce the percentage of 429 errors and bans, which directly decreases the number of retries and speeds up the execution of the entire task queue.