How to Bypass API Rate Limiting When Scraping with Proxies: IP Rotation Setup and Selection

You have set up a scraper, connected proxies, but the API still returns 429 "Too Many Requests" errors or blocks access? The problem is not with the proxies themselves, but with the incorrect strategy for their use. Rate limiting is a protective mechanism of the API that restricts the number of requests from a single IP address over a specified period. In this article, we will explore why blocks occur when working through proxies and how to properly configure the system to bypass limits.

What is API rate limiting and how does it work

Rate limiting is a mechanism to protect APIs from overload and abuse. The service sets a limit on the number of requests that can be made from a single source over a specific time period. For example, popular APIs use such limits:

• Twitter API: 300 requests per 15 minutes for standard access
• Instagram Graph API: 200 requests per hour per app
• Google Maps API: depends on the plan, usually 100-1000 requests per day
• Wildberries API: unofficial limits around 60 requests per minute from a single IP
• Avito API: 10 requests per second for scraping ads

There are several methods for determining the source of requests to which rate limiting is applied:

When the limit is exceeded, the API returns HTTP status 429 "Too Many Requests" and the Retry-After header, indicating the time until the limit is reset. Some services use a "sliding window," where the limit updates gradually, while others use a fixed window that resets at a specific time.

Why proxies do not automatically save you from rate limiting

Many developers mistakenly believe that simply connecting a proxy allows for an unlimited number of requests. In practice, the following problems arise:

Using a single proxy for all requests

If your script uses the same proxy IP address for all requests, the API sees this as a regular user and applies standard limits. For example, you set up a price scraper for Wildberries through a single residential proxy. The scraper makes 100 requests per minute, but the limit is 60 requests. The result: IP block for 10-30 minutes.

Slow IP rotation

Some use a pool of 5-10 proxies and switch between them sequentially. The problem is that each IP still reaches the limit faster than a full rotation occurs. Suppose you have 10 proxies and a limit of 100 requests per hour per IP. If you make 1000 requests per hour, each proxy will receive 100 requests — exactly at the limit. Any uneven distribution will lead to blocks.

Ignoring other identification factors

Even with perfect IP rotation, you can still be blocked if:

• All requests come from the same User-Agent (e.g., python-requests/2.28.0)
• One API key is used for all requests
• Requests come at perfect intervals (every 0.5 seconds) — this looks like a bot
• Proxy IPs are in the same subnet (e.g., all in the range 192.168.1.x)

IP address reputation

Data center proxies often end up on blacklists because their IPs are used by hundreds of other users for scraping. APIs may apply stricter limits to such addresses or block them outright. For example, Instagram and Facebook aggressively ban data centers, even if you do not exceed official limits.

IP rotation strategies to bypass limits

Proper proxy rotation is key to bypassing rate limiting. Let's consider effective strategies depending on the task.

Rotation after each request

The most aggressive strategy: each request goes through a new IP. Suitable for tasks with very strict limits (1-5 requests per IP) or when you need to spread the load as much as possible. For this, residential proxies with automatic rotation are used — they provide a pool of millions of IPs, and each request automatically gets a new address.

Example use case: scraping Instagram through an unofficial API, where the limit is 5 requests per minute from one IP. With rotation after each request, you can make 300 requests per minute through 300 different IPs.

Time-based rotation (sticky sessions)

The IP address is used for a certain period (5-30 minutes) and then changed to a new one. This strategy is suitable for APIs that require session persistence or when several related requests need to be made from one "user."

Calculating the optimal rotation time: if the API limit is 100 requests per hour, and you plan to make 50 requests through one IP, use a sticky session for 30 minutes. During this time, you will make 25 requests (with even load), which is half the limit.

Pool-based rotation with limit tracking

An advanced strategy: your script keeps track of the number of requests from each IP and automatically switches to a new one when approaching the limit. For example, you have a pool of 20 proxies, and the API limit is 100 requests per hour. The script tracks the counter for each IP and switches to the next one upon reaching 90 requests.

This strategy requires programming logic but provides maximum efficiency: you use each proxy to its full capacity without exceeding limits.

Geographical rotation

Some APIs apply different limits depending on the region. For example, a service may restrict requests from the USA more strictly than from Europe. In such cases, use proxies from different countries and distribute the load among them.

Configuring delays between requests

Even with perfect IP rotation, it is important to properly configure delays between requests. Too fast requests look like an attack, even if they come from different IPs.

Calculating minimum delay

Formula: delay = (time window in seconds / request limit) × safety factor

Example: The API allows 100 requests per hour (3600 seconds). Minimum delay = 3600 / 100 = 36 seconds. Adding a safety factor of 1.2: 36 × 1.2 = 43 seconds between requests from one IP.

If you are using 10 proxies with rotation, you can make requests every 4.3 seconds (43 / 10) without exceeding the limit on any IP.

Random delays (jitter)

Instead of a fixed delay of 5 seconds, use random intervals, for example, from 3 to 7 seconds. This makes your traffic resemble the actions of a real user. Many bot protection systems analyze patterns: if requests come exactly every 5.0 seconds, it looks suspicious.

Exponential delay on errors

When you receive a 429 error, do not continue sending requests immediately. Use exponential delay: the first attempt after 1 second, the second after 2, the third after 4, the fourth after 8, and so on. This is a standard practice that APIs expect from clients.

Which type of proxy to choose for working with APIs

The choice of proxy type critically affects the success of bypassing rate limiting. Let's analyze the pros and cons of each option for working with APIs.

Residential proxies

Residential proxies use IP addresses of real users assigned by Internet Service Providers. For APIs, this looks like regular home internet.

When to use: scraping Instagram, Facebook, TikTok, working with marketplace APIs (Wildberries, Ozon), any tasks where IP reputation is critical.

Mobile proxies

Mobile proxies use IPs from mobile operators (4G/5G). One IP is often used by thousands of real users simultaneously, so APIs rarely block them.

When to use: working with mobile APIs, scraping Instagram/TikTok in large volumes when maximum protection from bans is needed.

Data center proxies

Data center proxies are IP addresses of servers located in data centers. They are not associated with real users.

When to use: scraping public APIs without strict protection (weather, currency rates, news), working with your own APIs, testing, and development.

Practical implementation: code examples in Python

Let's consider specific examples of implementing rate limiting bypass using proxies. All examples are in Python using the requests library.

Simple rotation from a proxy pool

Basic implementation with cyclic rotation of IPs from a list:

import requests
import time
from itertools import cycle

# List of proxies (format: protocol://user:pass@host:port)
PROXY_LIST = [
    'http://user1:pass1@proxy1.example.com:8080',
    'http://user2:pass2@proxy2.example.com:8080',
    'http://user3:pass3@proxy3.example.com:8080',
]

# Create a cyclic iterator
proxy_pool = cycle(PROXY_LIST)

def make_request(url):
    proxy = next(proxy_pool)  # Get the next proxy from the pool
    proxies = {
        'http': proxy,
        'https': proxy
    }
    
    try:
        response = requests.get(url, proxies=proxies, timeout=10)
        return response
    except requests.exceptions.RequestException as e:
        print(f"Error with proxy {proxy}: {e}")
        return None

# Example usage
for i in range(10):
    response = make_request('https://api.example.com/data')
    if response and response.status_code == 200:
        print(f"Request {i+1}: success")
    time.sleep(2)  # Delay between requests

Rotation with limit tracking

A more advanced version that counts requests for each proxy and switches when approaching the limit:

import requests
import time
from collections import defaultdict

class ProxyRotator:
    def __init__(self, proxy_list, max_requests_per_ip=90, time_window=3600):
        self.proxy_list = proxy_list
        self.max_requests = max_requests_per_ip  # Request limit per IP
        self.time_window = time_window  # Time window in seconds
        self.request_counts = defaultdict(list)  # History of requests for each IP
        self.current_index = 0
    
    def get_proxy(self):
        """Returns the proxy with the least number of requests"""
        current_time = time.time()
        
        # Clear old records outside the time window
        for proxy in self.request_counts:
            self.request_counts[proxy] = [
                t for t in self.request_counts[proxy]
                if current_time - t < self.time_window
            ]
        
        # Find the proxy with the least number of requests
        available_proxies = []
        for proxy in self.proxy_list:
            count = len(self.request_counts[proxy])
            if count < self.max_requests:
                available_proxies.append((proxy, count))
        
        if not available_proxies:
            # If all proxies have exhausted their limit, wait
            oldest_request = min(
                min(times) for times in self.request_counts.values() if times
            )
            wait_time = self.time_window - (current_time - oldest_request) + 1
            print(f"All proxies have exhausted their limit. Waiting {wait_time:.0f} sec...")
            time.sleep(wait_time)
            return self.get_proxy()
        
        # Select the proxy with the least number of requests
        proxy = min(available_proxies, key=lambda x: x[1])[0]
        self.request_counts[proxy].append(current_time)
        return proxy
    
    def make_request(self, url, **kwargs):
        proxy = self.get_proxy()
        proxies = {'http': proxy, 'https': proxy}
        
        try:
            response = requests.get(url, proxies=proxies, timeout=10, **kwargs)
            return response
        except requests.exceptions.RequestException as e:
            print(f"Error with proxy {proxy}: {e}")
            return None

# Example usage
PROXY_LIST = [
    'http://user:pass@proxy1.example.com:8080',
    'http://user:pass@proxy2.example.com:8080',
]

rotator = ProxyRotator(PROXY_LIST, max_requests_per_ip=100, time_window=3600)

for i in range(500):  # Make 500 requests
    response = rotator.make_request('https://api.example.com/data')
    if response and response.status_code == 200:
        print(f"Request {i+1}: success")
    time.sleep(1)  # Minimum delay

Handling 429 errors with exponential delay

Proper handling of the "Too Many Requests" response considering the Retry-After header:

import requests
import time

def make_request_with_retry(url, proxies, max_retries=5):
    """Makes a request with automatic retries on 429 error"""
    
    for attempt in range(max_retries):
        try:
            response = requests.get(url, proxies=proxies, timeout=10)
            
            if response.status_code == 200:
                return response
            
            elif response.status_code == 429:
                # Check the Retry-After header
                retry_after = response.headers.get('Retry-After')
                
                if retry_after:
                    wait_time = int(retry_after)
                    print(f"Rate limit. Waiting {wait_time} sec (from Retry-After)")
                else:
                    # Exponential delay: 2^attempt seconds
                    wait_time = 2 ** attempt
                    print(f"Rate limit. Attempt {attempt+1}, waiting {wait_time} sec")
                
                time.sleep(wait_time)
                continue
            
            else:
                print(f"HTTP Error {response.status_code}")
                return None
        
        except requests.exceptions.RequestException as e:
            print(f"Connection error: {e}")
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)
                continue
            return None
    
    print(f"Exceeded number of attempts ({max_retries})")
    return None

# Example usage
proxies = {
    'http': 'http://user:pass@proxy.example.com:8080',
    'https': 'http://user:pass@proxy.example.com:8080'
}

response = make_request_with_retry('https://api.example.com/data', proxies)
if response:
    print("Data received:", response.json())

Using residential proxies with automatic rotation

Many residential proxy providers offer a single endpoint that automatically changes the IP with each request. Example configuration:

import requests
import random
import time

# Residential proxies with automatic rotation
# Format: protocol://username:password@gateway:port
ROTATING_PROXY = 'http://customer-USER:PASS@proxy.provider.com:12321'

def make_request_rotating(url):
    """Request through rotating proxy (new IP each time)"""
    proxies = {
        'http': ROTATING_PROXY,
        'https': ROTATING_PROXY
    }
    
    # Add a random User-Agent for more anonymity
    user_agents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36',
    ]
    
    headers = {
        'User-Agent': random.choice(user_agents)
    }
    
    try:
        response = requests.get(url, proxies=proxies, headers=headers, timeout=15)
        return response
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")
        return None

# Make 100 requests through different IPs
for i in range(100):
    response = make_request_rotating('https://api.example.com/data')
    if response and response.status_code == 200:
        print(f"Request {i+1}: success, IP changed")
    
    # Random delay of 1-3 seconds
    time.sleep(random.uniform(1, 3))

Monitoring limits and error handling

Effective work with APIs requires constant monitoring of limits and proper error handling. Here are key practices:

Analyzing response headers

Many APIs return information about limits in the response headers. Standard headers include:

• X-RateLimit-Limit — maximum number of requests in the window
• X-RateLimit-Remaining — how many requests are left
• X-RateLimit-Reset — time when the limit resets (Unix timestamp)
• Retry-After — how many seconds until you can repeat the request

Example of reading these headers:

response = requests.get(url, proxies=proxies)

# Check limit headers
limit = response.headers.get('X-RateLimit-Limit')
remaining = response.headers.get('X-RateLimit-Remaining')
reset_time = response.headers.get('X-RateLimit-Reset')

if remaining:
    remaining = int(remaining)
    if remaining < 10:
        print(f"Warning! Only {remaining} requests left")
        
if reset_time:
    import datetime
    reset_dt = datetime.datetime.fromtimestamp(int(reset_time))
    print(f"Limit will reset at {reset_dt}")

Logging and statistics

Keep detailed statistics of requests for each proxy. This will help identify problematic IPs and optimize rotation:

import json
from datetime import datetime

class RequestLogger:
    def __init__(self):
        self.stats = {}
    
    def log_request(self, proxy, status_code, response_time):
        if proxy not in self.stats:
            self.stats[proxy] = {
                'total': 0,
                'success': 0,
                'rate_limited': 0,
                'errors': 0,
                'avg_response_time': 0
            }
        
        self.stats[proxy]['total'] += 1
        
        if status_code == 200:
            self.stats[proxy]['success'] += 1
        elif status_code == 429:
            self.stats[proxy]['rate_limited'] += 1
        else:
            self.stats[proxy]['errors'] += 1
        
        # Update average response time
        current_avg = self.stats[proxy]['avg_response_time']
        total = self.stats[proxy]['total']
        self.stats[proxy]['avg_response_time'] = (
            (current_avg * (total - 1) + response_time) / total
        )
    
    def print_stats(self):
        print("\n=== Proxy Statistics ===")
        for proxy, data in self.stats.items():
            success_rate = (data['success'] / data['total'] * 100) if data['total'] > 0 else 0
            print(f"\nProxy: {proxy}")
            print(f"  Total requests: {data['total']}")
            print(f"  Successful: {data['success']} ({success_rate:.1f}%)")
            print(f"  Rate limited: {data['rate_limited']}")
            print(f"  Errors: {data['errors']}")
            print(f"  Average response time: {data['avg_response_time']:.2f}s")

# Usage
logger = RequestLogger()

start_time = time.time()
response = requests.get(url, proxies=proxies)
response_time = time.time() - start_time

logger.log_request(proxy, response.status_code, response_time)
logger.print_stats()

Automatic strategy switching

If you are constantly receiving 429 errors, automatically slow down requests or increase the proxy pool:

class AdaptiveRateLimiter:
    def __init__(self, initial_delay=1.0):
        self.delay = initial_delay
        self.consecutive_429 = 0
    
    def on_success(self):
        """Successful request - can speed up a bit"""
        self.consecutive_429 = 0
        self.delay = max(0.5, self.delay * 0.95)  # Decrease delay by 5%
    
    def on_rate_limit(self):
        """Received 429 - need to slow down"""
        self.consecutive_429 += 1
        self.delay *= 1.5  # Increase delay by 1.5 times
        
        if self.consecutive_429 > 5:
            print("WARNING: Too many 429 errors. Check settings!")
    
    def wait(self):
        """Wait before the next request"""
        time.sleep(self.delay)
        return self.delay

# Usage
limiter = AdaptiveRateLimiter(initial_delay=2.0)

for i in range(1000):
    response = make_request(url)
    
    if response.status_code == 200:
        limiter.on_success()
    elif response.status_code == 429:
        limiter.on_rate_limit()
    
    delay = limiter.wait()
    print(f"Request {i+1}, delay: {delay:.2f}s")

Handling captchas and other blocks

Some APIs display captchas instead of directly blocking when limits are exceeded. Signs include:

• Status code 403 with a response body containing "captcha" or "recaptcha"
• Redirect to a captcha page (status 302)
• Special headers like X-Captcha-Required: true

In such cases, you should:

1. Immediately stop using this IP
2. Switch to another proxy from the pool
3. Increase delays between requests
4. Add more variety to User-Agent and other headers

Conclusion

Bypassing API rate limiting when using proxies is not just a technical setup, but a comprehensive strategy that includes the right choice of proxy type, configuring IP rotation, managing delays, and monitoring limits. Key takeaways from the article:

• Proxies alone do not solve the rate limiting problem — a proper rotation strategy is needed
• For social networks and strict APIs, use residential or mobile proxies; for public APIs, data centers are suitable
• Calculate the size of the proxy pool based on API limits and desired scraping speed
• Always monitor limits and adjust your strategy accordingly