Automatic Proxy Pool Scaling: How to Increase IPs Under Load and Save Budget

When working with large volumes of tasks — scraping marketplaces, farming accounts, mass posting on social media — a static proxy pool quickly becomes a problem. You either overpay for unused IPs during low-load periods or face blocks due to a lack of addresses during peak times. Automatic scaling of the proxy pool solves both problems: the system automatically increases the number of IP addresses according to the current load and reduces them when the tasks decrease.

In this article, we will discuss how to set up automatic scaling for different scenarios: scraping, traffic arbitrage, multi-accounting on social media, and working with marketplaces. We will show specific tools, load distribution algorithms, and metrics for monitoring.

What is Proxy Pool Scaling and Why is it Needed

Proxy pool scaling is the automatic adjustment of the number of active IP addresses based on the current load. In simple terms: when there are many tasks, the system adds proxies; when there are few, it disables the excess to avoid paying for downtime.

A classic example: you are scraping prices on Wildberries. On regular days, you need 50 IP addresses for 10,000 requests per hour. But on Friday evenings and weekends, the marketplace tightens limits and starts blocking repeated requests from the same IP more frequently. Without scaling, you either buy 150 proxies "just in case" (overpaying 200% on weekdays) or face blocks during peak hours.

With automatic scaling, the system monitors the percentage of 429 errors (Too Many Requests) and captchas. As soon as the rate exceeds 5%, it adds 20-30 IPs. When the load decreases, it disables the excess. The result: you only pay for the proxies you actually use and do not lose data due to blocks.

Main Advantages of Automatic Scaling

• Saving 40-60% of the budget — pay only for actively used IPs, not for a static pool "at maximum"
• Protection against blocks — the system instantly reacts to an increase in errors and adds proxies before mass bans occur
• Stable performance speed — the load is evenly distributed, with no drops during peak hours
• Flexibility for tasks — different scaling rules can be set for scraping, account farming, advertising

When is Automatic Scaling Needed: 5 Scenarios

Scaling of the proxy pool is not always necessary. If you manage 5 Instagram accounts or scrape 100 products a day — a static pool of 10-20 proxies will be sufficient. But there are tasks where automatic pool management is essential.

1. Scraping Marketplaces with Variable Load

A typical situation for monitoring prices on Wildberries, Ozon, Yandex.Market. During regular hours (from 3:00 AM to 10:00 AM), marketplaces provide data easily, with soft limits. During peak hours (from 6:00 PM to 11:00 PM), strict restrictions begin: captchas after 3-5 requests from one IP, subnet blocks, response delays.

Example: you scrape 50,000 products a day. At night, 30 IPs are sufficient for 2,000 requests per hour from each. In the evening, the same volume requires 100-120 IPs because the limit drops to 500-700 requests per IP. A static pool of 120 proxies running 24/7 results in a 75% overpayment during night hours. Scaling automatically raises the pool to 120 IPs from 6:00 PM to 11:00 PM, keeping 30-40 during other times.

2. Farming Facebook Ads and TikTok Ads Accounts

Arbitrageurs massively create and warm up accounts in ad cabinets. The task: to raise 50 Facebook accounts from scratch to launch the first campaign within a week. Each account requires a separate IP (otherwise, chain bans link all profiles).

But accounts are farmed unevenly: in the first 2 days, 50 profiles are actively working (requiring 50 proxies), on the 3rd-4th day, some accounts go to "rest" (20-30 IPs for active ones are sufficient), on the 5th-7th day, there is again a peak of activity before launching campaigns (again 50 IPs). With scaling, the system connects proxies only for active accounts, saving up to 50% over a week.

3. Mass Posting on Instagram and TikTok via SMM Panels

SMM agencies manage 50-200 client accounts. Posting follows a schedule: in the morning (9:00-11:00) stories are published, during the day (14:00-16:00) — posts in the feed, in the evening (19:00-21:00) — reels and comments. At other times, accounts are idle.

Each account needs a separate mobile proxy (Instagram strictly bans for IP changes). A static pool of 200 mobile proxies costs $4,000-6,000 per month. With scaling, you can maintain a basic pool of 50 IPs for constantly active accounts, and during mass posting hours, buy an additional 100-150 for 2-3 hours. Savings: up to $2,000 per month.

4. Automating Actions on Social Media (Likes, Follows, Comments)

Promotion through mass following, mass liking on Instagram, VK, TikTok. The task: 100 accounts perform 200-300 actions a day (follows, likes). Social networks track activity over time: if all 100 accounts start liking simultaneously — it raises a red flag for anti-fraud.

The right strategy: distribute activity over 12-16 hours, with 20-30 accounts working at any given moment. Scaling connects proxies only for active profiles. Instead of 100 permanent IPs, a pool of 30-40 is sufficient, rotating among accounts.

5. Testing Ad Creatives from Different Geos

Arbitrageurs and marketers test how ads appear in Facebook Ads, Google Ads, Yandex.Direct from different countries and cities. The task: to check 50 combinations (10 creatives × 5 geos) within 2 hours before launching the campaign.

Proxies from specific locations are needed: USA (5 states), Germany (3 cities), Poland, Kazakhstan, Ukraine. Keeping 50 IPs from different geos constantly is unprofitable — they are needed 2-3 times a week for a couple of hours. With scaling, you can rent proxies for an hour, test creatives, and disconnect. Savings: instead of $1,500 per month for a constant pool — $200-300 for one-time sessions.

Types of Scaling: Vertical vs Horizontal

There are two approaches to scaling a proxy pool. The choice depends on the type of task, budget, and speed requirements.

Vertical Scaling (Increasing Limits on IP)

You do not add new IP addresses but increase the number of requests through existing proxies. For example, instead of 1,000 requests per hour from one IP, you make 2,000, using more aggressive session rotation or switching user-agents.

When it is suitable: scraping sites with soft limits (news portals, forums, open APIs), where blocks are rare. You save on the number of proxies but risk getting banned for exceeding reasonable load.

Pros: no need to buy additional IPs, easier pool management, lower costs for proxies.

Cons: high risk of blocks on platforms with anti-fraud (social networks, marketplaces, ad cabinets). Not suitable for tasks where each account needs a unique IP.

Horizontal Scaling (Adding New IPs)

You increase the number of proxies in the pool: from 50 IPs to 100. The load is evenly distributed, each address operates within safe limits.

When it is suitable: multi-accounting on social media (each account needs its own IP), farming ad cabinets, scraping marketplaces with strict limits, working with anti-detect browsers (Dolphin Anty, AdsPower, Multilogin).

Pros: minimal risk of blocks, stable operation, suitable for long-term tasks (managing accounts for months).

Cons: higher costs for proxies, more complex to set up automatic pool management.

For most tasks related to social media, ad cabinets, and marketplaces, horizontal scaling is optimal. Vertical scaling makes sense only for scraping open sources with minimal restrictions.

Metrics for Scaling: What to Monitor

To allow the system to automatically decide on adding or disabling proxies, it is necessary to set up monitoring of key metrics. Let's discuss which indicators are critical for different tasks.

1. Error Rate

The most important metric. Track the ratio of successful requests to the total number. Critical error codes: 429 (Too Many Requests), 403 (Forbidden), 503 (Service Unavailable), as well as timeouts and captchas.

Normal values: for scraping — up to 2-3% errors, for working with social media accounts — up to 1%. If the indicator exceeds the threshold, the system should add 20-30% proxies to the current pool.

Example: you scrape Wildberries, with a pool of 50 IPs. You make 5,000 requests per hour, of which 200 return error 429 (4% Error Rate). Scaling trigger: add 15 proxies to reduce the load on each IP from 100 to 77 requests per hour.

2. Response Time

When the server is overloaded with requests from your IP, it starts responding slower or queues requests. If the average response time increases by 30-50% from the baseline — this is a signal for scaling.

Example: usually Ozon responds within 300-500 ms. During peak hours, the response time increased to 1200-1500 ms. This means the marketplace is throttling your requests. Solution: add proxies to reduce the frequency of requests from each IP.

3. CAPTCHA Rate

Critical for scraping marketplaces, search engines, social networks. If more than 5% of requests return a captcha — the pool is overloaded.

Example: scraping Google Shopping, out of 1,000 requests, 80 returned reCAPTCHA (8%). The system automatically adds 20 IPs to reduce the CAPTCHA Rate to 2-3%.

4. Proxy Utilization

Shows what percentage of proxies are actively used. If utilization is below 40% — you are overpaying for extra IPs. If above 85% — the pool is operating at its limit, with a high risk of blocks.

Optimal utilization: 60-75%. This is a balance between savings and stability.

Example: in a pool of 100 proxies, 35 are actively working (utilization 35%). The system disables 30 unused IPs, leaving 70. Savings: 30% of the proxy budget.

5. Task Queue Length

If the number of tasks in the queue exceeds what the system can process with the current pool — scaling is needed. Monitor the queue length and average wait time.

Example: you scrape 10,000 products. There are 3,000 tasks in the queue, and the current pool of 40 IPs processes 500 tasks per hour. Total time to complete all tasks: 6 hours. If you add 20 IPs, the time will reduce to 4 hours.

Automatic Scaling Algorithms

There are several approaches to automatic management of the proxy pool size. The choice of algorithm depends on the predictability of the load and the speed of response requirements.

1. Reactive Scaling

The system reacts to current metrics: if the Error Rate exceeds the threshold — it adds proxies, if utilization drops — it disables excess ones. This is the simplest and most popular approach.

Algorithm: every 5-10 minutes, the system checks the metrics. If at least one indicator is out of the norm — it makes a decision about scaling.

Pros: easy to set up, does not require historical data, works out of the box.

Cons: reacts with a delay (5-10 minutes), does not predict peak loads in advance. If the load suddenly increases — you will face blocks while the system adds proxies.

When to use: scraping with relatively stable load, when peaks are predictable by time (e.g., daily scraping at the same hours).

2. Proactive Scaling

The system analyzes historical data and predicts when the load will increase. Proxies are added in advance, before problems arise.

Algorithm: based on data from the past 7-30 days, the system builds a load chart by hours and days of the week. For example, every Friday from 6:00 PM to 11:00 PM, the Error Rate rises from 2% to 8%. The system automatically adds proxies on Friday at 5:45 PM to prevent the increase in errors.

Pros: no reaction delay, blocks are prevented before they occur, optimal proxy utilization.

Cons: requires accumulation of statistics (at least 2-4 weeks), does not cope with unpredictable spikes in load.

When to use: tasks with recurring load patterns (scraping marketplaces, price monitoring, regular posting on social media).

3. Hybrid Scaling

A combination of reactive and proactive approaches. The system uses historical data for planning but also instantly reacts to anomalies.

Algorithm: primary scaling is based on prediction (using statistics). But if metrics sharply exceed the norm — the system urgently adds proxies without waiting for the scheduled time.

Example: usually on Mondays from 10:00 AM to 12:00 PM, the load is stable, and the system maintains 50 IPs. But this Monday, Wildberries updated its anti-fraud, and the Error Rate rose to 12%. The hybrid algorithm instantly adds 30 proxies, even though scaling was not required according to the plan.

Pros: maximum stability, protection against unpredictable situations, optimal savings.

Cons: more complex to set up, requires more computational resources for data analysis.

When to use: critical tasks where blocks are unacceptable (farming expensive ad accounts, managing VIP clients in an SMM agency).

4. Scheduled Scaling

The simplest option: you manually set rules for when to add or disable proxies. For example: from Monday to Friday from 9:00 AM to 6:00 PM, keep 100 IPs, and at other times — 30 IPs.

Pros: maximum simplicity, does not require monitoring metrics, suitable for tasks with a clear schedule.

Cons: inflexibility, overpayment during low-load periods, risk of blocks during sudden peaks.

When to use: testing ad creatives (proxies are only needed at the moment of launching campaigns), one-off scraping tasks.

Tools for Implementation: Ready-made Solutions and APIs

For automatic scaling of the proxy pool, you can use both ready-made platforms and your own scripts via provider APIs. Let's discuss both options.

Ready-made Platforms with Automatic Scaling

Some services offer built-in tools for managing the proxy pool:

1. Bright Data (Luminati) — has an Auto-Scaling feature in Enterprise plans. The system automatically increases the pool during load increases, but the cost is high (from $500 per month for the basic package).

2. Smartproxy — offers an API for managing the number of IPs in real-time. You can set up a script that adds or removes proxies based on metrics via the API.

3. Oxylabs — has a Dashboard for monitoring metrics (Error Rate, Response Time). Scaling is manual, but it can be integrated via API for automation.

The downside of ready-made platforms is the high cost and reliance on a single provider. If prices rise or quality drops, switching to another provider will require reworking the entire infrastructure.

Self-Implementation via Provider APIs

A more flexible option is to write a script that monitors your system's metrics and manages the number of proxies via the provider's API. Most providers offer APIs for:

• Getting a list of active proxies
• Adding new IPs to the pool
• Disabling unused proxies
• Changing geolocation or type of proxy

Example logic for a reactive scaling script:

1. Every 5 minutes check metrics (Error Rate, CAPTCHA Rate, Response Time)
2. If Error Rate > 3%:
   - Calculate how many proxies need to be added (20-30% of the current pool)
   - Send a request to the provider's API: add N proxies
   - Update the parser configuration with the new list of IPs
3. If Proxy Utilization < 40%:
   - Identify unused proxies (no requests in the last 30 minutes)
   - Send a request to the API: disable these IPs
   - Update the parser configuration
4. Log all actions for efficiency analysis

For monitoring metrics, you can use:

• Prometheus + Grafana — free tools for collecting and visualizing metrics. Set up a dashboard with graphs for Error Rate, Response Time, Proxy Utilization.
• Datadog — monitoring platform (from $15 per month). Has ready-made integrations with popular scrapers.
• Custom scripts — the simplest option: a script in Python or Node.js that queries metrics from the parser logs every 5 minutes and makes scaling decisions.

Integration with Anti-Detect Browsers

If you work with multi-accounting through Dolphin Anty, AdsPower, Multilogin, or GoLogin, proxy scaling can be automated via the APIs of these browsers:

Dolphin Anty API — allows creating new profiles with unique proxies, updating IPs for existing profiles, and mass-switching proxies for a group of accounts.

Example scenario: you are farming 50 Facebook accounts. The script monitors how many accounts are active at the moment. If 30 are active — it keeps 30 proxies. If activity rises to 45 — it adds 15 new profiles with new IPs via the Dolphin API.

Step-by-Step Scaling Setup for Different Tasks

Let's consider specific scenarios for setting up automatic scaling for popular tasks.

Scenario 1: Scraping Marketplaces (Wildberries, Ozon)

Task: scrape 50,000 products daily, update prices every 6 hours. The load is uneven: at night, the marketplace provides data easily, while in the evening, blocks begin.

Step 1: Define the base pool. Start scraping during night hours (3:00-6:00) with the minimum number of proxies. Track how many IPs are needed for an Error Rate < 2%. For example, for 50,000 products, 30 residential proxies are sufficient.

Step 2: Collect statistics for a week. Record the Error Rate and CAPTCHA Rate by hours. You will see that from 6:00 PM to 11:00 PM, errors rise to 8-12%, and captchas appear in 10% of requests.

Step 3: Set up proactive scaling. Create a rule: every day at 5:45 PM, add 60 proxies (total 90 IPs), and at 11:15 PM, disable 60 (returning to 30 IPs).

Step 4: Add a reactive trigger for anomalies. If at any time the Error Rate exceeds 5% — urgently add 20 proxies.

Result: instead of a constant pool of 90 IPs (costing $180-270 per month), you pay for 30 IPs around the clock + 60 IPs for 6 hours a day. Savings: 40-50% of the budget.

Scenario 2: Farming Facebook Ads Accounts

Task: create and warm up 100 ad accounts within a month. Each account requires a unique IP, and activity is distributed unevenly.

Step 1: Divide accounts into groups by farming stages: new (1-3 days), warming (4-10 days), ready to launch (11-30 days). New accounts require daily activity, while ready ones need it 2-3 times a week.

Step 2: Set up scaling based on activity. In the first week, all 100 accounts are active — requiring 100 proxies. In the second week, 40 accounts transition to "ready" mode (requiring proxies only 3 days a week) — you can reduce the pool to 70 IPs on weekdays, 100 IPs on active days for ready accounts.

Step 3: Use the Dolphin Anty API for automatic proxy switching. The script monitors the activity schedule of each account. If an account is not active today — its proxy is disabled and used for another profile.

Result: instead of 100 permanent proxies, you maintain a pool of 60-70 IPs that rotate among accounts. Savings: 30-40% of the budget without the risk of chain bans.

Scenario 3: Mass Posting on Instagram

Task: An SMM agency manages 150 client accounts. Posting follows a schedule: 9:00-11:00 (stories), 14:00-16:00 (posts), 19:00-21:00 (reels).

Step 1: Identify peak hours. During mass posting moments, all 150 accounts are active; at other times — 20-30 (responding to comments, browsing the feed).

Step 2: Set up scaling based on the schedule. From 8:45 AM to 11:15 AM, raise the pool to 150 IPs, from 11:15 AM to 1:45 PM, reduce to 30 IPs, from 1:45 PM to 4:15 PM again to 150 IPs, and so on.

Step 3: Use mobile proxies for critical accounts (VIP clients, verified profiles) — they need a constant IP. For others, you can use residential proxies with scheduled rotation.

Result: a basic pool of 30 mobile proxies for VIP accounts ($600 per month) + 120 residential proxies that work 9 hours a day (60% savings compared to 24/7 rental). Total savings: $1,500-2,000 per month.

Cost Optimization: How Not to Overpay for Proxies

Automatic scaling is not only protection against blocks but also a tool for savings. Let's discuss specific tactics for reducing costs.

1. Combine Proxy Types for Tasks

Not all tasks require expensive residential or mobile proxies. Use a hybrid approach:

• Residential proxies — for critical tasks: account farming, working with ad cabinets, posting on social media.
• Mobile proxies — only for VIP accounts and platforms with strict anti-fraud (Instagram, TikTok for verified profiles).
• Data center proxies — for scraping open sources, monitoring prices on sites without aggressive anti-fraud.

Example: scraping Avito. For collecting ads, use data center proxies (5-10 times cheaper than residential). For posting ads, switch to residential — Avito checks IPs more strictly when publishing.

2. Set Up Aggressive Disabling of Unused Proxies

Many keep a "reserve" of proxies for peak loads but forget to disable them after the decline. Set up automatic disabling of IPs that have not been used in the last 30-60 minutes.

Example: in a pool of 100 proxies, 60 are actively working. After 30 minutes of downtime, the system automatically disables the 20 least used IPs. Savings: 20% of the budget daily.

3. Use Hourly Rental for One-off Tasks

Some providers offer pay-as-you-go or hourly rental. This is beneficial for:

• Testing ad creatives (proxies needed for 1-2 hours)
• One-off scraping of large data volumes
• Checking website availability from different geos

Instead of a monthly subscription for 50 IPs ($150-300), you rent them for 3 hours ($5-15).

4. Monitor Utilization and Adjust the Base Pool

Analyze proxy utilization statistics weekly. If the average utilization is consistently below 50% — reduce the base pool by 20-30%.

Example: you maintain a base pool of 80 IPs, with an average utilization of 35%. Reduce the base pool to 50 IPs, set up scaling to 80-100 during peak hours. Savings: $30-40 per month.

Common Mistakes in Scaling and How to Avoid Them

Even properly configured scaling can work inefficiently due to common mistakes. Let's discuss the most frequent problems.

Mistake 1: Too Slow to React to Increased Load