Collecting thousands of images for training a computer vision model is a task that almost always faces two problems: blocking from the data source (Google Images, Pinterest, marketplaces, stock sites) and unexpectedly high traffic consumption. One IP address quickly gets banned, and the cost of proxy traffic can exceed the cost of labeling the dataset itself. In this article, we will discuss how to design an image parser that does not lose data due to blocks and how to calculate the traffic budget in advance.
Why Use Proxies When Collecting Images for CV
Datasets for computer vision are rarely collected from a single source. Typically, it is a combination of image search results, product cards on marketplaces, stock photos, social media, and open APIs. Almost all of these sources impose limits on the number of requests from a single IP: Google Images shows a captcha after just a few hundred consecutive requests, Pinterest temporarily blocks access to its API, and Wildberries and Ozon return empty cards or redirect to a verification page when suspicious activity is detected.
Without proxies, you hit a ceiling of several hundred or thousands of images per day from one address. If you need a dataset of 50,000–500,000 images (which is typically required for convolutional networks and transformers in classification and detection tasks), it is physically impossible to collect it in a reasonable time without a distributed pool of IP addresses — you will either constantly encounter blocks or obtain a heavily skewed sample due to time-of-day and geolocation restrictions.
The second reason is the geographical diversity of the data. If the model needs to recognize objects equally well regardless of the region of capture (for example, road signs in different countries or products on local marketplaces), the parser must connect with IPs from different countries. A single datacenter with one outgoing IP cannot provide such diversity.
How Much Traffic is Actually Used for the Dataset
Traffic consumption when collecting images is often underestimated. It is not only about the weight of the images themselves — HTML pages, JS bundles, previews, tracking pixels, and CSS also go through the proxy if the parser is not specifically configured for the required URLs.
| Content Type | Average Weight | Traffic for 10,000 Items |
|---|---|---|
| Preview/thumbnail (150–300 px) | 15–40 KB | ~250 MB |
| Medium Resolution Photo (800–1200 px) | 80–200 KB | ~1.4 GB |
| High Resolution Photo (2000+ px) | 400 KB – 1.5 MB | ~9.5 GB |
| HTML Page of Product Card (without images) | 150–400 KB | ~2.7 GB |
In practice, if you are collecting 100,000 medium resolution images along with product card pages (for extracting labels — categories, boxes, attributes), the total traffic consumption usually falls within the range of 25–45 GB. If high-resolution images are needed for detecting small objects — expect 80–150 GB for the same 100,000 items. These figures should be factored into your proxy plan in advance, not after the fact.
Datacenter, Residential, and Mobile: What to Choose
For image collection, three parameters are important: speed, traffic cost, and resistance to blocks. These parameters differ significantly among different types of proxies.
| Parameter | Datacenter | Residential | Mobile |
|---|---|---|---|
| Speed | High | Medium | Low-Medium |
| Resistance to Bans | Low | High | Very High |
| Traffic Cost | Low | Medium-High | High |
| Best For | Open APIs, stocks without strict protection | Google Images, Pinterest, marketplaces | Social networks with strict anti-bot protection |
For most image collection tasks, residential proxies provide the optimal balance — IPs appear as regular users, so anti-bot systems of marketplaces and image search engines trigger captchas less frequently. If the source is an open stock photo site or an API without strict limits, it is cheaper to use datacenter proxies — they are faster and allow for larger volumes of traffic at a lower cost per GB. For collecting images from mobile applications or social networks with advanced anti-bot protection, mobile proxies are better suited — they use real IPs from telecom operators and almost never get blocked.
Basic Parser Using Requests with Proxy Rotation
To get started, a simple synchronous script using requests with proxy rotation for each request or after a specified number of requests will suffice. Below is an example of downloading images from a list of URLs with content type checking and file size limitation to avoid accidentally downloading a heavy file of the wrong format.
import requests
import itertools
import hashlib
import os
PROXIES = [
"http://user:[email protected]:10000",
"http://user:[email protected]:10000",
"http://user:[email protected]:10000",
]
proxy_cycle = itertools.cycle(PROXIES)
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "image/avif,image/webp,image/*,*/*;q=0.8",
}
def download_image(url, save_dir="dataset", max_size_mb=10, timeout=15):
proxy = next(proxy_cycle)
proxies = {"http": proxy, "https": proxy}
try:
resp = requests.get(url, headers=HEADERS, proxies=proxies, timeout=timeout, stream=True)
resp.raise_for_status()
content_type = resp.headers.get("Content-Type", "")
if "image" not in content_type:
return None
content_length = int(resp.headers.get("Content-Length", 0))
if content_length > max_size_mb * 1024 * 1024:
return None
file_hash = hashlib.md5(url.encode()).hexdigest()
ext = content_type.split("/")[-1].split(";")[0]
filepath = os.path.join(save_dir, f"{file_hash}.{ext}")
os.makedirs(save_dir, exist_ok=True)
with open(filepath, "wb") as f:
for chunk in resp.iter_content(chunk_size=8192):
f.write(chunk)
return filepath
except requests.exceptions.RequestException as e:
print(f"Error while accessing {url} through {proxy}: {e}")
return None
The key point is to check Content-Length before downloading the response body. This saves traffic by filtering out overly large files before they are fully downloaded. Proxy rotation using itertools.cycle evenly distributes the load among IPs, reducing the risk of banning a specific address.
Asynchronous Collection Using aiohttp
A synchronous script using requests is well-suited for a few thousand images, but for datasets of 50,000 files or more, asynchronicity is needed — otherwise, the collection will stretch over days. aiohttp along with a semaphore for limiting parallelism is a working option for tens of thousands of simultaneous tasks without overloading the proxy pool.
import asyncio
import aiohttp
import itertools
PROXIES = [
"http://user:[email protected]:10000",
"http://user:[email protected]:10000",
]
proxy_cycle = itertools.cycle(PROXIES)
sem = asyncio.Semaphore(50) # limit on parallel requests
async def fetch_image(session, url, idx):
proxy = next(proxy_cycle)
async with sem:
try:
async with session.get(url, proxy=proxy, timeout=15) as resp:
if resp.status != 200:
return
content_type = resp.headers.get("Content-Type", "")
if "image" not in content_type:
return
data = await resp.read()
with open(f"dataset/{idx}.jpg", "wb") as f:
f.write(data)
except Exception as e:
print(f"Error {url}: {e}")
async def main(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch_image(session, url, i) for i, url in enumerate(urls)]
await asyncio.gather(*tasks)
# asyncio.run(main(url_list))A semaphore for 50 simultaneous connections is a starting point. If residential proxies handle the load without an increase in 429 errors and timeouts, you can gradually increase it to 100–150. The main thing is to monitor the percentage of unsuccessful requests: if it rises above 5–7%, it is better to reduce parallelism rather than endlessly increase the proxy pool.
Scrapy + Proxy Middleware for Large-Scale Collection
For production dataset collection with a built-in queue, retries, and URL deduplication, it is more convenient to use Scrapy with ImagesPipeline. It automatically downloads images, checks their validity using PIL, and can generate previews of different sizes.
# settings.py
ITEM_PIPELINES = {
'scrapy.pipelines.images.ImagesPipeline': 1,
}
IMAGES_STORE = 'dataset_images'
IMAGES_MIN_HEIGHT = 200
IMAGES_MIN_WIDTH = 200
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.ProxyRotationMiddleware': 350,
}
CONCURRENT_REQUESTS = 32
DOWNLOAD_DELAY = 0.5
RETRY_TIMES = 3# middlewares.py
import random
class ProxyRotationMiddleware:
PROXIES = [
"http://user:[email protected]:10000",
"http://user:[email protected]:10000",
"http://user:[email protected]:10000",
]
def process_request(self, request, spider):
request.meta['proxy'] = random.choice(self.PROXIES)
The parameters IMAGES_MIN_HEIGHT and IMAGES_MIN_WIDTH automatically filter out small icons and sprites that clutter the dataset and waste traffic. A DOWNLOAD_DELAY of 0.5 seconds reduces the request frequency from a single proxy, which decreases the likelihood of captchas on sources with strict anti-fraud monitoring like Pinterest.
How to Reduce Traffic Consumption
Traffic through proxies is usually billed by volume, so any optimization directly saves the budget. Here are some effective techniques:
- Request previews instead of originals if a resolution of 512–1024 px is sufficient for training — most sources provide the required image size via a separate URL or query parameter.
- Filter by Content-Length before downloading the body — a HEAD request or reading headers from a streaming GET allows you to discard overly large files without consuming traffic to load them.
- Disable loading of auxiliary resources if the parser operates through a headless browser: block CSS, fonts, and tracking scripts, leaving only HTML and necessary images.
- Deduplicate URLs before the request, not after downloading — identical images on different product card pages (for example, on Wildberries) do not need to be downloaded again.
- Use a local cache of responses during repeated runs of the parser for debugging, so you don't consume traffic through proxies for each test run of the code.
Common Errors, 403, and Rate Limits
When collecting images in large volumes, the parser will inevitably encounter 403, 429 responses, and temporary redirects to verification pages. Proper handling of these codes saves both traffic and the proxy pool.
| Code/Symptom | Reason | What to Do |
|---|---|---|
| 403 Forbidden | IP is blacklisted by the source | Rotate proxies, change User-Agent |
| 429 Too Many Requests | Request limit exceeded per minute | Increase DOWNLOAD_DELAY, reduce parallelism |
| Redirect to captcha | Behavioral analysis suspected a bot | Switch to residential/mobile IPs |
| Empty Content-Length | Chunked response or placeholder instead of an image | Check the actual size of downloaded bytes |
It is also worth logging the success rate of requests for each proxy. If a specific IP consistently returns 403 more often than others — automatically remove it from rotation, without waiting for it to drag down the overall block statistics across the entire pool.
Calculating the Budget for Proxy Traffic
Before starting the collection, it is useful to estimate the volume of traffic in advance, rather than calculating it afterward. The formula is simple: number of images × average file weight × overhead factor (HTML, headers, repeated requests due to errors — usually 1.3–1.6).
An example calculation for a dataset of 200,000 medium resolution images (~150 KB) with product card parsing: 200,000 × 150 KB = 30 GB of pure image weight. Considering HTML pages and repeated requests due to temporary blocks (overhead factor of 1.5), the total consumption is about 45 GB. If some sources require mobile IPs due to anti-bot protection, factor this into the proxy pool separately, as traffic on mobile IPs is usually more expensive than on residential or datacenter proxies.
A good practice is to collect a pilot batch of 1,000–2,000 images, measure the actual traffic consumption and error rate, and then extrapolate to the entire planned dataset volume. This is more accurate than any theoretical calculations, as it takes into account the specifics of the particular source.
Conclusion
Collecting images for training computer vision models is a task at the intersection of data engineering and effective network infrastructure management. The correct architecture of the parser (asynchronicity, filtering by Content-Length, URL deduplication) saves traffic, while thoughtful proxy selection determines whether you will collect the required dataset volume in days or face weeks of constant blocks.
If data sources use advanced anti-bot protection — image search engines, marketplaces, social networks — start with residential proxies: they provide the best balance between resistance to bans and traffic cost. For open APIs and stocks without strict limits, it makes sense to save on datacenter proxies, and for the most protected mobile applications — connect mobile proxies for the final, most challenging part of the collection.