How to Reduce Parser Traffic by 5 Times Without Data Loss: 7 Techniques for Data Engineers

Every extra megabyte of parser traffic means either paying a proxy provider or risking hitting limits and getting an IP ban. If you are collecting prices from Wildberries, Ozon, or monitoring ads on Avito through hundreds of proxy addresses, saving traffic directly affects the project's budget. In this article, we present specific technical techniques that allow you to reduce the volume of transmitted data by 4-5 times while maintaining the completeness and accuracy of the extracted information.

Why Parser Traffic Affects the Budget

Most proxy providers charge for residential and mobile proxies based on the volume of gigabytes transmitted, not on the time of use. If your parser loads the entire product page from Wildberries — with images, recommendation scripts, analytics trackers, and fonts — you pay for 2-3 MB per card, while you actually only need 15-20 KB of text: name, price, rating, availability.

When scaling up to 50,000-100,000 cards per day, the difference between "load everything" and "load only what is needed" turns into tens of gigabytes of excess traffic daily. This not only incurs costs for proxies but also increases the load on the target site, raising the chance of triggering anti-bot protection and receiving CAPTCHAs or temporary IP bans. Optimizing traffic is both a way to save money and reduce the risk of blocks.

There is a third effect: the less data transmitted per request, the faster the request itself is executed. This allows for increased parallelism — running more threads on the same number of proxies without exceeding the speed limits set by anti-detect browsers like Dolphin Anty or AdsPower when working with sessions.

Technique 1: Blocking Images, CSS, and Fonts

If the parser operates through a headless browser (Playwright, Puppeteer, Selenium) — the fastest way to reduce traffic by 2-3 times is to block the loading of static resources that do not affect the data in the DOM. Product images, website fonts, videos, and CSS styles account for up to 70% of the page weight, yet they do not participate in text and attribute extraction.

from playwright.sync_api import sync_playwright

def block_heavy_resources(route, request):
    if request.resource_type in ["image", "media", "font", "stylesheet"]:
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.route("**/*", block_heavy_resources)
    page.goto("https://example.com/product/123")
    html = page.content()
    browser.close()

Similar logic is implemented in Puppeteer through page.setRequestInterception(true) and in Selenium through Chrome profile settings with the parameter profile.managed_default_content_settings.images: 2. In practice, this single setting immediately cuts 50% to 70% of traffic when parsing marketplaces where pages are overloaded with visual content and advertising banners.

Technique 2: HTTP Requests Instead of a Full Browser

Many use Selenium or Playwright where it is not necessary. If the page does not require JavaScript execution to render data (this can be easily checked by opening "View Page Source" instead of DevTools), it is much more profitable to fetch HTML directly using libraries like requests or httpx in Python. Such a request weighs in kilobytes, not megabytes, because it does not carry the overhead of the browser's rendering engine, network calls for trackers, and secondary resources.

import httpx

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept-Encoding": "gzip, br",
    "Accept": "text/html,application/xhtml+xml"
}

proxies = {"http://": "http://user:pass@proxy_host:port",
           "https://": "http://user:pass@proxy_host:port"}

with httpx.Client(headers=headers, proxies=proxies, http2=True) as client:
    response = client.get("https://example.com/catalog/item/456")
    print(len(response.content), "bytes received")

Switching from browser emulation to direct HTTP requests where the site delivers ready HTML without client-side rendering reduces traffic by 3-8 times. The only caveat is that such requests are easier to distinguish from a real user, so for sites with strict anti-bot protection, it is advisable to combine this method with high-quality residential proxies that provide IPs from real home providers and reduce the likelihood of request blocking.

Technique 3: Parsing Through Hidden JSON APIs

Almost all modern marketplaces — including Wildberries, Ozon, and Yandex.Market — render product cards and lists through internal JSON APIs called by the frontend. You can find these endpoints through the Network tab in DevTools by filtering requests by type XHR/Fetch. Typically, one such request returns JSON of 5-30 KB with clean data: product ID, price, discount, stock, rating — without a single byte of HTML markup or CSS.

The difference in the volume of transmitted data between a full HTML page and a direct call to a JSON API can reach 10-15 times. An additional advantage is that JSON is easier to parse programmatically: XPath selectors are not needed; you just need to access the required field by the dictionary key. The downside is that such endpoints often require specific headers, session tokens, or request signature parameters that need to be extracted from the main page or mobile application beforehand.

Technique 4: Gzip and Brotli Compression

Even if you have to fetch the full HTML, enabling the right compression can reduce the transmission size by 60-80%. Many custom parsers do not send the Accept-Encoding: gzip, br header, causing the server to return an uncompressed response. The requests and httpx libraries automatically unpack Gzip and Brotli — it is only important to explicitly indicate support for compression in the request headers.

Brotli generally compresses text HTML more effectively than Gzip, by 15-20%, but not all servers support this algorithm — it is advisable to request both options and allow the server to choose the optimal one. For JSON APIs, the effect of compression is even more noticeable: repeating dictionary keys ("price", "name", "rating") compress almost perfectly, significantly reducing the response size.

Technique 5: Conditional Requests and Caching

If you are monitoring prices on the same products several times a day, most cards do not change between checks. Use the If-Modified-Since and If-None-Match headers with the ETag value obtained during the first request. If the content has not changed, the server returns a status of 304 Not Modified with almost no response body — traffic savings can reach up to 95% on unchanged pages.

import httpx

etag_store = {}

def fetch_with_cache(url, client):
    headers = {}
    if url in etag_store:
        headers["If-None-Match"] = etag_store[url]
    resp = client.get(url, headers=headers)
    if resp.status_code == 304:
        return None  # data has not changed
    etag_store[url] = resp.headers.get("ETag", "")
    return resp.content

Not all sites support ETag correctly, but for those that do, this technique becomes the most effective way to reduce traffic during regular monitoring — you effectively only pay for actual data changes, not for reloading unchanged content.

Technique 6: Selective Parsing of Required Fields

Sometimes it is impossible to reduce incoming traffic from the server — the site delivers the entire page regardless of the request. In this case, optimization occurs at the processing stage: do not reload the page just to extract one more field. Design your XPath or CSS selectors to extract all necessary attributes — price, name, article, availability, rating — in one pass through the DOM instead of making repeated requests to the same URL with different parsers for different tasks.

It is also useful to limit the depth of crawling nested pages. If monitoring prices only requires data from the category page (product list), do not navigate to each product card separately — this duplicates traffic, often yielding no new information besides descriptions and reviews that do not affect price and availability.

Technique 7: Optimizing the Crawl Pattern

URL deduplication is a basic but often ignored technique. Marketplace catalogs generate many links with the same content but different sorting parameters, UTM tags, or session IDs. Normalizing URLs before queuing (removing tracking parameters, sorting query parameters) eliminates 10-30% of redundant requests when crawling large catalogs.

Prioritizing crawling based on the frequency of data changes also saves traffic: high-demand products with volatile prices should be checked every hour, while rare items can be checked once a day. This adaptive schedule, as opposed to uniformly crawling all cards at the same frequency, reduces the overall number of requests by 2-4 times without losing the relevance of critical data.

How This Relates to Proxy Strategy

Reducing traffic directly affects the choice of proxy type. If you are parsing a large volume of pages with direct HTTP requests without complex anti-bot protection, fast and cheap data center proxies are sufficient — they provide high transfer speeds at low cost per gigabyte, which is critical when scaling to thousands of cards per day.

For sites with strict bot protection, where it is important to imitate real user behavior, it is better to use residential proxies — combining them with techniques to block unnecessary resources gives you both low traffic and a high level of trust from the site for the request. If parsing is conducted through mobile versions of marketplace APIs, where data is more compact and the anti-bot system is oriented towards mobile IP ranges, it is worth considering mobile proxies for additional risk reduction of blocks.

The combination of "minimal traffic per request" + "correct type of proxy for the task" allows you to simultaneously reduce infrastructure costs and increase data collection speed without losing reliability.

Comparison Table of Techniques

Technique	Traffic Reduction	Implementation Difficulty
Blocking Images/CSS/Fonts	50-70%	Low
HTTP Requests Instead of Browser	3-8 times	Medium
Hidden JSON APIs	10-15 times	High
Gzip/Brotli Compression	60-80%	Low
Conditional Requests (ETag)	Up to 95% on unchanged pages	Medium
URL Deduplication and Prioritization	2-4 times	Medium

Implementation Checklist

• Check if the target page requires JavaScript rendering, or if HTML can be fetched directly via httpx/requests
• Set up blocking of image/media/font/stylesheet in the headless browser if a browser is still needed
• Find internal JSON APIs through DevTools → Network → XHR/Fetch
• Add Accept-Encoding: gzip, br headers to all requests
• Implement ETag/Last-Modified storage for conditional requests on repeating URLs
• Normalize and deduplicate the URL queue before crawling
• Set up an adaptive crawling frequency based on the importance and volatility of data
• Select the type of proxy based on the final traffic profile — data center, residential, or mobile

Conclusion

Reducing parser traffic by 5 times is a realistic goal if techniques are applied sequentially: eliminate unnecessary resources, switch to direct HTTP requests or JSON APIs where possible, enable compression, use conditional requests for unchanged data, and optimize the crawl pattern itself. Each of these steps provides measurable effects, and together they fundamentally change the economics of data collection from marketplaces and other sites.

After optimizing traffic, it is important to correctly select the proxy infrastructure for the new load profile. For fast and cheap collection of large volumes of data, data center proxies are suitable, while for working with sites with strict anti-bot protection, residential proxies with real IP addresses from home providers reduce the risk of blocking even during intensive parsing.