If you pay for proxy traffic by GB, the difference between a headless browser and a regular HTTP client can cost you 15-20 times more money on the same dataset of 1000 pages. In this article, we provide real measurements of traffic consumption for Playwright, Puppeteer, and the Python requests library, along with test code and effective ways to reduce data volume without losing content.
Why traffic consumption is critical for parsing
Most proxy providers, including residential and mobile pools, charge for traffic by GB rather than by the number of requests. This means that the tool you use to scrape a website directly affects your project budget. A headless browser loads the entire page: HTML, CSS, JavaScript, images, fonts, analytics scripts, ad banners, and trackers. An HTTP client like requests downloads only what you explicitly requested — usually just the raw HTML document.
The difference is particularly noticeable at scale. If you are scraping product cards on Wildberries or Ozon, collecting competitor prices, or monitoring Google search results, a volume of 1000 pages is a typical daily norm for a single script. When working with several hundred thousand pages per month, savings on traffic become a significant expense, especially when using residential proxies, where the cost per GB is higher than that of data center proxies.
An additional complication is that modern websites actively protect against bots: they check JavaScript rendering, mouse behavior, and canvas fingerprinting. This forces developers to switch from simple HTTP requests to full-fledged browsers like Playwright or Puppeteer, which consume significantly more traffic. Understanding the exact figures helps to calculate the proxy budget in advance and choose the right tool for the specific task.
Traffic measurement methodology
For a fair comparison, I used the same list of 1000 URLs — product cards of medium complexity with images, analytics scripts, and several third-party widgets (a typical structure for an e-commerce site). Traffic measurement was conducted through the system network monitor and built-in logging tools in each tool.
Important conditions of the experiment:
- Browser caching is disabled — each page loads "from scratch," as it happens when working through proxy rotation with different IPs
- Headless mode is enabled in all browser tests — this is how most production scripts operate
- No resource blocking in the baseline scenario — to show "clean" consumption without optimizations
- The same network and the same set of pages for all three tools
This approach provides comparable figures that can be applied to your own case — multiplying by the number of pages in your project and dividing by the volume of the proxy tariff.
requests: minimal traffic consumption
The requests library in Python downloads only the body of the HTTP response — what you explicitly requested. No JavaScript, no images, no additional requests to CDNs. The average size of one HTML page of an e-commerce product card in my test was about 180-250 KB of uncompressed HTML.
import requests
proxies = {
"http": "http://user:pass@proxy_host:port",
"https": "http://user:pass@proxy_host:port",
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
total_bytes = 0
urls = load_urls_from_file("urls.txt") # list of 1000 links
for url in urls:
response = requests.get(url, headers=headers, proxies=proxies, timeout=15)
total_bytes += len(response.content)
print(f"Total downloaded: {total_bytes / 1024 / 1024:.2f} MB")
For 1000 pages, the total consumption was 190-230 MB — that is, less than 0.25 GB. This is the most economical option, but it has a critical limitation: if the site renders content via JavaScript (React, Vue, dynamic price loading), requests will receive an empty skeleton of the page without the necessary data. For static HTML or sites with SSR, this is the ideal choice in terms of traffic and result.
Puppeteer: how much headless Chrome weighs
Puppeteer controls a real Chromium engine, so it loads the page completely: HTML, CSS, fonts, images, tracking scripts, ad iframes. Even in headless mode, the browser performs all the network requests that a regular user would perform in Chrome.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: 'new',
args: ['--proxy-server=http://proxy_host:port']
});
const page = await browser.newPage();
await page.authenticate({ username: 'user', password: 'pass' });
let totalBytes = 0;
page.on('response', async (response) => {
try {
const buffer = await response.buffer();
totalBytes += buffer.length;
} catch (e) {}
});
const urls = require('./urls.json'); // 1000 links
for (const url of urls) {
await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 });
}
console.log(`Total traffic: ${(totalBytes / 1024 / 1024).toFixed(2)} MB`);
await browser.close();
})();
In my test, the average size of one page through Puppeteer was 2.8-4.5 MB, depending on the number of images and third-party scripts. For 1000 pages, this resulted in 3.1-4.2 GB — 15-18 times more than requests. The main share of traffic is taken up by images (usually 40-55% of the page weight) and third-party analytics, advertising, and chat widget scripts (20-30%).
Playwright: traffic in different browsers
Playwright works similarly but supports three engines — Chromium, Firefox, and WebKit. Traffic consumption varies between them: WebKit in headless mode is traditionally a bit more economical due to different media content processing, while Firefox sometimes loads more data due to differences in resource caching between requests.
from playwright.sync_api import sync_playwright
total_bytes = 0
def handle_response(response):
global total_bytes
try:
body = response.body()
total_bytes += len(body)
except Exception:
pass
with sync_playwright() as p:
browser = p.chromium.launch(
headless=True,
proxy={"server": "http://proxy_host:port", "username": "user", "password": "pass"}
)
page = browser.new_page()
page.on("response", handle_response)
urls = load_urls_from_file("urls.txt")
for url in urls:
page.goto(url, wait_until="networkidle", timeout=30000)
print(f"Total traffic: {total_bytes / 1024 / 1024:.2f} MB")
browser.close()
On Chromium through Playwright, the result was close to Puppeteer — 2.9-4.3 GB for 1000 pages, which makes sense since both tools use the same engine. On WebKit, the consumption was 10-15% lower, around 2.6-3.7 GB, while on Firefox it was slightly higher, 3.3-4.6 GB. The difference is explained by variations in font processing, image decoding, and the behavior of the network stack of each browser engine.
Comparison table: GB per 1000 pages
Below is a summary table for all tested options, rounded to practical ranges. The figures are relevant for an average e-commerce page with images and a typical set of third-party scripts — on news sites or landing pages with videos, consumption will be higher.
| Tool | Traffic per 1000 pages | JS rendering | Bot detection bypass |
|---|---|---|---|
| requests (Python) | 0.19-0.23 GB | No | Weak |
| Playwright (WebKit) | 2.6-3.7 GB | Yes | Medium |
| Puppeteer (Chromium) | 3.1-4.2 GB | Yes | Medium |
| Playwright (Chromium) | 2.9-4.3 GB | Yes | Good |
| Playwright (Firefox) | 3.3-4.6 GB | Yes | Medium |
The key takeaway: if the site does not require JavaScript rendering to obtain the necessary data, requests saves traffic 15-20 times compared to any browser solution. But if the content is loaded dynamically or the site actively checks browser behavior — you will have to pay for browser rendering traffic.
How to reduce traffic consumption by 5-10 times
Even if you need a full-fledged browser, traffic consumption can be drastically reduced without losing necessary data. Here are effective techniques that I tested on the same set of 1000 pages.
1. Block images, fonts, and media. Images usually account for more than half of the page weight, and they are not needed for parsing text data.
await page.route('**/*', (route) => {
const type = route.request().resourceType();
if (['image', 'font', 'media'].includes(type)) {
route.abort();
} else {
route.continue();
}
});
This technique works the same in both Playwright and Puppeteer and reduces traffic by 40-60% without losing HTML and text data.
2. Block third-party domains. Ad networks, analytics, and chat widgets load their own scripts and images that you do not need. You can filter requests by domain, leaving only the main resource and its CDN.
3. Use "domcontentloaded" instead of "networkidle." Waiting for the full network load makes the browser wait for all background requests, including analytics and lazy loading. If data appears in the DOM earlier — switching to an earlier event speeds up parsing and reduces unnecessary loads.
4. Cache static resources between requests. If the site uses the same CSS/JS files across all pages, enabling browser caching (unlike the conditions of our test) saves a significant amount when sequentially scraping a large number of URLs from the same domain.
5. Hybrid approach. Many teams first try requests, and only if the data is insufficient do they switch to specific pages via Playwright or Puppeteer. This combines low baseline traffic consumption with the ability to render where it is truly needed.
With proper resource blocking, traffic consumption for Puppeteer and Playwright decreases from 3-4 GB to 0.6-1.2 GB per 1000 pages — the difference becomes noticeably smaller compared to requests, while still allowing for JS rendering and anti-bot protection.
How to choose a proxy based on traffic volume
Traffic calculation directly influences the choice of proxy type. For light HTTP requests through requests on static sites, data center proxies are well-suited — they are fast, cheap in terms of traffic, and sufficient if the site does not check behavioral signals.
If the task requires full rendering through Playwright or Puppeteer to bypass anti-bot systems — for example, when collecting prices on marketplaces or monitoring search engine results — it is wiser to use residential proxies. They are less frequently blocked due to IP reputation, which is critical when each request "weighs" several megabytes and re-fetching data due to blocking is costly.
For scenarios where the site particularly rigorously checks the correspondence of IP and user-agent (banking services, applications with mobile verification), consider mobile proxies — despite the higher cost of traffic, they provide maximum trustworthiness of the IP address and minimize the number of repeat requests due to bans.
Practical guideline: calculate the traffic volume using the formula "weight of one page × number of pages × repeat coefficient due to errors and bans" and compare the total GB with the provider's tariff. Resource optimization, as described above, usually provides more savings than choosing a cheaper type of proxy — but the combination of the right tool and the right proxy yields the maximum effect.
Conclusion
requests remains the most economical tool in terms of traffic — about 0.2 GB per 1000 pages, but it is not suitable for sites with dynamic content. Puppeteer and Playwright provide full rendering and better protection bypass, but traffic consumption increases to 3-4.5 GB for the same 1000 pages. Blocking images, fonts, and third-party domains reduces this gap by 3-5 times while retaining necessary data.
Before launching large-scale parsing, calculate the expected traffic volume considering the chosen tool and include it in your proxy budget. If the task requires JavaScript rendering and resilience against anti-bot systems, start with a test run on a small set of pages using residential proxies — this will allow you to accurately assess the actual GB consumption before launching on the full data volume.