The question "how much does it cost to scrape a million pages" almost always receives a useless answer: "it depends on the website." It does depend — but not in a mystical way. The cost breaks down into four measurable components, each of which can be calculated in advance before the traffic bill arrives. Below is the working arithmetic: where gigabytes come from, why the price "per GB" says nothing about your budget, and which levers can realistically cut costs significantly.
The main metric — not the price per gigabyte, but the cost per successful request
Providers compete with numbers on display: $1/GB, $3/GB, $0.30 per thousand requests. These numbers are incomparable until you convert them to a single metric — the cost of one successful data extraction. This is what market reviews emphasize: raw figures "per GB" and "per 1000 requests" are not comparable without considering the page weight and the share of successful responses.
A formula sufficient for planning:
Budget = (number of target pages ÷ success rate) × average page weight × price per GB + fixed costs
Three variables in the first part and one in the second. Let's break each down.
Component 1: the weight of one page
This is the most underestimated variable. The range here is not percentages, but multiples, and it depends on how you retrieve the page:
- Regular HTTP request without rendering — 50–150 KB. You only get the HTML document.
- Full rendering in a headless browser — 100–300 KB with careful configuration.
- Browser "as is," without filters — 0.2–1 MB and above. You pay for images, fonts, ad scripts, analytics, and video players that you don't need.
The difference between the first and third points can be up to twenty times on the same page. For a million pages, this translates to a difference between 50 GB and 1000 GB of traffic.
Component 2: the price per gigabyte by proxy type
Market ranges relevant for 2026 look like this:
- Data center — $0.50–3 per GB or a monthly fee for IP.
- Residential — $1–8 per GB. Price quality benchmarks: around $1/GB is a great deal, $3–4/GB is market average, $5–8/GB is corporate segment.
- Mobile (4G/5G) — $2–15 per GB.
- ISP / static residential — $1.5–5 per IP per month.
- Ready-made scraping APIs — $0.30–12 per 1000 requests.
The temptation to go for the cheapest option is understandable, but it breaks down with the next component.
Component 3: success rate — the hidden budget multiplier
Every blocked request still costs you: traffic is spent, but no data is retrieved. Therefore, the actual number of requests equals the target number divided by the success rate, and the multiplier grows non-linearly:
- success 95% — overpayment 5%;
- success 70% — you will need about 400 thousand more requests for every million target records;
- success 60% — requests needed are 2.5 times more.
This is why a cheap data center at $0.50/GB on a secured site ends up costing more than residential proxies at $3/GB: Cloudflare, DataDome, and Akamai mark data center ASN by default, and you pay for the flow of refusals. As of July 2026, mobile proxies maintain a 90–95% success rate against modern anti-bot systems — where data centers fail completely.
The practical takeaway: the type of proxy is chosen not by price, but by purpose. Simple directories and open APIs — data center. Stores, social networks, aggregators with protection — residential. The most aggressive sites and mobile versions — mobile.
Three scenarios for a million pages
Let's put the numbers together. The condition is the same: a million useful pages.
Scenario A — "disciplined." HTTP client without a browser, 100 KB per page, residential at $2/GB, success 95%. Total: 1.05 million requests × 100 KB ≈ 105 GB ≈ $210.
Scenario B — "typical." Headless browser with resource filtering, 250 KB per page, residential at $3/GB, success 85%. Total: 1.18 million × 250 KB ≈ 295 GB ≈ $885.
Scenario C — "without configuration." The browser loads everything indiscriminately, 800 KB per page, residential at $5/GB, success 65%. Total: 1.54 million × 800 KB ≈ 1230 GB ≈ $6150.
The difference between A and C is almost thirtyfold, with the same output result. None of the variables were exotic: all of this is standard default settings versus ordinary diligence. For reference: industry estimates for a medium-sized company processing a million requests per month across various sites range from $1000 to $3000 per month — that is, between scenarios B and C.
What is not included in traffic but is included in the bill
Proxies are usually the largest, but not the only expense:
- Captcha. The solution costs $0.50–3 per thousand. With 5% of pages having captcha on a million, that's 50 thousand solutions — $25–150. Manageable. With 50% — it's already $250–1500, and the issue stops being technical.
- Infrastructure. $20–200 per month for modest volumes; $200–500 for small scale, and $3000–10,000 for truly large flows.
- Storage and outgoing traffic. About $0.023 per GB per month for object storage and $0.09 per GB for outgoing at major clouds — negligible at hundreds of gigabytes, significant at tens of terabytes.
- People. The most expensive line item that is often forgotten. A production-ready parser takes 8–12 weeks of development, then 4–8 hours per month on simple sites and 20+ hours on secured ones. At a rate of $75/hour, maintaining a secured source costs $1500+ per month — more than the total traffic in scenario B.
Five levers that can really cut costs
- Eliminate the browser where it's not needed. If the data is available in HTML or through an internal API — rendering is unnecessary. This is the main lever: reducing traffic by 60–80%.
- Block resources if the browser is necessary. Images, fonts, media, ad and analytics scripts can be disabled with a single request interception rule, yielding significant savings. A detailed breakdown of techniques is available in the article on optimizing traffic expenditure through proxies.
- Sort targets by complexity. Running the entire list through the most expensive channel is the most common budgeting mistake. Categorize sources into "open," "moderately protected," and "aggressive," and assign each a suitable type of proxy.
- Build a retry policy consciously. Blindly retrying three times on failure triples traffic on problematic targets. Only repeat those response codes where a retry makes sense, and with a different IP, not the same one.
- Cache and do not retrieve the same data twice. Lists, pagination, repeated passes through directories — this is where tens of percent of the budget can easily be lost.
When a ready-made API is cheaper, and when your own stack
You need to calculate the total cost of ownership, not just the line "per GB." Market benchmarks: a custom parser for 100 thousand pages per month costs $700–1650 including support, while a ready-made API for the same volume costs $50–300. At larger numbers, the picture flips: 1 million requests per month through an API costs $250–2490, while building your own infrastructure for 10 TB per month is estimated at about $230 thousand per year compared to $100–150 thousand for purchasing the service.
A rough rule: while the volume is small and there are many sites — opt for a ready-made solution, as you pay for someone else's support. When the volume is stable, there are few targets, and they are well understood — your own proxy stack is cheaper, as you stop paying a markup for rendering that you don't need. A detailed comparison of both approaches is available in the analysis of proxies versus scraping APIs and web unblockers.
What changes in the fall of 2026
There is a factor that will push calculations upward. Starting September 15, 2026, Cloudflare will default new domains to block categories Training and Agent — on pages with ads; the changes will affect new clients, new sites of existing clients, and all clients on the free plan. Search crawlers remain permitted. At the same time, Pay Per Crawl is turning into Pay Per Use: publishers will be compensated not for the fact of crawling a page, but for the use of content in AI responses.
The context is simple: according to Cloudflare, 52% of crawler requests in their network currently go toward training AI, compared to 22% in spring 2025. The network is closing, and the practical implication for budgeting is that the success rate on secured sources will generally decrease, meaning the hidden budget multiplier from the third component will become more significant. Factor this into your plan, rather than finding out through the bill.
In brief
The cost of parsing is not the price per gigabyte, but the product of page weight, channel price, and the inverse of the success rate. The difference between a careful and a careless pipeline on a million pages is measured not in percentages, but in orders of magnitude. Before negotiating a discount with a provider, calculate how much you are paying for loading images that you are not parsing, and for retries that should not have happened. Usually, there is more money lost there than in any discount.