Launched a parser on Wildberries or Ozon, set up proxies, and after an hour hit limits and got a bunch of banned IPs — even though you only collected a couple of thousand product cards? The problem is almost always the same: URL parameters. Filters, sorting, pagination, and UTM tags turn one catalog into tens of thousands of technical duplicates, and the parser diligently goes through them all instead of collecting unique data.
What are page duplicates due to URL parameters
A page duplicate is a URL that is technically different from another but delivers the same or nearly the same content. A classic example: a dress catalog on Wildberries is available at /catalog/women/clothing/dresses, but the same output, sorted by price, becomes /catalog/women/clothing/dresses?sort=price. Add a filter by size, brand, color, price range — and you end up with hundreds of parameter combinations, each of which is a formally unique URL.
For a live user, this is convenient: select a filter — see the desired product. But for a parser that crawls links by pattern or follows pagination, each such combination is a separate page that needs to be loaded, parsed, and saved. If rules are not set, the bot will "fall" into an endless funnel of filters: sorting by price → filter by brand → filter by size → sorting again, but in a different order. The number of unique URLs grows exponentially, while the useful data in them does not increase — these are the same product cards, just in a different order or with a reduced set.
A separate category of duplicates is UTM tags and session parameters (utm_source, clid, _openstat). They do not affect the content of the page at all, but if the parser collects links from other pages of the site as they are, it will pull the same product card dozens of times with different tails in the address.
Why this is critical for the parser
For an SEO specialist, page duplicates are an indexing issue and dilution of weight. For someone parsing a marketplace for price monitoring or assortment collection, duplicates are a direct loss of money and time, impacting three areas simultaneously.
First, each loading of a duplicate page is a request through a proxy. If you are using paid rotating proxies (and to bypass the protection of Wildberries and Ozon, you can't do without them), you are paying for traffic even for useless requests. At a scale of tens of thousands of product cards, an extra 30-40% of requests for duplicates significantly increases costs.
Second, the speed of collection drops significantly. If your parser can physically make 10 requests per second, and a third of them go to pages with the same data, the actual speed of collecting unique content drops to 6-7 cards per second. On large catalogs, this is the difference between collecting in an hour and half a day.
Third, and this is the most painful — the risk of IP bans increases proportionally to the number of requests. Anti-bot systems of marketplaces track the frequency of requests from a single address. If you make three times more requests than necessary to collect target data, you deplete the "trust credit" of the proxy three times faster and end up facing CAPTCHA or blocking.
Real examples: Wildberries, Ozon, Avito
On Wildberries, the same catalog can be served with parameters ?sort=popular, ?sort=rate, ?sort=newly, plus filters fbrand, fsize, priceU. If the parser collects links "card → similar products → again card", it easily generates URLs with combinations of all these parameters, even though the set of products overlaps by 90%.
On Ozon, a similar story with the parameter ?sorting= and filters by product characteristics — there are even more combinations because each category has its own set of filters (material, volume, color, etc.).
On Avito, a separate pain point is geolocation parameters and search radius (?radius=, ?district=). The same ad can "pop up" in the output with different combinations of these parameters, and a naive parser will consider this different findings, inflating the database with duplicate records.
How duplicates consume the proxy budget and time
Let's calculate with a hypothetical example. Suppose a category has 500 unique products, but due to 8 sorting options and 12 combinations of popular filters, the parser without URL normalization crawls not 500, but 4000-5000 pages with listings plus repeated visits to the same cards. If one request through a residential proxy costs a conditional unit of traffic, then with naive crawling, you spend 8-10 times more than necessary to obtain the same volume of unique data.
Worse, repeated requests from the same IP to similar URLs are a characteristic pattern for anti-bot systems. The higher the frequency and uniformity of requests, the faster the protection kicks in. This creates a vicious circle: duplicates increase the number of requests → load on the proxy pool increases → IPs get banned faster → you have to change proxies more often and restart collection → the total time and cost of parsing increase even more.
In practice, this looks like this: a seller wants to monitor prices of 200 competitors on Wildberries once a day, but due to uncleaned URLs, their script makes 15,000 requests instead of 2,000, hits the limits of the proxy pool, and gets incomplete data by evening instead of morning.
How to detect duplicates before launching the parser
Before scaling the collection, it is worth spending an hour analyzing the URL structure of the target site. Here is a practical order of actions:
- Open the target catalog in a browser and manually go through the filters and sorting — note which parameters appear in the address bar.
- Check if the page content changes when adding a parameter that does not affect the output (for example, a UTM tag) — if the content is the same, this parameter should be ignored during collection.
- Look at
rel="canonical"in the page source code — sites often indicate which URL is considered the "main" for duplicate options. - Collect a small test sample of 100-200 URLs and run it through simple deduplication by key parameters — see how much the list shrinks.
- Check robots.txt — sometimes there are already Disallow directives for parameterized URLs, which is also a hint about which parameters are considered "junk."
URL normalization: rules and examples
Normalization is the process of bringing URLs to a single canonical form before deciding whether to crawl them. The basic principle: isolate "significant" parameters (for example, the page number of pagination or category) and discard "insignificant" ones (sorting, UTM, session identifiers).
A practical example of a normalization rule for a Wildberries-like catalog:
import urllib.parse as up
IGNORED_PARAMS = {"sort", "utm_source", "utm_medium", "utm_campaign", "clid", "_openstat"}
def normalize_url(url: str) -> str:
parsed = up.urlparse(url)
query = up.parse_qsl(parsed.query)
# keep only significant parameters, discard the rest
filtered = [(k, v) for k, v in query if k not in IGNORED_PARAMS]
filtered.sort() # standardize the order of parameters
new_query = up.urlencode(filtered)
return up.urlunparse(parsed._replace(query=new_query))
After normalization, all variations of the same listing with different sorting collapse into one URL. Before sending a request through the proxy, the parser checks if this normalized address has been encountered before — if so, it skips it. This is the cheapest way to reduce the number of requests without losing data completeness.
For simple cases, this same logic can be implemented without code — through the settings of the parsing tool: specify "ignore parameters" and list the parameters, or set a regular expression to trim the URL tail after the question mark if the parameters do not carry significant information for your task.
Configuring the parser to ignore parameters
Most ready-made solutions for data collection from marketplaces and parsing builders support rules for excluding URL parameters at the project settings level. Here’s what to check and configure first:
- List of ignored parameters. Add all found during the analysis of sorting, UTM tags, and session identifiers.
- Deduplication by hash of the normalized URL. Many parsers keep a list of already visited addresses — it is important that the comparison is done against the normalized version, not the original string.
- Limiting the depth of filter crawling. If the task is to collect the assortment of a category, one "basic" pass without sorting and filters is enough, rather than a recursive crawl of all combinations.
- Prioritizing pagination over filters. If the site provides a complete list of products through pagination, this is usually sufficient — separately crawling the same list with applied filters is redundant.
After such a configuration, the volume of requests usually decreases by 3-6 times depending on how "parameterized" the catalog of the source site is.
The role of proxies when scaling parsing
Even after ideal URL normalization, parsing large catalogs on Wildberries or Ozon requires distributing requests across multiple IP addresses — otherwise, anti-bot protection will quickly track abnormal activity from a single source. It is important to choose the right type of proxy for the task.
For large-scale collection of product cards and price monitoring, where volume and speed at a reasonable cost are important, data center proxies are usually sufficient — they are faster and cheaper than residential ones, which is critical when making millions of requests per day. However, if the site's protection aggressively bans data center IP ranges, it is worth switching to residential proxies, which appear as regular users from real providers and are less likely to be flagged.
For specific tasks — for example, checking how a specific product's price is displayed from a mobile device in a specific region — mobile proxies will be useful, which emulate a connection through a cellular operator and are almost never blocked en masse.
It is important to understand: the correct choice of proxy does not replace URL normalization but complements it. Without cleaning up duplicates, you will just burn through even the highest quality pool of IP addresses faster. First, reduce the number of useless requests, then scale the remaining ones through proxy rotation — this order provides the maximum savings.
Checklist before launching data collection
Before launching the parser at full scale, go through a short checklist:
| Check Item | Why it's needed |
|---|---|
| List of sorting and filter parameters identified | To understand which URLs can be considered duplicates |
| URL normalization rule configured | To collapse duplicates into one canonical address |
| Deduplication by hash enabled | To exclude reloading the same data |
| Proxy type selected based on volume and aggressiveness of protection | To reduce the risk of bans and speed up collection |
| Request limits per IP set with a margin | To avoid mass banning of the proxy pool |
Conclusion
Filters, sorting, and service parameters in URLs are not just a technical detail, but one of the main sources of useless traffic when parsing marketplaces. Without URL normalization, the parser can make 5-10 times more requests than necessary to obtain the same data, which directly impacts the budget for proxies, the speed of collection, and the risk of blocks.
The solution is simple in formulation but requires discipline at the preparation stage: analyze the URL structure of the target site, determine which parameters affect content and which do not, configure deduplication by normalized address, and only after that scale collection through proxy rotation. If your task is stable price and assortment monitoring on Wildberries, Ozon, or Avito in large volumes, it is worth starting with residential proxies — they provide the best balance between trust from anti-bot systems and flexibility in settings for specific tasks.