Most beginners in parsing start with a full crawl of the site: they go to the homepage, collect all the links, navigate through each one, collect links again — and so on thousands of times. This consumes proxies, time, and often ends with a ban. There is a simpler and cheaper way: almost every major site, including Wildberries, Ozon, and Avito, already has a ready-made map of all pages — sitemap.xml. Let’s figure out how to find, read, and use it instead of thousands of unnecessary requests.
What is sitemap.xml and why it matters to parsers
Sitemap.xml is a file that the site creates for search engines like Google and Yandex. It lists all the pages that the site owner wants to be indexed: product cards, categories, blog articles. Essentially, it is a ready-made list of URLs that the site provides to you — there’s no need to navigate through links manually, click through pagination, or decipher the navigation logic.
For sellers monitoring competitor prices on Wildberries or Ozon, this means one thing: instead of crawling through thousands of categories and pagination pages, you can get a list of all product cards in one request to the sitemap. For marketers collecting a base of URLs for competitor content analysis, this saves hours of manual work. For arbitrageurs parsing competitor landing pages for creative analysis, the sitemap shows all pages of the site at once.
An important nuance: not all pages make it into the sitemap. Sometimes pages with filters, temporary promotions, or pages with parameters in the URL are excluded. Therefore, the sitemap does not always provide 100% coverage of the site, but usually covers 80-95% of the useful pages you need for cataloging or monitoring.
Full crawl vs sitemap-first: cost comparison
The difference in approaches is significant when considering not in abstract "attempts," but in real resources: the number of requests, proxy traffic consumption, and time to results. Below is a comparison based on collecting a catalog of 50,000 product cards.
| Parameter | Full site crawl | Parsing through sitemap |
|---|---|---|
| Requests to collect URLs | 15,000–20,000 (categories, pagination, filters) | 50–300 (only sitemap files) |
| Time to collect URL list | 4–8 hours | 15–30 minutes |
| Risk of IP blocking | High due to the volume of navigation requests | Low, requests are significantly fewer |
| Load on proxy traffic | High, a large pool of IPs is needed | Minimal, can manage with a small pool |
| Complexity of parser logic | High, navigation needs to be parsed | Low, XML structure parsing |
The figure "10 times cheaper" from the title is not an exaggeration. If a full crawl requires 15-20 thousand requests just to compile a list of pages, and the sitemap provides the same list with 50-300 requests, the savings on proxy traffic and time are indeed substantial. At the same time, you will still have to download the product cards separately — but you will know exactly which URLs to download and won’t waste resources on empty pages, duplicates, and junk links.
How to find sitemap.xml on any site
The first and simplest way is to add /sitemap.xml to the domain of the site. For example, to check, try opening a browser with an address like https://example.com/sitemap.xml. This will work for most sites, including many online stores on popular CMS.
If nothing is found at this address, check the robots.txt file — it is almost always located at /robots.txt and usually contains a line like Sitemap: https://example.com/sitemap_index.xml. This is the official way that sites inform search engines where to find the map. For large marketplaces like Wildberries and Ozon, the sitemaps are often split into dozens of files and specified through robots.txt.
The third option is to look for mentions in systems like Google Search Console (if you have access to your site) or use specialized site scanners that find all sitemap files themselves. For competitor sites, the first two methods are usually sufficient — robots.txt is almost never hidden from the public eye; it is a public file.
Tip:
Sitemap file names are not always standard. Variants like sitemap_index.xml, sitemap-products.xml, product-sitemap.xml, and similar can be encountered. Always start with robots.txt — it will indicate the exact paths.
Sitemap structure for marketplaces: index and nested maps
Small sites usually have one sitemap.xml file with all URLs. But large catalogs, such as Wildberries, Ozon, or Avito, have millions of pages, and one XML file of such volume would be inconvenient. Therefore, they use what is called a sitemap index — a summary file that links to dozens or hundreds of separate sitemap files, divided by categories, dates, or simply in batches of 10-50 thousand URLs.
The structure looks something like this: first, you open the main index file, where you find a list of links to child sitemaps, such as sitemap-products-1.xml, sitemap-products-2.xml, and so on. Then you need to download each of these files and extract the final URLs of the product cards from them. This is a two-level process, but it is still an order of magnitude faster than manually navigating the site.
Sometimes child sitemap files are compressed in gzip format (with the .xml.gz extension) — this is standard practice to save space on the server. If the script downloads a file and sees unreadable characters instead of XML, it likely needs to decompress the content before parsing.
Step-by-step extraction of URLs from sitemap
For those working with ready-made no-code parsing tools, most modern marketplace catalog parsers can already read sitemaps automatically — just specify the root URL of the site, and the tool will find robots.txt, locate the sitemap index, and collect a list of all cards. But if you are working with a developer or writing a small script yourself, the logic is simple, and here’s an example in Python:
import requests
import gzip
from io import BytesIO
from lxml import etree
def get_sitemap_urls(sitemap_url, proxies=None):
response = requests.get(sitemap_url, proxies=proxies, timeout=15)
content = response.content
# Decompress gzip if the file is compressed
if sitemap_url.endswith('.gz'):
content = gzip.decompress(content)
root = etree.fromstring(content)
namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
# If this is a sitemap index — collect links to child maps
sub_sitemaps = root.findall('.//ns:sitemap/ns:loc', namespace)
if sub_sitemaps:
all_urls = []
for sub in sub_sitemaps:
all_urls.extend(get_sitemap_urls(sub.text, proxies))
return all_urls
# If this is a regular sitemap — collect final URLs
urls = root.findall('.//ns:url/ns:loc', namespace)
return [url.text for url in urls]
proxy = {
"http": "http://user:pass@proxy_host:port",
"https": "http://user:pass@proxy_host:port"
}
urls = get_sitemap_urls("https://example.com/sitemap_index.xml", proxies=proxy)
print(f"Found URLs: {len(urls)}")This script recursively goes through the index file, downloads each child map, and collects all final links into one list. In the end, you get a complete list of product card URLs without a single unnecessary navigation request.
Filtering necessary pages by URL patterns
After you have obtained the complete list of URLs from the sitemap, you almost always need to filter it. In one sitemap file of a marketplace, product cards, category pages, blog articles, brand pages, and service pages may coexist. You will likely only need product cards of a specific category.
URL patterns are usually used for filtering. For example, if product cards on the site look like /catalog/product/12345, and categories look like /catalog/category/electronics, it’s easy to separate one from the other by simply comparing path segments. For more complex cases, where the URL structure is not so obvious, regular expressions are used — they allow you to set a pattern once and instantly filter hundreds of thousands of links.
Additionally, it is useful to look at the <lastmod> tags inside the sitemap — they show the date of the last modification of the page. If you only need recently updated products (for example, for monitoring new competitor positions), you can filter URLs based on this date and ignore pages that haven’t changed for months.
Proxy setup for mass downloading of sitemaps
Although the number of requests to the sitemap is significantly lower than with a full crawl, sites like Wildberries and Ozon still monitor suspicious activity — too frequent requests from one IP, atypical User-Agent, lack of browser headers. Therefore, even for downloading sitemap files, it is advisable to use proxies, especially if you plan to conduct regular monitoring rather than a one-time catalog collection.
For downloading the XML sitemap files, data center proxies are usually sufficient — they are fast and inexpensive, and since there are fewer requests to the sitemap, sites apply strict protection less often compared to regular catalog pages. However, at the stage of downloading product cards by the collected URLs, where the number of requests is an order of magnitude higher and the protection is stricter, it is better to switch to residential proxies — they use real IPs of regular users and are less likely to be blocked by anti-bot systems of marketplaces.
If you are monitoring the catalog from mobile versions of the site or an API that only provides data with mobile traffic, consider using mobile proxies — they use IPs from mobile operators and are considered the most trusted by most anti-fraud systems.
The practical setup in the script or parser is simple: specify the host, port, username, and password of the proxy in the respective fields of the request (as shown in the code example above in the proxies parameter), or if you are using a ready-made no-code parser or an anti-detect browser like Dolphin Anty or AdsPower for manual checking of the site structure before setting up automatic collection, insert the proxy data in the profile settings: open the "Proxies" tab → select the HTTP or SOCKS5 type → insert the host, port, username, password → save the profile.
Common mistakes when working with sitemap
The first and most common mistake is to ignore robots.txt and try to guess the sitemap address at random. This is a waste of time that can be easily avoided by simply checking the indexing rules file.
The second mistake is not considering that the sitemap may be incomplete. Some sites deliberately exclude low-value pages for SEO from the sitemap, but which are still important for price monitoring — for example, out-of-stock product pages. If the number of collected cards after parsing through the sitemap is significantly lower than expected, it’s worth checking the number of products that the site shows in the category, and if necessary, supplement the collection with targeted crawling of the missing sections.
The third mistake is not caching already downloaded sitemap files. If you are running monitoring daily, there’s no point in downloading the index file and all child maps again each time — most of them do not change every day. Save the list of URLs locally and only update what has actually changed based on the lastmod date.
The fourth mistake is forgetting about request headers. Even downloading a simple XML file can be blocked if the request looks like it’s from a bot: without User-Agent, without Accept-Language, with atypical default headers set by the request library. Adding realistic browser headers along with proxies significantly reduces the likelihood of blocking at this stage.
Checklist for starting parsing through sitemap
Before launching a full catalog collection, go through a short list of steps to avoid wasting time on mistakes:
- Checked the site's robots.txt and found the exact address of the sitemap or sitemap index
- Ensured that the sitemap structure is two-level (index + child maps) or one-level
- Checked if the child files are compressed in gzip
- Defined the URL pattern for the necessary pages (product cards, categories, etc.)
- Set up filtering by pattern or by lastmod date if only fresh positions are needed
- Connected proxies for downloading sitemap files and separately for downloading the product cards
- Added realistic request headers (User-Agent, Accept-Language)
- Configured caching and date comparison for repeated monitoring runs
- Compared the final number of URLs with the approximate number of products on the site
Conclusion
The sitemap-first approach is not a hack or a way to bypass protection, but rather the use of information that the site itself publishes for search engines. Instead of thousands of requests to navigation, categories, and pagination, you get a ready-made list of necessary pages with just a few dozen requests to XML files. This saves time on launching parsing, reduces proxy traffic consumption, and significantly decreases the risk of IP blocking, as the load on the site is much lower than with a full crawl.
If you are just starting to monitor the catalog of Wildberries, Ozon, or another large site, start with the sitemap — you will likely save hours of work on your first run. For downloading the sitemap files themselves, inexpensive data center proxies will suffice, while for subsequent regular data collection on product cards, where the risk of blocking is higher, it’s worth considering residential proxies — they provide stable parser operation without frequent bans over the long term.