If you sell on marketplaces and change prices manually once a week, competitors who adjust their prices every hour will take sales away from you. A proprietary monitoring system solves this problem, but marketplaces and competitor websites actively ban bots, IP addresses, and entire data center subnets. In this article, we will discuss a working scheme for data collection without blocks — with the right choice of proxies, tools, and request frequency.
Why You Need Your Own Competitor Monitoring System
Ready-made price monitoring services (there are dozens on the market) cost from several thousand rubles per month and usually limit the number of products or the frequency of data updates. If you have a catalog of 500–5000 SKUs and sell simultaneously on Wildberries, Ozon, and Avito, subscribing to three different services becomes more expensive than having your own system, and you still lack flexibility: you cannot set custom price recalculation logic or add monitoring for a non-standard competitor website.
Your own system provides three specific advantages. First, you control the frequency of collection — you can check the prices of top competitors every 15 minutes, while others once a day. Second, you collect not only the price but also stock levels, ratings, the number of reviews, and positions in search results — any parameters that are important for your business. Third, the data remains yours: you can cross-reference it with internal margin analytics and automatically recalculate prices to avoid losses.
Sellers who have implemented automatic price recalculation based on competitor monitoring typically increase the conversion rate of their position in the top 10 category by 15-25% because the price reacts to market changes faster than manual adjustments. This is especially critical in highly competitive categories — clothing, electronics, home goods.
Why Websites and Marketplaces Ban Scraping
Wildberries, Ozon, and most online stores use anti-fraud systems that analyze request behavior. If 200 requests per minute come from one IP without pauses, without typical browser headers, and without executing JavaScript — it is instantly flagged as a bot. The system either serves a captcha, provides deliberately incorrect data (fake prices), or bans the IP for several hours.
A separate problem is data center subnets. IP addresses that belong to Amazon AWS, Google Cloud, DigitalOcean, and similar providers are blacklisted by most anti-fraud systems even before you make your first request. Marketplaces see the ASN (autonomous system number) of the provider and block the entire range, even if you send one request per hour.
This is why classic server proxies for monitoring marketplaces perform poorly: they are fast and cheap, but they get flagged at the IP reputation level. To bypass this, you need IPs that look like regular users — this is the key idea of the scheme we will discuss next.
System Architecture: What It Consists Of
A working competitor monitoring system consists of five blocks, and each can be assembled without deep programming skills — using ready-made no-code and low-code tools.
- Target List — a table with links to competitor product cards that need to be monitored. Usually, this is Google Sheets or Excel with columns: URL, marketplace, your SKU, desired check frequency.
- Data Collector (Scraper) — a program or service that opens a page and retrieves the price, stock, and rating. It can be a ready-made no-code parser or a script in Python.
- Proxy Layer — a layer through which all scraper requests pass to prevent the IP from being banned by the marketplace.
- Data Storage — a database or table where the results of each pass are stored with timestamps.
- Reaction Logic — rules by which the system notifies you (via Telegram, email) or automatically changes the price through the marketplace API.
To start, a combination of: Google Sheets + no-code parser (for example, based on Make/Zapier or specialized parsing services) + proxies + Telegram bot for notifications is sufficient. This can be set up in one day without writing code. For a more serious scale (thousands of SKUs), a Python script with requests/BeautifulSoup or Selenium libraries, which runs on a schedule on a server, will be needed.
Which Proxies to Choose for Monitoring
Choosing proxies is 70% of the success of the entire system. Let's discuss three types and when each is appropriate.
| Proxy Type | When to Use | Risk of Blocking |
|---|---|---|
| Data Center Proxies | Simple websites without anti-fraud, low-frequency monitoring of small online stores | High on marketplaces, low on regular sites |
| Residential Proxies | Wildberries, Ozon, Avito — the main tool for monitoring marketplaces | Low with proper rotation |
| Mobile Proxies | Point checks of cards with strong bot protection, bypassing captchas during manual checks | Minimal, but more expensive to operate |
For daily monitoring of thousands of cards on Wildberries and Ozon, the optimal choice is residential proxies, as their IPs belong to real home providers and are not listed in data center blacklists. They are more expensive than server proxies, but save time on bypassing blocks and captchas, ultimately reducing the total cost of ownership of the system.
Mobile proxies should be kept in reserve for situations when even a residential IP receives a captcha — for example, when monitoring a specific complex category of goods with enhanced protection. The mobile operator's server uses CGNAT (shared IP for thousands of subscribers), so banning such an IP affects real users, and platforms use it very cautiously.
Data Collection Tools
If you are not a programmer, there are several ways to build a system without code. First, no-code parsing and price monitoring services allow you to specify the product card URL and select the CSS selector for the price element through a visual interface — literally by clicking on the desired block of the page. Second, you can use a combination of Google Sheets + a browser extension that checks pages on a schedule and records the results in a table.
If there is someone on the team with basic technical skills, a Python system provides maximum flexibility. A minimal script looks like this:
import requests
from bs4 import BeautifulSoup
proxies = {
"http": "http://login:password@proxy_host:port",
"https": "http://login:password@proxy_host:port",
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
url = "https://example-shop.ru/product/12345"
response = requests.get(url, proxies=proxies, headers=headers, timeout=15)
soup = BeautifulSoup(response.text, "html.parser")
price = soup.select_one(".product-price").get_text(strip=True)
print(f"Current price: {price}")
For pages that load prices via JavaScript (this is typical for Wildberries and Ozon), regular requests will not work — a headless browser based on Selenium or Playwright is needed, which fully renders the page as a regular user would. This approach is slower but provides accurate data even with dynamic content.
Setting Up Rotation and Request Frequency
Even with quality proxies, the system can get banned if requests are made too frequently and uniformly. The anti-fraud algorithms of marketplaces analyze not only the IP but also the behavior pattern: intervals between requests, the order of accessing pages, and the repetition of headers.
Working rotation rules that reduce the risk of blocking:
- Change IP for each request or every 5-10 requests — most residential proxy providers support automatic rotation by timer or on demand.
- Add random delays between requests from 3 to 15 seconds instead of a fixed interval.
- Alternate User-Agents from a list of real browsers — do not use the same header for thousands of consecutive requests.
- Distribute the load throughout the day: do not check the entire catalog in one go at 00:00, spread the checks over several hours.
- Use geographically relevant IPs — if a competitor sells in Russia, the proxy should have a Russian geolocation; otherwise, prices and availability may differ from the actual ones for your region.
For monitoring 1000-3000 cards per day, a pool of several dozen concurrent IPs with rotation every 5-10 minutes is sufficient. For larger scales (tens of thousands of checks), a provider with a large pool of residential IPs and API rotation support is needed to integrate IP changes directly into the data collection script.
Features of Monitoring Wildberries, Ozon, and Avito
Each platform has its own protection specifics, and the scheme needs to be adapted to it.
Wildberries provides data on prices and stock through an internal API that can be called directly, receiving a JSON response without the need to parse HTML markup. This is faster and more stable than rendering the page in a browser, but this API has limits on request frequency from one IP, so proxy rotation is mandatory when monitoring more than 50-100 products per hour.
Ozon actively uses JavaScript rendering and more aggressive bot protection, including mouse behavior and scrolling speed checks on some sections. For stable monitoring, Ozon usually requires a headless browser that emulates real user actions, rather than just HTTP requests.
Avito primarily tracks the frequency of requests from one IP and the patterns of searching for ads. For monitoring competitor ads (for example, tracking new ads in your niche or price changes), it is critical to maintain a low request frequency — no more than one request per ad every 10-15 minutes from one IP, otherwise the account or IP goes into a shadow ban, where outdated data is provided without an explicit error.
Many SMM specialists and arbitrageurs for similar tasks of multi-accounting use anti-detect browsers like Dolphin Anty, AdsPower, or Multilogin — these same tools are suitable for manual point checks of competitor cards if automatic collection shows suspicious data and manual verification is needed through a separate profile with a linked proxy.
Data Storage and Dashboards
Collected data is useless without a convenient way to analyze it. For small catalogs (up to 500 SKUs), Google Sheets with price history by dates and conditional formatting that highlights competitor price changes in red or green is sufficient. For larger volumes, a database is needed — PostgreSQL or even SQLite will work if the system operates on one server.
Based on the collected data, it is convenient to build simple dashboards in Google Data Studio (Looker Studio) or Power BI, where you can see: the dynamics of competitor prices over the last 30 days, the price difference relative to your position, and the frequency of price changes for each competitor. This visualization helps to see patterns — for example, a competitor changes prices every Monday morning, and you can adjust your strategy to this cycle.
It is important to store not only the current price but also the entire history of changes with timestamps. This allows for later analysis of seasonality, identifying dumping before sales, and forecasting competitor behavior in the next similar period (for example, before 11.11 or Black Friday).
Common Mistakes When Building a System
In practice, most homemade monitoring systems fail due to several typical mistakes that can be easily avoided in advance.
- One IP for the entire monitoring. Even a quality proxy will eventually get banned if thousands of requests go through it without rotation.
- Ignoring browser headers. A request without a User-Agent or with an outdated header is an instant bot signal for the anti-fraud system.
- Too high check frequency. Monitoring every minute makes sense only for 5-10 key competitors; for the rest of the catalog, checking every few hours is sufficient.
- Absence of error handling. If the site returns a captcha or is temporarily unavailable, the system should recognize this and retry later, rather than recording an empty or erroneous value as the current price.
- Data center proxies for marketplaces. Saving on proxy types leads to constant bans and the need to change IPs manually every day — ultimately, the system requires more manual labor than it should.
Conclusion
A proprietary competitor monitoring system on marketplaces is not a one-time setup but a working tool that requires the right architecture: a target list, scraper, proxy layer, data storage, and reaction logic to changes. The key link that determines the stability of the entire scheme is the proxies. Server IPs are quickly banned by the anti-fraud systems of Wildberries and Ozon, so for regular monitoring of marketplaces, it is worth budgeting for quality IPs with rotation.
If you plan to monitor competitor prices on Wildberries, Ozon, or Avito, start with residential proxies — they provide high reliability of collected data and minimal risk of blocks with proper rotation. For point checks of the most protected cards, keep mobile proxies in reserve, and for auxiliary tasks on less protected sites, more budget-friendly data center proxies will suffice.