If you regularly scrape Wildberries, monitor competitor prices on Ozon, or automate data collection — you know that proxy costs can seriously impact your budget. Requests to the same pages, reloading static data, updating unchanged information — all of this consumes traffic and money. The solution is simple: properly configured data caching can reduce the load on proxies by 50-70% without losing the relevance of the information.
In this guide, we will explore practical ways to cache data for various tasks: from scraping marketplaces to monitoring competitors. You will learn which data can be safely cached, how to set storage times, and which tools to use without programming skills.
Why Caching is Critical for Working with Proxies
Imagine a situation: you monitor prices for 500 products on Wildberries every hour. Without caching, your scraper makes 500 requests through proxies every hour — that’s 12,000 requests per day. At the average cost of residential proxies, this results in significant expenses, especially if most of the data doesn’t change at all.
Statistics show that when scraping marketplaces, up to 60-70% of requests return identical data: product descriptions don’t change, specifications remain the same, images are static. Only prices, stock levels, and search positions change. If you cache static data and only update dynamic data — traffic savings can reach 50-70%.
Real Example: An online store monitored prices for 1200 competitor products on Ozon without caching — resulting in 28,800 requests per day. After implementing caching for static data (descriptions, specifications) with updates every 7 days and price caching for 1 hour — the consumption dropped to 9,600 requests. The proxy traffic savings amounted to 67%.
Caching addresses three key issues:
- Reducing proxy traffic costs — fewer requests = lower payment for gigabytes
- Decreasing the risk of bans — fewer requests to the target site = lower likelihood of being banned for frequency
- Speeding up scraper performance — cached data is delivered instantly, without delays for network requests
What Data Can Be Cached When Scraping
Not all data is equally suitable for caching. It is important to differentiate between static information (rarely changes) and dynamic information (frequently updates). An incorrect caching strategy will lead either to outdated data or a lack of savings.
| Data Type | Update Frequency | Cache Duration | Traffic Savings |
|---|---|---|---|
| Product Descriptions | Once a month | 7-14 days | Up to 80% |
| Specifications and Parameters | Once a month | 7-14 days | Up to 75% |
| Product Images | Every 2-4 weeks | 14-30 days | Up to 90% |
| Customer Reviews | Daily | 12-24 hours | Up to 50% |
| Product Prices | Several times a day | 1-3 hours | Up to 40% |
| Stock Levels | Every hour | 30-60 minutes | Up to 30% |
| Search Positions | Constantly | Do not cache | 0% |
The golden rule: the less frequently data changes, the longer it can be stored in the cache. Product descriptions on Wildberries or Ozon are updated very rarely — they can be confidently cached for a week or two. Prices change more often, but even here a cache of 1-3 hours will provide significant savings if you do not need real-time monitoring.
Caching Strategies for Different Tasks
Effective caching is not just about "saving data for a day." Each task requires its own strategy that balances data relevance and traffic savings. Let's consider proven approaches for typical scenarios.
Multi-Level Caching
The most effective strategy is to divide data into several levels with different storage times. This allows you to minimize the load on proxies while maintaining the relevance of critical data.
Example of Multi-Level Cache for Scraping Wildberries:
- Level 1 (30 days): Product images, brands, categories
- Level 2 (7 days): Descriptions, specifications, composition
- Level 3 (24 hours): Ratings, number of reviews
- Level 4 (2 hours): Prices, discounts, promotions
- No cache: Stock levels, search positions
With this strategy, instead of making 1000 requests every 2 hours for 1000 products, you make about 300-350 requests: most of the data is taken from the cache, and only requests for fresh prices and stock levels go through proxies.
Caching with Change Checks
A more advanced approach is to use conditional requests. Instead of fully loading the page, you send a lightweight request to check if the data has changed since the last time. If not — use the cache; if yes — load the update.
Many sites support HTTP headers for conditional requests: If-Modified-Since or ETag. If the page has not changed, the server will return a 304 (Not Modified) code without a response body — you save 95% of traffic on this request.
Intelligent Cache Updates
Instead of updating all data on a schedule, update only those that are likely to have changed. For example, if a product is on promotion — check the price every hour. If a regular product has not changed in the last 2 weeks — check once a day.
Tip: Track the history of changes. If a product's price changes every day — reduce the cache time to 1 hour. If the price has been stable for a month — increase it to 6-12 hours. Adaptive caching can provide an additional 20-30% savings.
Caching Tools Without Programming
You don't need to be a programmer to set up caching. Modern scraping and automation tools have built-in caching features that can be configured through a graphical interface.
Octoparse — A Parser with a Visual Builder
Octoparse is a popular tool for scraping websites without coding. In the task settings, there is a section "Advanced Settings" → "Cache Management," where you can specify:
- Which page elements to cache (images, text blocks, tables)
- Cache duration (from 1 hour to 30 days)
- Update conditions (on a schedule or when certain fields change)
Example setup for scraping Ozon: cache the product description block for 7 days, the price block for 2 hours. Octoparse will automatically skip requests to descriptions if they are already in the cache and will only update prices through proxies.
ParseHub — Caching for Complex Sites
ParseHub specializes in scraping sites with dynamic content (JavaScript, AJAX). In the "Project Settings" section, there is an option for "Data Caching":
- Smart Cache — automatically identifies static elements and caches them
- Custom Cache Rules — you manually specify CSS selectors for elements to cache
- Cache Duration — cache lifespan from 30 minutes to 90 days
ParseHub works well with marketplaces that have a lot of JavaScript: Wildberries, AliExpress, Yandex.Market. The tool automatically determines which data is loaded dynamically and caches repeated requests.
Screaming Frog — For SEO Specialists
If you use Screaming Frog to analyze competitor websites or monitor rankings, the built-in caching will save a lot of traffic. In the settings "Configuration" → "Spider" → "Advanced," enable:
- Cache Pages — save HTML pages locally
- Cache Images & CSS — do not reload static resources
- Use Cached Data — use saved data during re-scanning
This is especially useful when regularly monitoring the same sites: the first scan loads everything through proxies, subsequent scans only load changed pages.
Caching When Scraping Marketplaces
Marketplaces are the most popular scraping task among e-commerce businesses. Wildberries, Ozon, and Yandex.Market have similar data structures, allowing for a universal caching strategy.
Scraping Wildberries with Minimal Traffic Consumption
A typical task: monitoring 500 competitor products. Without caching — 500 requests every 2 hours = 6000 requests per day. With the right cache — up to 1500-2000 requests per day.
Step-by-step Cache Setup for Wildberries:
- First request for a product: save the full card (description, specifications, images) in a local database or JSON file
- Extract and separately save the product article — this is the unique identifier
- On the next request: check if the article is in the cache and if the storage time has expired
- If the cache is valid: take the description and specifications from the cache, and through proxies request only the block with the price and stock levels (this is a separate API endpoint at Wildberries)
- Combine the cached data with the fresh price — you get complete up-to-date information
Wildberries provides prices and stock levels through a separate lightweight API request (about 2-5 KB instead of 200-500 KB for the full page). If you cache the heavy part and only request prices — traffic savings can reach 90-95%.
Optimizing Ozon Scraping
Ozon has more aggressive protection against scraping, so every unnecessary request increases the risk of being banned. Caching here not only saves money but also reduces the likelihood of bans.
A feature of Ozon: product cards often contain identical blocks (brand description, standard category specifications). If you scrape 100 products of the same brand — the brand description will be identical. Cache such repeating blocks separately:
- Brand description → cache for 30 days
- Standard category specifications (e.g., "Composition" for clothing) → cache for 14 days
- Unique description of a specific product → cache for 7 days
- Price and availability → request every 2-4 hours
Avito: Caching Ads
When scraping Avito (monitoring competitors, tracking new ads), it is important to consider that ads are often removed from publication. It is pointless to store data of a removed ad in the cache.
Strategy: cache only active ads and regularly check their status with a lightweight request. If the ad is removed — clear the cache. This will prevent database clutter and speed up scraper performance.
Optimizing Competitor Price Monitoring
Price monitoring is a task where caching provides maximum effect. Prices do not change every minute, but they need to be checked regularly. Proper cache setup allows tracking changes without unnecessary requests.
Adaptive Check Frequency
Not all products require the same monitoring frequency. Products with dynamic prices (electronics, sale items) need to be checked more often. Products with stable prices (building materials, furniture) — less often.
Example of Adaptive Price Caching:
- Product with price change in the last 7 days → check every 2 hours, cache for 2 hours
- Product without changes for 7-30 days → check every 6 hours, cache for 6 hours
- Product without changes for more than 30 days → check once a day, cache for 24 hours
This approach reduces the number of requests by 40-60% compared to a fixed check frequency. When monitoring 1000 products instead of 12,000 requests per day (every 2 hours), you make 5000-7000.
Caching with Change Notifications
Instead of constantly updating all prices, set up a system: check prices on a schedule, but update the cache only when changes occur. If the price has not changed — extend the current cache duration without a new request to the site.
Many parsers (Octoparse, ParseHub) support the "Update only if changed" mode. The tool makes a request, compares the new data with the cache, and if there is no difference — it does not overwrite the cache, but simply updates the last check time.
Common Mistakes When Setting Up Cache
Incorrect caching can lead to outdated data, loss of important information, or, conversely, a lack of savings. Let's discuss common mistakes and how to avoid them.
Mistake 1: Too Long Cache for Dynamic Data
Caching prices for 24 hours when monitoring competitors is a bad idea. Prices can change 3-5 times in a day, especially in highly competitive niches. You will achieve traffic savings, but lose data relevance.
Solution: Determine the actual frequency of data changes. Conduct a test: monitor 50-100 products every hour for a week and see how often prices change. Based on this, choose the optimal cache time.
Mistake 2: Caching Without Versioning
If you simply overwrite the cache with each update, you lose the history of changes. This is critical for analyzing price dynamics: it is impossible to build a price change graph for a month if old data is erased.
Solution: Keep cache versions with timestamps. For example, instead of a file product_12345.json, create product_12345_2024-01-15.json. This will allow you to analyze history and revert to previous data versions if necessary.
Mistake 3: Ignoring Cache Size
Caching thousands of products with full HTML pages will quickly fill up the disk. A cache for 10,000 products can take up 5-10 GB if full pages with images and scripts are saved.
Solution: Cache only the necessary data. Instead of saving the entire HTML page, extract specific fields (name, price, description) and save them in a structured format (JSON, CSV). This will reduce the cache size by 10-20 times.
Tip: Set up automatic cleaning of outdated cache. Data older than 30-90 days is usually not needed for current work — archive it separately or delete it. This will speed up scraper performance and free up disk space.
Mistake 4: Lack of Cache Error Handling
If the cache is corrupted (write failure, disk error), the scraper may use incorrect data or crash altogether. This is especially critical during automated monitoring: you may receive outdated data for several days without knowing it.
Solution: Add cache integrity checks. Store a checksum (hash) of the data along with the cache. When reading, check: if the hash does not match — the cache is corrupted, a fresh request through the proxy is needed.
Conclusion
Properly configured caching is a simple way to reduce proxy costs by 50-70% without losing data quality. Key principles: separate data into static and dynamic, use multi-level caching with different storage times, and adapt the update frequency to the actual dynamics of changes.
For most scraping tasks on marketplaces and price monitoring, complex technical solutions are not needed — modern tools like Octoparse or ParseHub have built-in caching features that can be configured in 10-15 minutes through a graphical interface.
Start simple: cache product descriptions for a week, prices for 2-3 hours. Track the results for a week and adjust the settings based on real change statistics. Even basic caching will provide a 30-40% traffic savings, while optimized caching can yield up to 70%.
If you are engaged in scraping marketplaces or monitoring competitor prices, we recommend using residential proxies in conjunction with caching — this will ensure stable operation without bans and minimal traffic costs. For tasks where speed is critical and large volumes of data are needed, data center proxies are suitable — they are faster and cheaper with the right rotation and cache settings.