Fixing Layout Drift: When Your Parser Returns Empty Fields and Proxies Aren't the Issue

The parser had been running for six months, and today empty lines were added to the database. The first thought is that it got banned, and it's time to change the proxy. You change the pool, improve the quality of the IPs, pay for residential instead of datacenter ones — and the fields are still empty. Because the reason was not a ban: the site has moved to a new layout, and your CSS selector is no longer attached to anything.

This is the most expensive type of failure because it remains silent. A ban is immediately visible: 403, CAPTCHA, redirect. Layout drift doesn’t drop anything — HTTP 200, page received, traffic paid, and at the output None. Let’s break down how to distinguish one from the other in five minutes and how to stop rewriting selectors manually after each redesign.

Who Needs This

This guide is for those who keep a parser in production for longer than one sprint: monitoring competitor prices, collecting reviews, aggregating job listings, daily exports for analytics. If you run the script once and discard it — layout drift doesn’t concern you. If the script runs via cron for months, it’s your main expense for maintenance.

The scale of the problem is not exaggerated. According to analysts at GroupBWT, unmanaged structural changes on websites account for about 40–60% of recurring maintenance costs for scrapers in large projects. In certain industries, 10–15% of crawlers require repairs weekly — due to DOM shifts, fingerprinting, and throttling of endpoints. This means that fixing selectors competes in cost with bypassing anti-bot measures, yet it receives far less attention.

The background is not encouraging: in the Apify report "State of Web Scraping 2026," 65.8% of respondents increased their use of proxies, 58.3% noted a year-over-year increase in proxy costs, and more than 62% reported an overall rise in infrastructure costs, primarily due to enhanced bot protection. Against this backdrop, burning paid traffic on pages from which you still retrieve nothing is doubly frustrating.

Step 1. Distinguishing a Ban from Layout Drift

Diagnosis takes a few minutes and must be done in strict order — otherwise, it’s easy to "fix" the wrong issue.

1. Check the response code and body size. 403, 429, 503, redirect to a verification page, or a body of 2–5 KB — that’s anti-bot. HTTP 200 and a full page of 200–800 KB — the site has let you in, it’s not about the proxy.
2. Save the raw HTML to disk and open it visually. Not in the debugger, but in a browser. If the product/review/price is present, but the parser doesn’t see them — that’s layout drift.
3. Search for the required text in the file. If it exists in the HTML but is not accessible via your selector — the markup has changed. If it’s not there at all — content is being loaded by a script, requiring a browser engine rather than an HTTP request.
4. Compare with the previous successful export. Diff the old and new HTML of the same URL: usually, you can immediately see a new wrapper class, a moved block, or a replacement of id with data-*.
5. Check if the site has delivered a different version of the page. This will be discussed separately below, as proxies are indeed relevant here.

If after the third point the diagnosis is "markup has drifted," changing proxies is pointless. You need a parser that can find an element even when the selector has expired.

Step 2. What Adaptive Selectors Are

The idea is simple: instead of being tightly bound to the string .product-card > h3.title, the library remembers the "portrait" of the required element once, and on the next run, it searches for the element on the page that most closely resembles this portrait.

This is most practically implemented in Scrapling — an open Python framework by Karim Shoair. The project was released in October 2024 and by September 2026 had gathered over 78,000 stars on GitHub; the latest release at the time of writing is v0.4.15 from August 23, 2026, with commits happening daily. Python 3.10+ is required.

The mechanics of adaptive search work like this. When you call a selector with auto_save=True, Scrapling saves the element's fingerprint:

• the tag name, text, and all attributes with their values;
• the names of neighboring tags;
• the path to the element — only by tag names;
• the tag, attributes, and text of the parent.

The fingerprint is stored in a local SQLite database and is keyed by the pair "domain + identifier." The domain is taken from the page URL (or set via the adaptive_domain parameter), and the identifier defaults to the selector string itself — or your own if you pass identifier=.

When the layout changes and the usual selector returns nothing, a call with adaptive=True retrieves the saved fingerprint and runs through all the elements on the page, calculating a fuzzy similarity score — even considering the order of attributes. The element with the highest match is returned.

This is inexpensive. According to the project's official benchmarks, parsing takes 1.99 ms compared to 2.01 ms for Parsel/Scrapy, 22.93 ms for PyQuery, 80.57 ms for Selectolax, and 1541 ms for BeautifulSoup with lxml. The adaptive search for a similar element takes 2.46 ms compared to 13.3 ms for AutoScraper. This means that insurance against redesign adds about two milliseconds to the request amidst network delays of hundreds of milliseconds.

Step 3. Installing and Enabling

Installation depends on whether you need a browser:

1. pip install scrapling — just the parser, without the networking part. This is sufficient if you obtain HTML with your own code.
2. pip install "scrapling[fetchers]", then scrapling install — adds fetchers and downloads browsers with dependencies.
3. Additionally: [ai] — MCP server, [rag] — wrapper for RAG, [shell] — interactive console, [all] — everything at once. There is a ready image pyd4vinci/scrapling.

Next — two runs. The first on the live working layout saves the fingerprint, the second already knows how to survive a redesign:

1. Reference run. Create a Selector object with adaptive=True and be sure to pass url — otherwise, the domain will go into the key "default", and fingerprints from different sites will mix. Call the required selector with auto_save=True.
2. Production run. The same selector, but with adaptive=True. As long as the markup is intact, the usual path will work. When it breaks — similarity search will kick in.
3. Log discrepancies. The moment when the usual selector returns empty while the adaptive one finds something is a signal that "the site has moved," and it should be visible in monitoring, not swallowed silently.

An important detail about overwriting: saving does not accumulate. A repeated auto_save for the same pair "domain + identifier" overwrites the previous fingerprint. Therefore, the reference should be taken from a known correct page, not in a loop over the entire pool of URLs.

Step 4. Proxies: Where They Come into Play

We started with the fact that layout drift is not about proxies. This is true only halfway, and the other half costs money.

The site may deliver you a different markup due to the proxy exit. Locale, language, and country change the page template: different block orders, different classes, different price and date formats. This is not a hypothesis — in Scrapling itself, there is a notable fix: in version 0.4.12, the forced locale en-US was removed from StealthyFetcher because the imposed locale diverged from the actual geo and broke behavior. Hence the working rule: take the reference fingerprint from the same geo from which you later collect data. A fingerprint taken through a German IP will match worse with a page obtained through a Brazilian one — and you will get a false alarm of "the site changed its layout."

Practical implications:

• If the pool is multi-country — separate fingerprints via adaptive_domain, setting a key like "domain + country." Otherwise, a single record in SQLite will constantly be overwritten by versions from different geos.
• For long scenarios, keep one country and one session for the entire task. How to arrange this is detailed in the material about sticky sessions and when to use them.
• A/B tests and gradual rollouts provide two live layouts on the same domain simultaneously. Here, adaptive search is especially useful: it will extract the element from both branches, while a rigid selector will randomly return empty on half the requests.

You can set proxies in Scrapling at all levels. For quick HTTP requests, Fetcher and AsyncFetcher have a proxies parameter. For sessions, there is ProxyRotator, which takes a list of addresses — it is substituted into FetcherSession. Browser-based DynamicSession and StealthySession accept proxies at the session level so that the IP does not change in the middle of the scenario.

Another feature that saves both the pool and nerves appeared in version 0.4.12 — AutoThrottle: the library automatically adjusts pauses between requests based on server responses, doubling the delay during a block and respecting the Retry-After header. This is exactly the behavior that distinguishes careful scraping from ramping up bans with naive retries.

Pitfalls

• Do not commit SQLite fingerprints to git. The documentation warns about this directly. Also, do not use auto_save on pages with personal data — the fingerprint captures the text and attributes of the element.
• Adaptive search is not a replacement for monitoring. It will return the "most similar" element, and the most similar is not always correct. If the site swapped the discounted price and the regular price, similarity is high, but the data is incorrect. Keep checks on value ranges and the share of empty fields in the export.
• A silent failure is more expensive than a loud one. While the selector silently returns None, the pipeline continues to traverse pages and burn paid traffic. There is a separate analysis on what a gigabyte from which no data was extracted really costs — why the price of proxies per GB lies.
• The fingerprint ages. After a confirmed redesign, re-take the reference anew; otherwise, the next site edit will be considered based on an outdated portrait, and accuracy will drop.
• If there is no content in the HTML at all — adaptability won’t help; you need a browser fetcher. In 0.4.15, browser tabs began to be reused between requests, and the close_pages() method forcibly closes them; it also fixed hangs in headless mode and the Turnstile solution no longer depends on the browser's locale.

What Type of Proxy to Use for This Task

The choice is dictated not by the parser, but by the target site:

• Datacenter proxies — for sites without serious anti-bot measures: documentation, government registries, open catalogs, RSS and CSV feeds (for the latter, XMLFeedSpider and CSVFeedSpider with automatic gzip unpacking were added in 0.4.13). Cheap and fast, and the stability of the markup is usually higher here.
• Residential proxies — for marketplaces, aggregators, and anything that personalizes output by geo. This is where it’s critical to take a reference and collect data from one country; otherwise, you will be fixing not a failure, but your own geography.
• Mobile proxies — when the site delivers a mobile template and it needs to be parsed as is, or when trust in the IP is more important than the price per gigabyte.

In Brief

Empty fields in the export represent two different diagnoses with different treatments. First, check the response code and raw HTML: if the page arrived intact, there’s no need to change the proxy; the markup has drifted. Scrapling's adaptive selectors cover this class of failures in a couple of milliseconds per request — save the fingerprint on the working layout, enable adaptive=True in production, and log the triggering moments as a signal of a redesign. And keep the geo stable: half of the "sudden redesigns" in practice turn out to be a different language version of the page that arrived due to a change in exit country.

If stable geo and predictable session are just what your parser lacks, check out ProxyCove residential proxies: country selection, sticky sessions, and payment for actual traffic used.