How to Scrape Google Search Results in 2026, Including AI Overviews: A Guide and Required Proxies

```html

A year ago, you could scrape Google’s results with a single GET request using the parameter num=100 — a hundred results would come back as clean HTML. In 2026, that no longer works: Google has eliminated access without JavaScript, reduced the number of results per page to a hundred, and added the AI Overviews block, which renders asynchronously and is not visible from every IP. Let’s explore how to collect SERP under these new conditions, where hidden traps lie, and why the choice of proxy type has become more important than the scraper itself.

Who This Guide Is For

Parsing search results (SERP scraping) is not just about SEO positions. Today, it is used by:

• SEO specialists and agencies — tracking organic results, featured snippets, "People also ask," local results, and whether a site appears in AI Overviews.
• Market analysts — monitoring whom Google cites in AI blocks for commercial queries and how the first page changes for competitors.
• AI and data teams — collecting SERP as a data source for RAG systems, model training, and fact-checking.

They all share one problem: Google in 2026 actively distinguishes between automated traffic and real users, and without the right infrastructure, data collection breaks down on the first few queries.

What Has Changed: Three Strikes from Google Against Scrapers

To keep this guide honest, let’s start with why the old instructions no longer work.

January 2025 — SearchGuard. Google rolled out a system of JavaScript challenges: a regular HTTP request through requests or httpx now returns not HTML, but a challenge page. Without executing JavaScript, you cannot see the results — direct parsing "head-on" fails instantly.

September 2025 — the end of num=100. Google removed the parameter that provided 100 results in a single request. Now, the top 100 consists of ten separate requests with pagination. For deep monitoring, this literally means a tenfold increase in the number of requests (and, consequently, the load on proxies and budget).

December 2025 — legal pressure. On December 19, 2025, Google filed a DMCA complaint against SerpApi, stating that SearchGuard is a "technological protection measure," and circumventing it falls under anti-circumvention laws. The precedent has not yet been resolved, but it sets the tone: gray scraping of Google is becoming both technically and legally more expensive.

Additionally, note that Google is phasing out the official Custom Search API — existing clients have been given a migration deadline of January 1, 2027. This means that the "legal" alternative is also narrowing.

The Main New Feature of the Results — AI Overviews

AI Overviews (formerly SGE) are AI-generated summaries at the top of the results with links to sources. For scraping, this is the most challenging element of 2026 for three reasons.

There are many of them. According to Ahrefs, AI Overviews appear in about 30% of queries; later estimates (Olostep) suggest up to 48% of all queries and up to 80% of informational ones. Ignoring this block means collecting an inherently incomplete picture of the results.

They render asynchronously. The block exists in three states: it can arrive immediately in HTML (the rarest case), load via JavaScript seconds after the main page (the most common case), or not appear at all. In the case of delayed loading, the raw HTTP response contains an empty container — the content is pulled in later, and the parser needs to wait (in practice, about 8 seconds in browser automation).

They are not visible from every IP. This is the key point that old guides overlook: Google considers mobile users a priority audience for AI search. In practice, this means that from data center IPs, AI Overviews often do not return at all, while the same query through a mobile operator returns the full block. Even top aggregators acknowledge the incompleteness: SerpApi reported about 68% successful detection of AI Overviews at the beginning of 2026.

Step-by-Step Breakdown: How to Collect SERP in 2026

1. Determine the volume. Up to ~100 queries per day can realistically be handled with your own browser automation. From 100 to 10,000 — you will need a managed parser or SERP API. Over 10,000 per day requires enterprise infrastructure with batches and webhooks. This defines the entire subsequent stack.
2. Gather the correct URL. The basic endpoint is /search, key parameters: q (query, URL encoding), hl (interface language), gl (country of results), start (pagination: start=10 — second page, start=20 — third page, and so on). Remember: num=100 no longer works; depth is only gained through pagination.
3. Use browser rendering. Since there are no results without JavaScript, the basic stack is Playwright or Selenium with headless Chromium. Be sure to remove automation markers (the flag --disable-blink-features=AutomationControlled), otherwise the anti-bot will detect the controlled browser by navigator properties.
4. Wait for the AI Overview. After the page loads, do not grab the DOM immediately: allow networkidle to settle and wait for the block to load (the guideline is up to 8 seconds). It is more reliable to determine the presence of the block by the text of the header "AI Overview," rather than by CSS classes — they are dynamic in Google and change (conditional Kevs9, Y3BBE today are one thing, tomorrow another).
5. Parse by structure, not by classes. Take organic results by header tags (h3) and semantics, not by fragile class names. Available from the results in 2026 are: organic results, featured snippets, "People also ask," related queries, knowledge graph, local pack, ads, and citations within AI Overview.
6. Rotate IPs and throttle. Set realistic pauses between requests (4–12 seconds) and change IPs approximately every 5 minutes, varying city/operator. A too uniform rhythm and a single IP is the fastest way to a CAPTCHA.

Hidden Pitfalls

• "Empty" AI Overview. If you grab the DOM immediately after loading, the delayed block will be empty — and you will conclude that it does not exist. Always allow for waiting and rechecking.
• One-time sessions for loading. Some APIs have a session key for loading the delayed AI Overview that is one-time and lasts about 60 seconds — do not expect to reuse it later.
• False savings on data centers. Cheap data center IPs trigger CAPTCHA after just 5–10 requests and additionally do not show AI Overviews. Savings turn into incomplete data and wasted time.
• Fragile selectors. If you rely on CSS class names, the parser will break on the nearest redesign of the results. Stick to text and structure.
• Uniform request fingerprint. Identical User-Agent, timings, and headers across all streams reveal a botnet. Diversify the fingerprint just as you do with IPs.

What Type of Proxy to Choose

In 2026, it is the proxy, not the parser, that determines whether you will see the complete results. Let’s break it down by tasks.

Mobile proxies — for AI Overviews and the most "heavy" queries. Since Google delivers AI blocks primarily to mobile audiences, real operator IPs (T-Mobile, Verizon, Vodafone, and similar) trigger AI Overview more reliably and handle significantly more — based on observations, 50–200 requests before friction appears compared to 5–10 for data centers. Additionally, mobile CGNAT IP shares one address with hundreds of live subscribers, so Google is hesitant to ban it. If your task is to specifically collect AI Overviews or monitor the most protected SERP, start with mobile proxies.

Residential proxies — the workhorse for organic and volume. For collecting regular results, positions, featured snippets, and local packs, residential IPs (addresses from home providers) offer the best price-to-success ratio. They are hard to distinguish from real users, and rotation allows scaling collection without flooding from a single address. The optimal option when AI Overview is not the focus, but volume and geography are important, is residential proxies with rotation.

Data center — only for rough runs. Fast and cheap, but against Google in 2026, only a few queries survive and do not see AI blocks. Suitable for debugging parser logic, not for serious collection.

If you are unsure what to choose for a specific task, start with the breakdown of residential vs. mobile proxies in 2026: it details where each type saves money and where it saves data.

Conclusion

Scraping Google in 2026 has ceased to be a task of "writing a parser." SearchGuard forced JavaScript rendering, the cancellation of num=100 has multiplied the number of requests, and AI Overviews added a block that is mainly visible from mobile IPs and loads with a delay. Technically, everything is solvable: browser automation, structure-based parsing, reasonable pauses, and rotation. But the foundation on which the completeness and stability of collection rests is the right proxies: mobile for AI Overviews and protected queries, residential for organic and volume. Start with the type of proxy suited to your task — and the parser will stop stumbling over CAPTCHA.