Google has fewer and fewer "legitimate doors" to its search results. The Custom Search JSON API is closed to new clients and will cease to function entirely on January 1, 2027, and since September 13, 2026, the share of blocked requests for services that scrape results has significantly increased. For SEO teams, aggregators, price monitoring, and AI agents using web search, this means one thing: by the end of the year, you need to decide where you will source Google results in 2027.
What happened: two news items in one month
The official API has three months left
Google announced the closure of the Custom Search JSON API back on January 20, 2026, but for most teams, the deadline has only just begun to feel real. The documentation page clearly states: the API is closed to new clients, and current users have time until January 1, 2027 to switch to an alternative.
Until that date, the previous conditions apply:
- 100 free requests per day;
- then $5 for 1000 requests;
- the ceiling is 10,000 requests per day.
This was the cheapest and simplest official way to obtain search results in JSON. Developers on Google forums complained back in July about 403 errors when the API was enabled, billing was active, and the quota had not been exhausted, so for some, the API effectively stopped working even before the official date.
Scrapers are facing tougher restrictions
On September 18, Search Engine Roundtable gathered signals from SEO tool operators. Derek Perkins from Nozzle reported that since around September 13, most attempts to retrieve data from Google Search are being blocked: the volume of collected data has dropped by about 80%. According to him, a similar picture was observed with DataForSEO's data. Steve Payne from Sistrix noted that Google has been continuously changing the way it delivers results for about a year, and now the pace of changes is very high, leading to a slowdown in their data collection.
Glenn Gabe compared tracker data with actual website traffic: Ahrefs showed the best recovery, Semrush displayed it with a delay, and Sistrix lagged significantly behind. There has been no official comment from Google.
Why this is not a coincidence, but a trend
If you line up the events, a consistent policy emerges: first, make automated data collection more expensive, then close off cheap official access.
- January 2025. Google tightened protection on search results, leading to data gaps for Semrush, SE Ranking, and other trackers.
- September 2025. The parameter
&num=100, which allowed for 100 results on one page, disappeared. To gather the same depth, significantly more requests are needed. - January 2026. The closure of the Custom Search JSON API was announced.
- August 2026. Links in search results for automated clients were replaced with encrypted redirects to google.com/goto. We discussed in detail how goto links disrupt tracker reports and the costs involved in decoding them.
- September 2026. A new wave of scraper blocks began on September 13.
Microsoft has already gone through this path: on August 11, 2025, it completely shut down the Bing Search API family (web, images, news, video, Custom Search, and others) and offered Grounding with Bing Search in Azure AI Agents as a replacement. This is no longer a JSON output, but context for LLM responses. The two largest search engines are no longer selling raw results to anyone who wants them.
What Google offers as a replacement
There are three official options, and none of them is a "one-to-one" replacement.
- Vertex AI Search. Google recommends this for searching across sites within 50 domains. It is an enterprise product for searching your own index and does not return public Google results.
- Search Element. A free site search widget, with the same limitations of up to 50 domains. It is not suitable for programmatic data collection.
- Web Search Service API. On September 11, Search Engine Journal noted the documentation for a new "full web search" API. It returns JSON via REST or gRPC, up to 20 results per request, but requires a Google Cloud project, API key, and client ID tied to a partner agreement. Each request must include the user's IP. The documentation does not specify how to become a partner, the costs involved, or any limits. Officially, Google only offers to fill out an interest form.
In short: for a small project searching across a few sites, there is a solution. However, for tasks like "finding out what Google shows for query X in country Y," there will be no official cheap route after January 1.
Who will be most affected
- Internal SEO scripts. Many teams checked positions or indexing through CSE: 10,000 requests per day were sufficient for a small pool of keywords. Starting in 2027, this code will start returning errors.
- AI agents and RAG pipelines, where web search was connected through Custom Search as the most accessible "official Google."
- Brand and content monitoring: searching for mentions, text copies, phishing domains through Google queries.
- Users of SEO platforms. Scrapers will be forced to make more requests for the same volume of data, and the cost will ultimately be passed on to subscriptions. In September, many also experienced gaps in their position graphs.
What to do before January 1: four paths
1. Check if you actually need Google results
Start with an audit: where in the code is customsearch/v1 called and why. Often, CSE was used for searching within one's own site or across several domains, and in that case, Vertex AI Search or Search Element can fulfill the task. If you specifically need positions in Google, this route won't work.
2. Third-party SERP API
The fastest migration option: the provider handles blocks, CAPTCHAs, and HTML parsing, and you receive JSON. Prices in the market vary widely. According to a comparison published by the SERP provider cloro (which is not an independent benchmark), the range is from about $1–2 to $12 for 1000 requests of 10 results each. The downsides are dependence on someone else's infrastructure: the wave of blocks in September showed that even a provider selling data to others can experience drops.
3. Your own scraper on proxies
This gives control over geo, frequency, and data format, but requires engineering work and adherence to rules. Google prohibits automated search queries without permission in its policies, so this solution comes with conscious legal and operational risks, rather than being a "free replacement for the API." If you choose this path:
- for requests to the search results, use IPs that look like regular users: residential proxies with country and city selection or mobile for the most sensitive scenarios. Data center addresses are recognized by Google the fastest;
- tie geo IPs to request parameters (
gl,hl, location). The results are personalized by location, and a mismatch between IP and parameters will yield "incorrect" results; - keep requests similar to browser requests: real rendering or correct TLS and headers, pauses, moderate frequency per IP;
- for auxiliary traffic that does not go to google.com (downloading found pages, checking redirects), cheaper data center proxies will suffice. Do not waste residential traffic where protection does not require it;
- measure the share of successful responses by day. If it drops sharply, like on September 13, stop the collection and investigate, rather than increasing frequency: this is the fastest way to "burn" your pool.
The practical aspects of scraping—rendering, parsing AI Overviews blocks, geo selection—are described in our guide to scraping Google results with AI Overviews.
4. Change the source of truth for positions
For your own sites, you have data that does not depend on scrapers: Google Search Console provides average positions, impressions, and clicks for queries through a free API. Glenn Gabe's checks show why this is important: trackers diverged in September, while actual traffic remained a point of reference. Use Search Console as a benchmark, and external trackers for competitors and those queries where you do not yet have impressions.
How to understand that a drop in the report is not your problem
If you see a sharp drop in positions or gaps in data in the tracker after September 13, before changing the site:
- compare the drop dates with Search Console: if there are steady impressions and clicks there, the problem lies in data collection, not ranking;
- check if all keywords are affected at once. A simultaneous drop for hundreds of queries usually indicates a collection failure;
- check the status page and blog of your SEO tool: most providers publish notifications about collection issues;
- do not draw conclusions based on a single tool—Ahrefs, Semrush, and Sistrix recovered at different speeds in September.
Conclusion
Google is systematically closing off cheap access to its results: first num=100, then goto redirects, now the end of the Custom Search API and a new wave of blocks. There is an official replacement for full web search, but only for partners and without public pricing. Before January 1, it is worth finding all CSE calls in the code, determining whether you actually need Google results, and choosing a path: SERP API, your own scraper on quality proxies, or Search Console as the source of truth for your sites. Those who postpone migration until December will have to rush to do it while trackers and providers adapt to the new rules.