An SEO specialist launches a technical audit of a large site — and after 10 minutes, Screaming Frog starts returning a stream of 403 Forbidden or 429 Too Many Requests. The protection server has mistaken the crawler for an attacking bot and blocked the IP address. The result — the audit will have to be restarted, and the client will need an explanation for the delay in the report. This problem can be solved with proper proxy configuration for the crawler — let’s break down how exactly.
Why sites block Screaming Frog during audits
Screaming Frog, Sitebulb, JetOctopus, and Netpeak Spider technically operate similarly to search engine bots: they quickly crawl thousands of URLs in succession, downloading HTML, CSS, JS, and images, checking redirects and status codes. From the perspective of website protection (Cloudflare, Qrator, DDoS-Guard, built-in WAF of hosting), this looks like suspicious activity — too many requests from a single IP in a short time.
Blocks occur particularly frequently during audits of:
- large online stores on Wildberries-like infrastructure with their own anti-scraping protection;
- sites behind Cloudflare with "Under Attack" mode enabled or a high Security Level;
- portals with dynamic content, where each request generates server load (search, filters, product listings);
- competitor sites that you are analyzing not from your domain — they have no reason to trust your IP.
An additional complication: if an agency has one static office IP, and audits of sites are conducted regularly (several clients per day), this IP quickly ends up on the blacklists of protection systems. After that, not only current audits are blocked, but also future ones from the same address — even for sites that previously opened without issues.
How to recognize a block by response codes
Before changing proxy settings, it is important to understand what is happening. Screaming Frog shows the response code in the Status Code column — it is easy to determine the reason for the crawling halt from it.
| Response Code | What it means | What to do |
|---|---|---|
| 403 Forbidden | IP is blocked by website protection | Change IP, reduce crawling speed |
| 429 Too Many Requests | Request limit exceeded in a given time | Reduce the number of threads, add delay |
| 503 Service Unavailable | Server is overloaded or bot protection is enabled | Change User-Agent, use residential IP |
| No response / Timeout | IP is blacklisted, requests do not reach | Complete replacement of the proxy pool |
If the same code appears en masse on thousands of URLs in a row — it is almost always an IP block, not a real problem with the site. To check this hypothesis, simply open the problematic URL in a regular browser from the same IP. If the page opens normally, but Screaming Frog receives a 403 — the issue lies with the crawler, not the site.
Which proxies are suitable for SEO crawling
The choice of proxy type depends on the task: a one-time audit of your site, regular monitoring of competitors, or scanning a protected marketplace. Let’s break down three main types and their applicability for crawlers.
| Proxy Type | Speed | Risk of Blocking | When to Use |
|---|---|---|---|
| Data Center Proxies | High | Medium-high on protected sites | Audit of your own site without strict protection |
| Residential Proxies | Medium | Low | Competitor audits, sites behind Cloudflare, marketplaces |
| Mobile Proxies | Medium-low | Minimal | Most aggressive protections, rare targeted audits |
For most regular technical audits, the optimal balance is provided by residential proxies — they use real IP addresses from providers, so the website protection perceives the crawler's requests as traffic from a regular user. This is especially important if you are analyzing not your own domain, but a competitor's site or a marketplace with a strict anti-bot system.
Data center proxies remain a good option when auditing your own site and you can whitelist the IP range in advance at the WAF or Cloudflare level — then the scanning speed will be maximum without the risk of blocks.
Step-by-step proxy setup in Screaming Frog
Screaming Frog supports proxies "out of the box", and the setup takes a couple of minutes:
- Open the program and go to the menu Configuration → System → Proxy.
- Check the box Use Proxy Server for All Traffic.
- Enter the address and port of the proxy server provided by your provider.
- If the proxy requires authentication with a username and password — specify them in the popup window on the first request.
- Click OK and restart the crawling to apply the settings.
An important nuance: Screaming Frog in both the free and paid versions can only work with one proxy server at a time through the built-in interface. If you need IP rotation on the fly (relevant when scanning from 50,000 pages), use proxies with automatic IP switching on the provider's side — this is configured not in the crawler itself, but in the personal account of the proxy service, where the IP change interval is set (for example, every 10 minutes or with each new request).
This scheme relieves the load from one address: the crawler continues to work with one port, while the IP behind it changes automatically, and the site does not see an abnormally high request frequency from one specific address.
Speed and thread settings for large sites
Even with a working proxy, overly aggressive crawling speed settings will provoke a block. In Screaming Frog, this is managed in the Configuration → Speed section. Recommended values for sites with 100,000 pages or more:
- Max Threads: 2–5 for protected sites, 10–15 for sites without strict protection;
- Max URI/s: limit to 2–3 requests per second at the start, gradually increase while monitoring response codes;
- Custom User-Agent: in the Configuration → HTTP Header section, use a User-Agent of a real browser instead of the standard "Screaming Frog SEO Spider" if the site blocks crawlers by header;
- Respect robots.txt / crawl-delay: consider the Crawl-delay directive if it is specified — ignoring it often leads to a ban at the server level.
For sites with aggressive bot protection, it is also useful to enable JavaScript Rendering only where it is genuinely necessary (SPA sites on React/Vue) — rendering JS significantly increases server load and raises the chance of blocking when crawling through one IP.
IP rotation when scanning 100,000+ pages
The larger the site, the more important it is to distribute the load correctly among IP addresses. For audits of this scale, two schemes work:
Time-based rotation. The IP changes automatically every 5–15 minutes regardless of the number of requests. Suitable for even long crawling, where scanning stretches over several hours.
Request-based rotation. Each new HTTP request goes out with a new IP from the pool. This mode provides maximum protection against blocks but may interfere with proper operation with cookie-based authentication or session parameters of the site — for a pure technical audit (collecting status codes, redirects, meta tags), this is not critical.
For agencies conducting parallel audits of several client sites, it makes sense to separate IP pools by projects — this prevents a situation where a block on one domain "accidentally" affects the reputation of the IP when working with another site.
Practical Tip: before a large audit, run a test crawl on 500–1000 URLs with the current speed and proxy settings. If no 403/429 errors appear during this time, you can confidently scale the scan to the entire site.
Checklist before starting the audit
- ✅ Proxy is connected and tested for availability (check via curl or a separate browser);
- ✅ Max Threads and Max URI/s are set conservatively for the first run;
- ✅ User-Agent is configured correctly, does not raise suspicions with WAF;
- ✅ Respect robots.txt is enabled if the site explicitly requests to observe Crawl-delay;
- ✅ JavaScript Rendering is enabled only when genuinely necessary;
- ✅ IP rotation is configured for site volumes of 20,000–30,000 pages;
- ✅ Crawl progress saving is enabled (Configuration → Save Crawl) in case of session interruption;
- ✅ For auditing marketplaces and protected portals, residential or mobile proxies are chosen instead of data center ones.
Common mistakes when crawling through proxies
Too high speed at the start. Specialists often set the maximum threads immediately, hoping to speed up the audit. As a result, the website protection reacts instantly, and within a couple of minutes, the entire proxy pool ends up on the blacklist.
One static IP for all client projects. If an agency uses the same data center proxy for dozens of audits a month, the reputation of that IP quickly declines — protection systems of different sites exchange data about suspicious addresses through shared databases (for example, Cloudflare IP Reputation).
Ignoring geo-targeting. If the audited site is set for a specific region (for example, a Russian online store with geo-blocking of foreign traffic), using proxies with IPs from another country will itself provoke access restrictions — even without aggressive crawling.
Absence of monitoring during the process. After launching a crawl for several hours, specialists often do not check intermediate response codes. As a result, they only discover the problem upon completion, when half of the report consists of blocking errors rather than real data about the site.
Conclusion
Blocks during a technical audit in Screaming Frog are almost always resolved by a combination of three factors: the right type of proxy, reasonable crawling speed, and proper IP rotation. For your own sites without strict protection, data center proxies with whitelisting are sufficient. For auditing competitors, marketplaces, and sites behind Cloudflare, IPs that appear as regular user traffic are needed.
If you regularly audit other domains or protected platforms, we recommend starting with residential proxies — they reduce the risk of blocking on the most aggressive anti-bot systems. For targeted tasks on highly protected resources, consider mobile proxies, while for large-scale audits of your own infrastructure, data center proxies remain a good and fast solution.