Proxies for Firecrawl, Crawl4AI, and Crawlee: guide 2026

The scheme "launched Firecrawl in Docker, pointed it at a list of domains, and received markdown for RAG" works perfectly up to the first thousand pages. After that, two bills come in. The first is from anti-bots: some domains start returning 403 instead of content, creating gaps in the knowledge base that you discover when the assistant responds, "there is no information in the provided materials." The second bill is for traffic: the crawler dutifully pulls every image and font, which will ultimately not make it into the final markdown.

Let's discuss how to connect proxies to the three most popular LLM crawlers of 2026 — Firecrawl, Crawl4AI, and Crawlee — and how to configure them so that the proxy only works where it's needed, rather than consuming gigabytes on every page.

Who this guide is for

If you are collecting a corpus of documents for RAG, filling an internal knowledge base, building a data pipeline for fine-tuning, or simply regularly scraping hundreds of domains — this is for you. All three tools below, in their default configuration, operate with the IP of your server and load the entire page. Both of these defaults need to be changed.

The scale of the problem is evident from popularity metrics: Firecrawl has around 170,000 stars on GitHub at the time of publication (license AGPL-3.0), Crawl4AI has about 79,000, and Crawlee from Apify has around 25,000. These are no longer niche experiments but standard tools, and anti-bot systems are familiar with their behavior just as well as you are.

First bill: 403 instead of content

The key mistake when collecting a corpus is assuming that the crawler has worked successfully if it hasn't crashed. Firecrawl and Crawl4AI return not an exception but a result on a blocked page: an anti-bot placeholder page, a browser check page, or a short text denying access. Formally, this is valid markdown, and it can be stored in the vector database until the first user request.

Therefore, the first thing to do before any proxy configuration is to add quality control for the results. The minimum option: filter out documents shorter than a certain threshold (for a typical content page, 500–800 characters of text is reasonable) and separately catch characteristic markers in the text — mentions of connection checks, enabled JavaScript, "Access denied." Such documents should not be sent to the database but queued for a re-crawl — already through the proxy.

Second bill: gigabytes you are throwing away

Here, arithmetic helps. According to Web Almanac from HTTP Archive for 2025, the median homepage weighs about 2.86 MB on desktop and 2.56 MB on mobile. Of this, images account for about 1,059 KB on homepages and 911 KB on internal pages, while JavaScript accounts for 697 KB and 632 KB, respectively. This means images are the heaviest category, making up about a third of the page weight.

Now remember what you do with the result. You convert the page into markdown and cut it into chunks for embeddings. Images do not make it into this pipeline at all — at best, you are left with a line containing alt text. Videos, fonts, analytics scripts, advertising pixels — also bypassed.

If you run the crawl through a residential proxy with payment per gigabyte, you are literally paying for the delivery of data that you throw away at the next step of the pipeline. With a corpus of 100,000 pages, the difference between "pull everything" and "pull only HTML and text" is measured not in percentages but in multiples. The exact savings depend on the themes of the sites: media and e-commerce are heavier than documentation and blogs.

Step 1. Escalation instead of "proxy for everything"

The main architectural technique that saves the most is: do not route all traffic through the proxy. Most domains when collecting a knowledge base — documentation, blogs, reference sites, government portals — deliver content directly and do not block anyone. The proxy is needed by a minority.

The correct scheme is a multi-level escalation: first a direct request, and upon signs of blocking — move to the next level. Moreover, this is not a homemade hack; both major frameworks can do this out of the box.

In Crawlee, this is achieved with tieredProxyUrls. Levels are listed from cheap to expensive, and the crawler automatically escalates when blocks occur and then periodically tries to revert to the lower level:

const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        [null],
        ['http://user:pass@datacenter-proxy:8080'],
        ['http://user:pass@residential-proxy:8000'],
    ]
});

An important nuance from the documentation: tieredProxyUrls only works when used through an instance of the crawler. Direct calls to newUrl() will yield unexpected results.

A similar mechanism appeared in Crawl4AI version 0.8.5 and is present in the current branch (the latest release at the time of publication is v0.9.2 from July 15, 2026). It is called proxy escalation and is configured directly in CrawlerRunConfig: a three-level block detection — known anti-bot vendors, general block indicators, and structural integrity checks of the page — plus automatic retries through the proxy chain.

from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import ProxyConfig

config = CrawlerRunConfig(
    proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server="http://my-proxy:8080")],
    max_retries=2,
)

Note that ProxyConfig.DIRECT is the first element — this means "first try without a proxy."

Step 2. Connecting proxies in each tool

Next, let's get into specifics about the configuration. The order of actions is the same: first the proxy, then cutting unnecessary traffic, then verification.

Firecrawl (self-hosted). The proxy is set with three environment variables passed to Playwright: PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD. They are specified in .env for apps/api; in the comments, the developers explicitly state that instead of a static address, you can specify a proxy service that rotates IPs for each request.
Crawl4AI. The proxy is located in BrowserConfig, in the proxy_config field — this is a ProxyConfig object or a dictionary with fields server, username, password. One browser configuration is used for the entire crawling session; a separate CrawlerRunConfig is passed for each call to arun().
Crawlee. The ProxyConfiguration class with the proxyUrls option — a list of addresses that the library cycles through (round-robin). A null value in the list means "no proxy." The integration is seamless: HttpCrawler, CheerioCrawler, JSDOMCrawler, PlaywrightCrawler, PuppeteerCrawler.
Point rules. If you know which domains block and which do not, Crawlee has newUrlFunction — your own logic for selecting a proxy based on the request URL. For whitelisted domains, return null; for others — the proxy address. This is the cheapest option when the target list is stable.
Verification. Before the production run, pass a page that returns your external IP through the configured crawler and ensure that you see the proxy address, not the server's. Three lines that save a day of troubleshooting.

Step 3. Cut everything that won't become text

Once the proxy is connected, enable traffic saving — otherwise, the bill for gigabytes will come faster than the corpus is collected.

In Firecrawl, this is managed by the BLOCK_MEDIA variable. In the official configuration example, it has a literal comment: set it if you want to block media requests to save proxy bandwidth. This is the quickest way to eliminate the main expense.

In Crawl4AI, similar levers are found in BrowserConfig: text_mode disables images and speeds up text crawling, light_mode turns off some background browser functions, and avoid_css blocks CSS loading. They can be combined. For collecting a corpus for RAG, this is almost always the right set — you don't need layout, you need text.

In Crawlee, the logic is different: if the content is delivered in HTML, use CheerioCrawler or HttpCrawler instead of browser-based ones. A regular HTTP request instead of a full render is not just traffic saving; it's a different order of expenses. Keep browser crawlers (PlaywrightCrawler, PuppeteerCrawler) only for pages that cannot be scraped without JavaScript.

Pitfalls

Sessions vs. rotation. Changing the IP for each request looks suspicious in itself and breaks multi-step scenarios — pagination, transitions within a single domain. In Crawlee, each call to newUrl() ties the proxy to a Session object, and they rotate together with browser fingerprints and headers. Do not break this link manually.

Media disabled, but content is missing. Some sites with lazy loading pull not only images but also text. After enabling text_mode or BLOCK_MEDIA, be sure to run a control sample of 20–30 pages and compare the text volume with the benchmark.

Retries without a ceiling. Escalation through proxy levels means that one stubborn page can be downloaded three times — and all three times paid for. Limit max_retries and create a list of domains that are excluded from crawling altogether after N failures.

Robots.txt and legal framework. Data collection for training and RAG in 2026 is regulated more strictly than a couple of years ago — from requirements for source disclosure to mechanisms for opting out of text and data mining. Ensure that your pipeline respects these signals before it operates on hundreds of thousands of pages.

What type of proxy to choose for the RAG pipeline

The answer depends on what level of escalation you are at.

Zero level — no proxy. Documentation, open-source projects, government sites, most corporate blogs. Here, the server's IP works fine, and there is nothing to pay for.
Medium level — data center proxies. Fast and cheap, suitable against simple rate limiting and regional restrictions. When collecting large corpuses, this is the workhorse: when the volume is measured in hundreds of gigabytes, the price difference per gigabyte becomes the main factor.
Upper level — residential proxies. For domains with serious anti-bot protection, where data center subnets are filtered out at the entrance. This is why they cannot be set as the default level — payment per gigabyte turns every unnecessary image into a line of expenses.

Before building the pipeline, it is worth honestly calculating the economics: we analyzed the total cost of parsing a million pages considering page weight, retries, and hidden costs. And a separate question that is useful to ask before writing the first line of code: is crawling even necessary — in the analysis of official API vs. ready-made datasets and parsing, it is clear that for some sources, ready-made data is cheaper than running your own crawler.

Conclusion

Proxies in an LLM crawler are not a simple "on/off" switch, but a three-level scheme. Direct requests as the default level, data center proxies at the medium level, and residential proxies only for domains that cannot be scraped otherwise. Plus, strict media cutting, because you are collecting text while paying for bytes.

The order of operations is simple: first, quality control of results (otherwise, you won't know that half of the corpus consists of placeholder pages), then proxy escalation using the framework's built-in tools, and finally, traffic saving. In this order — both the corpus will be complete, and the bill predictable.