Proxies in Rust reqwest: SOCKS5 and rotation for the parser.

Any parser that runs for more than five minutes on real websites will eventually hit an IP block. Rust with the reqwest library provides excellent performance, but out of the box, it does not solve the proxy rotation problem — this needs to be implemented manually. In this article, we will discuss how to connect SOCKS5 and HTTP(S) proxies, build a client pool, and organize IP rotation so that the parser does not fail after the first 429 and 403 responses.

Why does a parser need a proxy and what are the dangers of not having one

Websites like marketplaces, price aggregators, and social networks track anomalous activity from a single IP: dozens of requests per second, identical User-Agents, and lack of cookies between requests. As soon as the counter of suspicious actions exceeds a threshold, the IP gets added to a ban list — either temporary or permanent. For a parser, this means a complete halt in data collection until the IP is changed manually.

Proxies solve this problem by distributing requests among dozens or hundreds of IP addresses. It is important not just to "plug in a proxy," but to properly organize a pool of connections and rotation; otherwise, you will either hit the limits of reqwest itself (one client = one pool of connections), or you will start sending requests from the same proxy too frequently. We will discuss both problems in turn: first the basic connection, then the architecture of the pool.

For a production parser, not only IP rotation is important, but also the quality of the addresses themselves. If you are parsing Wildberries, Ozon, or Avito, datacenter IPs get banned faster than real user IPs — more on this in the section about choosing the type of proxy.

Basic setup of HTTP/HTTPS proxy in reqwest

reqwest supports proxies through the Proxy structure, which needs to be passed to ClientBuilder. Add the dependencies to Cargo.toml:

[dependencies]
reqwest = { version = "0.12", features = ["socks", "json"] }
tokio = { version = "1", features = ["full"] }

A minimal example with an HTTP proxy that requires authentication via username and password:

use reqwest::{Client, Proxy};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let proxy = Proxy::all("http://gate.proxycove.com:8000")?
        .basic_auth("user_login", "user_password");

    let client = Client::builder()
        .proxy(proxy)
        .timeout(std::time::Duration::from_secs(15))
        .build()?;

    let resp = client
        .get("https://httpbin.org/ip")
        .send()
        .await?
        .text()
        .await?;

    println!("Response through proxy: {resp}");
    Ok(())
}

The method Proxy::all routes both HTTP and HTTPS traffic through the proxy. If you need to separate the protocols, use Proxy::http() and Proxy::https() separately — this is useful if the proxy provider listens on different ports for different protocols.

Connecting SOCKS5 proxy

SOCKS5 operates at a lower level than HTTP proxies and supports TCP connections without examining the request content — this is faster and more stable for parsing, especially if you are accessing not only HTTP(S), but also hitting WebSocket endpoints. For SOCKS5, make sure to enable the socks feature in Cargo.toml (already mentioned above). Here’s an example of connecting:

use reqwest::{Client, Proxy};

async fn build_socks5_client(
    host: &str,
    login: &str,
    password: &str,
) -> reqwest::Result<Client> {
    let proxy_url = format!("socks5h://{host}");
    let proxy = Proxy::all(&proxy_url)?.basic_auth(login, password);

    Client::builder()
        .proxy(proxy)
        .timeout(std::time::Duration::from_secs(20))
        .build()
}

Note the scheme socks5h instead of socks5. The letter "h" at the end means that DNS resolution of the domain occurs on the proxy server side, not locally. This is critical for anonymity: without it, your real DNS resolver may "leak" requested domains past the proxy, even if the traffic itself goes through a SOCKS5 tunnel.

If the proxy provider does not require authentication via username/password (IP-whitelisting), simply do not call basic_auth — specify your server's IP address in the provider's panel from which requests are made.

reqwest client pool for different proxies

A common mistake is to create a new Client for each request. This is costly: reqwest internally maintains a pool of TCP connections and TLS sessions, and recreating the client kills all the advantages of keep-alive. The correct approach is to create one client for each proxy in advance and reuse them from a common pool.

use reqwest::{Client, Proxy};
use std::sync::Arc;

#[derive(Clone)]
struct ProxyEndpoint {
    url: String,
    login: String,
    password: String,
}

struct ClientPool {
    clients: Vec<Client>,
}

impl ClientPool {
    fn build(endpoints: &[ProxyEndpoint]) -> reqwest::Result<Self> {
        let mut clients = Vec::with_capacity(endpoints.len());

        for ep in endpoints {
            let proxy = Proxy::all(&ep.url)?
                .basic_auth(&ep.login, &ep.password);

            let client = Client::builder()
                .proxy(proxy)
                .pool_max_idle_per_host(4)
                .timeout(std::time::Duration::from_secs(15))
                .build()?;

            clients.push(client);
        }

        Ok(Self { clients })
    }
}

Such a pool is created once when the parser starts. The number of clients usually equals the number of unique proxy addresses in your pool — if the provider gives access to rotation at the gateway level, one or two clients are sufficient, as IP rotation occurs automatically on the provider's side with each new TCP connection.

For tasks with high parallelism, wrap the pool in Arc and share it among tokio tasks — reqwest clients are thread-safe and designed for concurrent use.

IP rotation strategies

There are two fundamentally different approaches to rotation. The first is rotation on the provider's side: you connect to one gateway address, and the provider automatically substitutes a new IP for each request or every N minutes. The second is rotation on your side: you maintain a list of static proxies and choose which one to use for the next request.

If you are using residential proxies with automatic rotation through the gateway, the code simplifies to a minimum — you just send requests through the same client, and the provider changes the outgoing IP by itself. But if you need control (for example, to bind an IP to a session for 10 minutes to simulate real user behavior), implement rotation manually:

use rand::seq::SliceRandom;
use std::sync::atomic::{AtomicUsize, Ordering};

struct RotatingPool {
    clients: Vec<Client>,
    counter: AtomicUsize,
}

impl RotatingPool {
    // Round-robin: iterating through clients in turn
    fn next_round_robin(&self) -> &Client {
        let idx = self.counter.fetch_add(1, Ordering::SeqCst) % self.clients.len();
        &self.clients[idx]
    }

    // Random selection — useful against pattern analysis
    fn next_random(&self) -> &Client {
        self.clients.choose(&mut rand::thread_rng()).unwrap()
    }
}

Round-robin is suitable when even load distribution across all IPs is important. Random selection better masks the pattern of requests if the target site analyzes the intervals between requests from a single address. For tasks like monitoring prices on Ozon or collecting ads from Avito, a combination is practically more effective: random proxy selection plus a random delay of 1–3 seconds between requests from the same client.

It is worth mentioning sticky sessions separately. If the parser goes through a multi-step scenario (login → navigating pages → action), the IP cannot be changed in the middle of the session — the site will notice the IP change during an active cookie session and either log out or block the account. For such scenarios, bind one client (and accordingly one IP) to a specific "session" of the parser for its entire lifetime.

Error handling and retry logic

Proxies occasionally fail: timeouts, 407 (Proxy Authentication Required), TCP connection drops. The parser must be able to distinguish between "proxy is broken — exclude from the pool" and "site returned 429 — just wait and change IP." Here’s a working pattern with retry and excluding dead proxies:

use reqwest::StatusCode;
use std::time::Duration;

async fn fetch_with_retry(
    pool: &RotatingPool,
    url: &str,
    max_attempts: u32,
) -> Result<String, String> {
    let mut last_error = String::new();

    for attempt in 0..max_attempts {
        let client = pool.next_random();

        match client.get(url).send().await {
            Ok(resp) if resp.status() == StatusCode::TOO_MANY_REQUESTS => {
                last_error = "429: rate limit, changing IP".into();
                tokio::time::sleep(Duration::from_millis(500 * (attempt as u64 + 1))).await;
                continue;
            }
            Ok(resp) if resp.status().is_success() => {
                return resp.text().await.map_err(|e| e.to_string());
            }
            Ok(resp) => {
                last_error = format!("Unexpected status: {}", resp.status());
            }
            Err(e) if e.is_timeout() || e.is_connect() => {
                last_error = format!("Proxy issue: {e}");
                continue;
            }
            Err(e) => {
                last_error = e.to_string();
            }
        }
    }

    Err(format!("All attempts exhausted: {last_error}"))
}

An important nuance: exponential backoff (500 * attempt) reduces the load on the target site during mass 429 responses and simultaneously decreases the risk of completely blocking the entire proxy pool at once. For production parsers, add an error counter for each proxy in the pool and automatically exclude the address if it gives 5+ errors in a row — this will protect against futile attempts through an already banned IP.

Which type of proxy to choose for parsing

The choice of proxy type directly affects how many successful requests you will get before being blocked. Datacenter IPs are the cheapest and fastest, but belong to well-known ranges of hosting providers — anti-bot systems of marketplaces and social networks detect them almost instantly based on ASN databases. Residential and mobile IPs look like ordinary users, so they last much longer.

Proxy Type	Speed	Risk of Blocking	When to Use
Datacenter	High	High	Open APIs, sites without anti-bot
Residential	Medium	Low	Wildberries, Ozon, Avito, social networks
Mobile	Medium	Minimal	Instagram, TikTok, strict anti-bots

For parsing marketplaces at large volumes, a reasonable balance is residential proxies: they are not as expensive in terms of traffic as mobile ones, but pass checks at the level of "real user," which cut off datacenter ranges. If the parser works through social media APIs or simulates a mobile application, mobile proxies provide the lowest percentage of captchas and temporary bans due to IPs assigned to telecom operators.

Performance and load tips

Rust allows you to maintain thousands of concurrent requests on a single instance, but the proxy provider and target site still limit the actual throughput. Here are some practical rules for a production parser:

Limit parallelism using tokio::sync::Semaphore — no more than 20-50 simultaneous requests per proxy pool, otherwise some connections will start dropping due to timeout.
Use pool_idle_timeout in ClientBuilder to prevent reqwest from holding dead connections through unstable proxies.
Log HTTP statuses and response times for each proxy separately — this way you will quickly see which addresses have started to "decay" and need replacement.
Rotating User-Agent along with IP rotation significantly reduces the likelihood of detection: change them synchronously, not in separate cycles.
For long-lived parsers, add a health-check: every few minutes, send a test request to https://httpbin.org/ip through each proxy and exclude those that do not respond.

Pay special attention to TLS fingerprints (JA3/JA3S). reqwest by default uses rustls or nativetls depending on the build features, and both have different TLS handshake fingerprints. Some anti-bot systems (for example, at major marketplaces) analyze the TLS fingerprint, not just the IP — so even a perfect proxy will not save you if the client itself presents a non-standard TLS handshake. If you encounter blocks with "clean" proxies, check reqwest features and try the native-tls feature instead of rustls, or vice versa.

Conclusion

Proxies in Rust reqwest are set up quickly — just a couple of lines with Proxy::all or SOCKS5 scheme socks5h://. The main work begins afterward: building a client pool, choosing a rotation strategy (round-robin, random or sticky sessions) and proper error handling with the exclusion of non-working addresses from the pool. Without this, even the fastest parser in Rust will hit blocks after just a few minutes of operation.

If your parser works with marketplaces or social networks where anti-bot protection analyzes not only the frequency of requests but also the origin of the IP, start with residential proxies — they provide the best balance of price and session longevity. For stricter targets like Instagram and TikTok, you should test mobile IPs, while for simple tasks without anti-bot, fast and inexpensive datacenter proxies will suffice.