Asynchronous Proxies in httpx for Python: Rotation, HTTP/2, and 5 Common Beginner Mistakes

The httpx library has become the standard for asynchronous HTTP requests in Python, but when working with proxies, there are nuances that can break parsers in production. Let's explore how to set up asynchronous proxy rotation, enable HTTP/2, and why your script crashes every 50-100 requests.

Why httpx instead of requests or aiohttp

The requests library is synchronous and does not support HTTP/2 out of the box. aiohttp is asynchronous, but its proxy setup is less user-friendly, and it lacks HTTP/2 support altogether. httpx addresses both issues: it provides a synchronous and asynchronous client with a unified API, native HTTP/2 support via the h2 extension, and flexible proxy handling at the transport level.

For scraping tasks on marketplaces (Wildberries, Ozon, Avito), collecting data from social media APIs, or monitoring competitors' prices, asynchronicity is critical—it provides a significant speed boost compared to sequential requests. However, without proper proxy rotation and connection management, asynchronicity can lead to new problems: exhaustion of the connection pool, hanging coroutines, and file descriptor leaks.

Another important point is that HTTP/2 can multiplex multiple requests over a single TCP connection, reducing the load on the target server and helping to bypass some naive anti-bot systems that look at the number of TCP handshakes from a single IP.

Basic proxy setup in httpx

Starting from version httpx 0.26+, the syntax for setting up proxies has changed—rather than using a dictionary proxies, the proxy parameter is used. This is a common source of confusion if you are looking at old tutorials.

import httpx
import asyncio

PROXY_URL = "http://user:[email protected]:8000"

async def fetch(url: str):
    async with httpx.AsyncClient(proxy=PROXY_URL, timeout=15.0) as client:
        response = await client.get(url)
        return response.status_code, response.text[:200]

asyncio.run(fetch("https://httpbin.org/ip"))

If you need to separate proxies for HTTP and HTTPS traffic or set different proxies for different domains, use httpx.Client(mounts=...)—transports are mounted based on the URL pattern:

mounts = {
    "http://": httpx.AsyncHTTPTransport(proxy="http://user:[email protected]:8000"),
    "https://": httpx.AsyncHTTPTransport(proxy="http://user:[email protected]:8000"),
}

async with httpx.AsyncClient(mounts=mounts) as client:
    response = await client.get("https://example.com")

For authentication with a username and password, the format http://user:pass@host:port works reliably for HTTP proxies. For SOCKS5, you will need to additionally install httpx[socks], otherwise you will encounter an ImportError at runtime, not during coding—this is a trap that many fall into.

Asynchronous proxy rotation: working code

The main mistake in proxy rotation is creating a new AsyncClient for each request. This kills performance because the client maintains a pool of connections, and recreating it negates all the benefits of keep-alive connections. The correct approach is to create a pool of clients in advance, one for each proxy, and distribute tasks among them.

import httpx
import asyncio
import itertools
import random

PROXIES = [
    "http://user:[email protected]:8000",
    "http://user:[email protected]:8000",
    "http://user:[email protected]:8000",
]

class ProxyPool:
    def __init__(self, proxies: list[str]):
        self.clients = [
            httpx.AsyncClient(proxy=p, timeout=httpx.Timeout(15.0, connect=5.0))
            for p in proxies
        ]
        self._cycle = itertools.cycle(self.clients)

    def get_client(self) -> httpx.AsyncClient:
        return next(self._cycle)  # round-robin
        # for random selection: random.choice(self.clients)

    async def close_all(self):
        for client in self.clients:
            await client.aclose()

async def fetch_with_retry(pool: ProxyPool, url: str, retries: int = 3):
    for attempt in range(retries):
        client = pool.get_client()
        try:
            response = await client.get(url)
            if response.status_code == 200:
                return response.text
        except (httpx.ConnectTimeout, httpx.ProxyError, httpx.ReadTimeout):
            await asyncio.sleep(0.5 * (attempt + 1))
            continue
    return None

async def main():
    pool = ProxyPool(PROXIES)
    urls = [f"https://example.com/page/{i}" for i in range(100)]

    sem = asyncio.Semaphore(20)  # limit parallelism

    async def bound_fetch(url):
        async with sem:
            return await fetch_with_retry(pool, url)

    results = await asyncio.gather(*(bound_fetch(u) for u in urls))
    await pool.close_all()
    return results

asyncio.run(main())

This pattern addresses three problems at once: it reuses TCP connections through keep-alive, distributes the load across proxies in a round-robin manner, and limits parallelism through a semaphore to avoid overwhelming the target server and triggering a rate-limit ban.

For production parsers that work with Wildberries, Ozon, or Instagram API, it is important that the proxy pool is large enough and geographically diverse—otherwise, rotation within a single subnet will not protect against blocking. Residential proxies are well-suited here, as they have IPs of real users and are less likely to fall under the filters of anti-bot systems that analyze ASN and range reputation.

HTTP/2 in httpx: why and how to enable it

By default, httpx uses HTTP/1.1. To enable HTTP/2, you need to install the extension and explicitly pass the flag:

# pip install httpx[http2]

async with httpx.AsyncClient(http2=True, proxy=PROXY_URL) as client:
    response = await client.get("https://example.com")
    print(response.http_version)  # expecting "HTTP/2"

An important nuance: not all proxies support HTTP/2 tunneling correctly. Some cheap datacenter proxies only work as HTTP/1.1 proxies even if the target server supports HTTP/2—in such cases, httpx will quietly fall back to HTTP/1.1 via a CONNECT tunnel, and you may not notice this without explicitly checking response.http_version.

HTTP/2 provides real benefits in three scenarios: when you make many requests to a single domain (multiplexing in one connection reduces latency), when it is important to reduce the visibility of requests for anti-bot protection (fewer TCP handshakes from a single IP looks more natural), and when the target site itself requires HTTP/2 and returns errors or truncated content to clients on HTTP/1.1—this is sometimes encountered on Cloudflare sites.

For tasks where speed and stability of HTTP/2 connections through proxies are important, dedicated datacenter proxies are better suited—they are faster than residential ones and usually correctly support tunneling of modern protocols.

5 common mistakes with proxies in httpx

1. Recreating the client for each request

Code like async with httpx.AsyncClient(...) as client: await client.get(url) inside a loop is an anti-pattern. Each client creation results in a new TCP connect, a new TLS handshake, and complete disregard for connection pooling. For 1000 requests, the speed difference between the correct and incorrect approach can reach 5-10 times.

2. Missing timeouts or too large default timeouts

By default, httpx sets a 5-second timeout for everything at once—connect, read, write, pool. When working through unstable proxies, this is insufficient for connect but too much for read if the proxy is simply "hanging." Explicitly separate timeouts:

timeout = httpx.Timeout(
    connect=5.0,
    read=10.0,
    write=10.0,
    pool=5.0
)
client = httpx.AsyncClient(proxy=PROXY_URL, timeout=timeout)

3. Ignoring connection pool limits

With a large number of parallel tasks (gathering 500+ coroutines without a semaphore), httpx starts throwing PoolTimeout because the connection pool limit (default 100) has been exhausted. Explicitly set limits through httpx.Limits(max_connections=200, max_keepalive_connections=50) and always use a semaphore to control parallelism.

4. Disabling verify without understanding the consequences

Many copy verify=False from Stack Overflow to eliminate SSL errors when working through proxies with MITM certificates. This opens the door to man-in-the-middle attacks and can lead to data leaks through a compromised proxy. If a proxy requires a special certificate—add it explicitly via verify="/path/to/cert.pem", rather than disabling verification entirely.

5. Unclosed clients and file descriptor leaks

If you create a pool of clients manually (as in the example above), remember to call await client.aclose() for each of them when finished. Long-running scripts without explicit connection closures hit the limit of open file descriptors in the operating system (usually 1024 on Linux by default), and the parser crashes with the mysterious error Too many open files after a few hours of operation.

Which type of proxy to choose for scraping

The choice of proxy directly impacts how many errors from the list above you will actually encounter—unstable proxies provoke timeouts and connection drops, even if the code is perfectly written.

Proxy Type	Speed	Risk of Blocking	When to Use
Datacenter	High	High on secure sites	Open API, weak anti-bot protection
Residential	Medium	Low	Wildberries, Ozon, social media
Mobile	Medium-low	Minimal	Instagram, TikTok API, strict protection

For asynchronous scraping with high request speeds to APIs with moderate protection, datacenter proxies remain a cost-effective option—the key is to evenly distribute the load across the pool, as shown in the code above.

Checklist before production

Conclusion

httpx provides a powerful toolkit for asynchronous work with proxies and HTTP/2, but it requires careful configuration of the client pool, timeouts, and connection limits. Most parser crashes in production are related not to the library itself, but to architectural mistakes: recreating clients, lack of parallelism control, and improper handling of proxy exceptions.

If you are building a parser for marketplaces or social media APIs that actively monitor suspicious activity, the stability of the proxy infrastructure is just as important as the code. For such tasks, consider residential proxies—they reduce the incidence of ProxyError and ConnectTimeout due to higher reputation of IP addresses compared to cheap datacenter pools.