Proxy Setup in Go (Golang): http.Client, SOCKS5, and IP Rotation for Web Scraping and Automation

```html

If you are writing a parser, automating data collection, or testing APIs from different regions, sooner or later you will encounter IP blocks. Go is one of the most popular languages for writing high-performance network tools, and setting up proxies here has its peculiarities. In this guide, we will cover everything from the basic configuration of http.Client to advanced IP rotation with SOCKS5 support.

Why use proxies in Go applications

Go has become the de facto standard for writing high-load network tools: parsers, crawlers, price monitoring bots, API automation tools. This is where the need for proxies arises most often — and it is crucial to set everything up correctly from the start.

The main scenarios where proxies are indispensable:

• Parsing marketplaces — Wildberries, Ozon, AliExpress block IPs after 50–200 requests from a single address. Without changing the IP, the parser stops.
• Geo-testing — checking search engine results, advertisements, or prices from different countries and regions.
• Bypassing rate-limits — distributing the load across multiple IPs allows bypassing restrictions on the number of requests per minute.
• Anonymity — hiding the real IP of the server when working with external APIs.
• Testing CDN and geolocation — checking the correctness of content delivery for different regions.

The Go standard library provides flexible tools for working with proxies — through http.Transport and http.Client. Let's break down each method in detail.

Basic setup of HTTP proxy via http.Client

In Go, proxies are configured at the http.Transport level — the transport layer of the HTTP client. This gives you full control over which proxy requests go through and allows you to create multiple clients with different proxies simultaneously.

A minimal working example of setting up an HTTP proxy:

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
)

func main() {
    // Specify the proxy server address
    proxyURL, err := url.Parse("http://123.45.67.89:8080")
    if err != nil {
        panic(err)
    }

    // Create transport with proxy
    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
    }

    // Create client with our transport
    client := &http.Client{
        Transport: transport,
    }

    // Make a request through the proxy
    resp, err := client.Get("https://httpbin.org/ip")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Println(string(body))
}

What happens in this code:

1. url.Parse — parses the proxy address string into a *url.URL structure
2. http.Transport{Proxy: http.ProxyURL(proxyURL)} — creates a transport that directs all requests through the specified proxy
3. http.Client{Transport: transport} — creates a client with our transport

An important point: if you use http.DefaultClient (that is, just http.Get() without an explicit client), the proxy will not be applied. Always create your own http.Client to work with proxies.

Proxy via environment variables

Go supports standard environment variables for proxies — HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. The standard transport http.DefaultTransport reads them automatically via the http.ProxyFromEnvironment function.

Setting environment variables before running the application:

# In the terminal (Linux/macOS)
export HTTP_PROXY="http://123.45.67.89:8080"
export HTTPS_PROXY="http://123.45.67.89:8080"
export NO_PROXY="localhost,127.0.0.1"

# Run the application
go run main.go

If you want to use ProxyFromEnvironment in your own transport (for example, with additional TLS settings), do it like this:

transport := &http.Transport{
    Proxy: http.ProxyFromEnvironment,
    // Additional settings...
    TLSHandshakeTimeout:   10 * time.Second,
    ResponseHeaderTimeout: 30 * time.Second,
}

client := &http.Client{
    Transport: transport,
    Timeout:   60 * time.Second,
}

This approach is convenient for production environments, where the proxy address is set through deployment configuration (Docker, Kubernetes, systemd), rather than hardcoded in the code. It also simplifies changing proxies without recompiling the application.

Connecting SOCKS5 proxy in Golang

SOCKS5 is a more versatile protocol compared to HTTP proxies. It operates at a lower level and supports any TCP/UDP connections, not just HTTP/HTTPS. For parsers and automation, SOCKS5 is often preferable — especially when using residential proxies, which are most commonly provided via this protocol.

To work with SOCKS5 in Go, you need the package golang.org/x/net/proxy:

go get golang.org/x/net/proxy

Example of connecting a SOCKS5 proxy without authentication:

package main

import (
    "fmt"
    "io"
    "net/http"
    "golang.org/x/net/proxy"
)

func main() {
    // Create a SOCKS5 dialer
    dialer, err := proxy.SOCKS5("tcp", "123.45.67.89:1080", nil, proxy.Direct)
    if err != nil {
        panic(err)
    }

    // Create transport with custom dialer
    transport := &http.Transport{
        Dial: dialer.Dial,
    }

    client := &http.Client{
        Transport: transport,
    }

    resp, err := client.Get("https://httpbin.org/ip")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Println(string(body))
}

Starting from Go 1.20+, you can also use built-in SOCKS5 support via url.Parse with the prefix socks5://:

proxyURL, _ := url.Parse("socks5://123.45.67.89:1080")

transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
}

client := &http.Client{Transport: transport}

This syntax is simpler and does not require additional dependencies. However, for more complex scenarios (e.g., SOCKS5 with authentication in older versions of Go), the package golang.org/x/net/proxy remains relevant.

Authentication: username and password in proxy

Most commercial proxy providers use username and password authentication. There are two ways to pass credentials in Go.

Method 1: Embed in URL

// Format: protocol://username:password@host:port
proxyURL, err := url.Parse("http://myuser:[email protected]:8080")
if err != nil {
    panic(err)
}

transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
}

client := &http.Client{
    Transport: transport,
    Timeout:   30 * time.Second,
}

Method 2: Through url.URL structure (more secure)

proxyURL := &url.URL{
    Scheme: "http",
    Host:   "proxy.example.com:8080",
    User:   url.UserPassword("myuser", "mypassword"),
}

transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
}

client := &http.Client{
    Transport: transport,
    Timeout:   30 * time.Second,
}

The second method is preferable — it correctly handles special characters in the password (e.g., @, #, :), which can break URL parsing in the first variant.

For SOCKS5 with authentication using the package golang.org/x/net/proxy:

auth := &proxy.Auth{
    User:     "myuser",
    Password: "mypassword",
}

dialer, err := proxy.SOCKS5("tcp", "proxy.example.com:1080", auth, proxy.Direct)
if err != nil {
    panic(err)
}

transport := &http.Transport{
    Dial: dialer.Dial,
}

client := &http.Client{
    Transport: transport,
    Timeout:   30 * time.Second,
}

Proxy rotation: switching IPs between requests

One proxy is good for testing, but for real parsing, IP rotation is needed. Websites track the frequency of requests from a single address, and even a good proxy will be blocked if you send thousands of requests through it in a row.

There are two approaches to rotation in Go:

Approach 1: Proxy list with round-robin

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "sync/atomic"
    "time"
)

type ProxyRotator struct {
    proxies []*url.URL
    counter uint64
}

func NewProxyRotator(proxyURLs []string) (*ProxyRotator, error) {
    proxies := make([]*url.URL, 0, len(proxyURLs))
    for _, p := range proxyURLs {
        u, err := url.Parse(p)
        if err != nil {
            return nil, err
        }
        proxies = append(proxies, u)
    }
    return &ProxyRotator{proxies: proxies}, nil
}

// GetClient returns a client with the next proxy in the round
func (r *ProxyRotator) GetClient() *http.Client {
    idx := atomic.AddUint64(&r.counter, 1) % uint64(len(r.proxies))
    proxyURL := r.proxies[idx]

    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
    }

    return &http.Client{
        Transport: transport,
        Timeout:   30 * time.Second,
    }
}

func main() {
    proxyList := []string{
        "http://user:[email protected]:8080",
        "http://user:[email protected]:8080",
        "http://user:[email protected]:8080",
    }

    rotator, err := NewProxyRotator(proxyList)
    if err != nil {
        panic(err)
    }

    // Each request goes through the next proxy
    for i := 0; i < 10; i++ {
        client := rotator.GetClient()
        resp, err := client.Get("https://httpbin.org/ip")
        if err != nil {
            fmt.Printf("Request error %d: %v\n", i, err)
            continue
        }
        body, _ := io.ReadAll(resp.Body)
        resp.Body.Close()
        fmt.Printf("Request %d: %s\n", i, string(body))
    }
}

Approach 2: Dynamic proxy selection via function

The Proxy field in http.Transport accepts a function with the signature func(*http.Request) (*url.URL, error). This allows you to select proxies dynamically for each request:

proxies := []*url.URL{
    mustParseURL("http://proxy1.example.com:8080"),
    mustParseURL("http://proxy2.example.com:8080"),
    mustParseURL("http://proxy3.example.com:8080"),
}

var counter uint64

transport := &http.Transport{
    Proxy: func(req *http.Request) (*url.URL, error) {
        // Select proxy based on the request (round-robin)
        idx := atomic.AddUint64(&counter, 1) % uint64(len(proxies))
        return proxies[idx], nil
    },
}

client := &http.Client{
    Transport: transport,
    Timeout:   30 * time.Second,
}

func mustParseURL(s string) *url.URL {
    u, err := url.Parse(s)
    if err != nil {
        panic(err)
    }
    return u
}

The second approach is more elegant — one client automatically rotates proxies for each request. This is especially convenient for concurrent requests via goroutines.

Which type of proxy to choose for a Go project

The choice of proxy type directly affects the success of your project. Different tasks require different types — let's go through the main options:

For parsing marketplaces (Wildberries, Ozon, Avito): use residential proxies. These platforms actively use bot detection systems (Cloudflare, proprietary solutions) and easily block data center IPs. Residential IPs look like regular home users.

For API testing and load testing: data center proxies are the optimal choice. High speed, stable connection, predictable latency.

For geo-testing: residential proxies with country/city selection. They allow you to check how your service or advertisement appears to users from a specific region.

Common mistakes and best practices

Over the years of working with proxies in Go projects, a list of typical mistakes has crystallized that slow down development and reduce the reliability of parsers. Let's go through each one.

Mistake 1: Reusing transport without resetting connections

http.Transport caches TCP connections (keep-alive). If you create a new transport for each request, you lose this advantage. But if you reuse one transport with a fixed proxy during rotation — requests may go through the old connection, bypassing the new proxy.

Solution: when changing proxies, call transport.CloseIdleConnections() or use a functional approach with dynamic proxy selection (described in the rotation section).

Mistake 2: No proxy error handling

Proxies can be unavailable, overloaded, or blocked by the target site. Without retry logic, your parser will stop at the first error.

func fetchWithRetry(client *http.Client, targetURL string, maxRetries int) ([]byte, error) {
    var lastErr error
    for i := 0; i < maxRetries; i++ {
        resp, err := client.Get(targetURL)
        if err != nil {
            lastErr = err
            time.Sleep(time.Duration(i+1) * time.Second) // Exponential backoff
            continue
        }
        defer resp.Body.Close()

        if resp.StatusCode == 429 || resp.StatusCode == 403 {
            lastErr = fmt.Errorf("blocked: status %d", resp.StatusCode)
            time.Sleep(time.Duration(i+1) * 2 * time.Second)
            continue
        }

        return io.ReadAll(resp.Body)
    }
    return nil, fmt.Errorf("all attempts exhausted: %w", lastErr)
}

Mistake 3: Ignoring TLS verification

Some developers disable TLS certificate verification for proxies (InsecureSkipVerify: true). This creates a vulnerability for MITM attacks. If the proxy server uses a self-signed certificate — add it to the trusted pool, rather than disabling verification entirely.

Mistake 4: Same headers for all requests

Even with proxy rotation, you can be blocked based on header patterns. Change User-Agent along with the proxy, and add realistic browser headers:

userAgents := []string{
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/118.0.0.0 Safari/537.36",
}

req, _ := http.NewRequest("GET", targetURL, nil)
req.Header.Set("User-Agent", userAgents[rand.Intn(len(userAgents))])
req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
req.Header.Set("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7")
req.Header.Set("Accept-Encoding", "gzip, deflate, br")

Checklist for a reliable proxy client in Go

• ✅ Timeout on the client (Timeout: 30 * time.Second)
• ✅ Retry logic with exponential backoff
• ✅ Proxy rotation on 403/429 errors
• ✅ Rotate User-Agent along with the proxy
• ✅ Log errors indicating the proxy used
• ✅ Credentials via environment variables, no hardcoding
• ✅ Closing resp.Body using defer
• ✅ Limit concurrency using a semaphore or worker pool

Example of a complete worker pool with proxy rotation

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "sync"
    "sync/atomic"
    "time"
)

type Scraper struct {
    proxies []*url.URL
    counter uint64
    workers int
}

func (s *Scraper) getClient() *http.Client {
    idx := atomic.AddUint64(&s.counter, 1) % uint64(len(s.proxies))
    return &http.Client{
        Transport: &http.Transport{
            Proxy: http.ProxyURL(s.proxies[idx]),
        },
        Timeout: 30 * time.Second,
    }
}

func (s *Scraper) Scrape(urls []string) map[string]string {
    results := make(map[string]string)
    var mu sync.Mutex
    var wg sync.WaitGroup

    sem := make(chan struct{}, s.workers) // Semaphore to limit concurrency

    for _, u := range urls {
        wg.Add(1)
        go func(targetURL string) {
            defer wg.Done()
            sem <- struct{}{}
            defer func() { <-sem }()

            client := s.getClient()
            resp, err := client.Get(targetURL)
            if err != nil {
                fmt.Printf("Error: %v\n", err)
                return
            }
            defer resp.Body.Close()

            body, _ := io.ReadAll(resp.Body)

            mu.Lock()
            results[targetURL] = string(body)
            mu.Unlock()
        }(u)
    }

    wg.Wait()
    return results
}

Conclusion

Setting up proxies in Go is a task that can be solved in a few lines of code, but it requires understanding the nuances: proper use of http.Transport, correct handling of authentication, smart IP rotation, and error handling. The standard library provides everything necessary for HTTP proxies, and the package golang.org/x/net/proxy fulfills the need for SOCKS5.

Key takeaways from the article:

• Always create your own http.Client — http.DefaultClient does not support proxies directly
• Use a functional approach (Proxy: func(r *Request) (*url.URL, error)) for dynamic rotation
• Store credentials in environment variables, not in code
• Add retry logic — proxies are not 100% reliable
• Change User-Agent along with the proxy for maximum effectiveness

If you are writing a parser for marketplaces like Wildberries or Ozon, we recommend using residential proxies — they have real home user IPs and are significantly less likely to be blocked by anti-bot systems. For high-load scenarios with thousands of requests per minute, data center proxies will be the optimal choice — they provide maximum speed and connection stability.