Proxy Configuration for curl and wget: Practical Examples for System Administrators and DevOps

If you administer servers, write automation scripts, or deploy applications in a corporate infrastructure, sooner or later you will face the need to route curl or wget traffic through a proxy. This could be a corporate proxy, bypassing geo-blocks when downloading packages, or rotating IPs for bulk requests to external APIs. In this article, we focus solely on practical aspects: commands, configurations, and code examples without fluff.

1. How curl and wget work with proxies: basic mechanisms

Before diving into configurations, it’s important to understand what happens under the hood. Both tools support two main proxy protocols: HTTP/HTTPS and SOCKS5. The mechanics are different, and this affects which type of proxy to choose for a specific task.

HTTP proxy acts as a mediator at the application protocol level. When curl sends a request through an HTTP proxy, it literally tells the proxy server: "make a GET request to this URL instead of me." For HTTPS traffic, the CONNECT method is used — curl asks the proxy to establish a tunnel to the target host, after which the TLS handshake occurs directly between the client and the destination server. This is important: in this case, the proxy does not see the content of the HTTPS traffic.

SOCKS5 operates at a lower level — it proxies TCP/UDP connections without being tied to the application-level protocol. This makes SOCKS5 more versatile: it can handle not only HTTP/HTTPS but also other protocols. Additionally, SOCKS5 supports DNS resolution on the proxy server side — this is critically important to prevent DNS leaks.

curl has supported both protocols natively since very early versions. wget historically had more limited support for SOCKS5 — in older versions (up to 1.19), there is no SOCKS5 support at all, only HTTP proxies. This should be taken into account when writing scripts that need to work on different distributions.

You can check the version of wget with the command wget --version. For curl — curl --version, where you can also see which protocols the libcurl library was built with.

2. Environment variables: the quickest way to configure

The most elegant way to set up a proxy for curl and wget is through environment variables. This works system-wide: you don’t need to change each script, just set the variables once in the session or add them to the user profile.

Both tools read the same standard variables:

# For HTTP traffic
export http_proxy="http://proxy.example.com:3128"

# For HTTPS traffic
export https_proxy="http://proxy.example.com:3128"

# Duplicate in uppercase (some programs only read these)
export HTTP_PROXY="http://proxy.example.com:3128"
export HTTPS_PROXY="http://proxy.example.com:3128"

# For FTP (if needed)
export ftp_proxy="http://proxy.example.com:3128"

# Exceptions — addresses that DO NOT go through the proxy
export no_proxy="localhost,127.0.0.1,::1,192.168.0.0/16,.internal.company.com"

An important nuance: curl is case-sensitive regarding variables depending on the version. To avoid surprises — always set both versions: in lowercase and uppercase. This is standard practice in DevOps.

To make the settings persistently apply for a specific user, add the lines to ~/.bashrc or ~/.profile. For system scripts and services — in /etc/environment or in a systemd unit file via the Environment= directive.

# /etc/systemd/system/myservice.service
[Service]
Environment="http_proxy=http://proxy.example.com:3128"
Environment="https_proxy=http://proxy.example.com:3128"
Environment="no_proxy=localhost,127.0.0.1"
ExecStart=/usr/bin/myapp

If the proxy requires authentication, the username and password are inserted directly into the URL variable:

export http_proxy="http://username:[email protected]:3128"
export https_proxy="http://username:[email protected]:3128"

3. curl flags for working with proxies: a complete breakdown

curl provides a rich set of flags for managing proxies directly from the command line. This is convenient when you need to make a one-off request through a proxy without changing system settings.

The main flag is -x or --proxy:

# Basic HTTP proxy
curl -x http://proxy.example.com:3128 https://api.example.com/data

# Short form
curl -x proxy.example.com:3128 https://api.example.com/data

# Explicitly specifying the protocol
curl --proxy http://proxy.example.com:3128 https://api.example.com/data

# Check external IP through the proxy
curl -x http://proxy.example.com:3128 https://api.ipify.org

To ignore environment variables and connect directly (bypassing the configured proxy), the --noproxy flag is used:

# Ignore proxy for a specific host
curl --noproxy "internal.company.com" https://internal.company.com/api

# Ignore proxy completely (even if env variables are set)
curl --noproxy "*" https://api.example.com/data

Useful flags for debugging proxy connections:

# Verbose mode: shows all headers, including CONNECT to the proxy
curl -v -x http://proxy.example.com:3128 https://api.example.com/data

# Only response headers
curl -I -x http://proxy.example.com:3128 https://api.example.com/data

# Show connection time through the proxy
curl -w "Connect: %{time_connect}s\nTotal: %{time_total}s\n" \
  -x http://proxy.example.com:3128 \
  -o /dev/null -s https://api.example.com/data

Configuring the proxy through the config file ~/.curlrc is convenient if you don’t want to specify flags every time:

# ~/.curlrc
proxy = http://proxy.example.com:3128
proxy-user = username:password
noproxy = localhost,127.0.0.1,.internal.company.com

For system scripts, you can create a separate config and specify it explicitly with curl -K /path/to/config — this allows you to have different proxy profiles for different tasks.

4. Configuring proxy in wget: flags and config file

wget is less flexible than curl, but for typical tasks — downloading files, recursive mirroring of websites — its capabilities are quite sufficient. You can configure the proxy in wget in three ways: through environment variables (discussed above), through command-line flags, and through the config file ~/.wgetrc.

wget command-line flags for proxy:

# HTTP proxy via flag
wget -e use_proxy=yes \
     -e http_proxy=http://proxy.example.com:3128 \
     https://example.com/file.tar.gz

# HTTPS through proxy
wget -e use_proxy=yes \
     -e https_proxy=http://proxy.example.com:3128 \
     https://example.com/file.tar.gz

# With authentication
wget -e use_proxy=yes \
     -e http_proxy=http://username:[email protected]:3128 \
     https://example.com/file.tar.gz

# Disable proxy for a specific command (if env variables are set)
wget --no-proxy https://internal.company.com/file.tar.gz

The config file ~/.wgetrc is the preferred method for persistent configuration:

# ~/.wgetrc
use_proxy = on
http_proxy = http://proxy.example.com:3128
https_proxy = http://proxy.example.com:3128
ftp_proxy = http://proxy.example.com:3128
no_proxy = localhost,127.0.0.1,.internal.company.com

# If the proxy requires authentication
proxy_user = username
proxy_password = secretpassword

For system-wide application (all users), use /etc/wgetrc — the same format, but applied globally. This is convenient for servers where all download operations must go through the corporate proxy.

A practical example: recursive downloading of a website through a proxy with depth and speed limits:

wget -e use_proxy=yes \
     -e http_proxy=http://proxy.example.com:3128 \
     --recursive \
     --level=2 \
     --limit-rate=500k \
     --wait=1 \
     --random-wait \
     --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1)" \
     https://example.com/docs/

5. SOCKS5 proxy in curl and wget: setup and examples

SOCKS5 is a more preferred protocol for tasks where anonymity is important or when working with non-standard ports. For system administrators and DevOps engineers, SOCKS5 is often used when working through SSH tunnels, as well as when connecting to residential proxies, which simulate traffic from real users.

In curl, SOCKS5 is supported through a special prefix in the proxy URL:

# SOCKS5 with DNS resolution on the client side (may cause DNS leak!)
curl --socks5 proxy.example.com:1080 https://api.example.com/data

# SOCKS5 with DNS resolution on the proxy side (recommended!)
curl --socks5-hostname proxy.example.com:1080 https://api.example.com/data

# Via -x flag with explicit protocol
curl -x socks5h://proxy.example.com:1080 https://api.example.com/data

# With authentication
curl -x socks5h://username:[email protected]:1080 https://api.example.com/data

# SOCKS5 via environment variable
export all_proxy="socks5h://proxy.example.com:1080"
curl https://api.example.com/data

A popular scenario in DevOps is using SSH as a SOCKS5 proxy to tunnel traffic through a bastion host:

# Opening an SSH tunnel with SOCKS5 on local port 1080
ssh -D 1080 -f -C -q -N [email protected]

# Now using this tunnel in curl
curl -x socks5h://localhost:1080 https://internal-api.private.network/data

# Or via environment variable for all requests in the session
export all_proxy="socks5h://localhost:1080"
wget https://internal-resource.private.network/file.tar.gz

wget has supported SOCKS5 since version 1.19. In older versions (CentOS 7, Ubuntu 16.04), you will have to use workarounds: proxychains, tsocks, or switch to curl for tasks requiring SOCKS5.

6. Proxy authentication: username and password

Most commercial proxies and corporate proxy servers require authentication. There are several ways to pass credentials — each with its own pros and cons in terms of security.

Method 1: Credentials in URL — simple but insecure (password visible in the process list):

curl -x http://user:p%[email protected]:3128 https://api.example.com/data
# Special characters in the password need to be URL-encoded: @ → %40, : → %3A

Method 2: --proxy-user flag in curl — you can omit the password in the command line, curl will prompt for it interactively:

# Username and password in the flag (still visible in ps aux)
curl -x http://proxy.example.com:3128 \
     --proxy-user "username:password" \
     https://api.example.com/data

# Only username — curl will ask for the password interactively
curl -x http://proxy.example.com:3128 \
     --proxy-user "username" \
     https://api.example.com/data

Method 3: Through .netrc file — the most secure for scripts:

# ~/.netrc
machine proxy.example.com
login username
password secretpassword

# Restrict file access permissions
chmod 600 ~/.netrc

# Use in curl
curl -x http://proxy.example.com:3128 --proxy-netrc https://api.example.com/data

Method 4: Through environment variables from a secret store — recommended for CI/CD and production environments:

# In the script, read credentials from variables (set by CI/CD)
#!/bin/bash
PROXY_URL="http://${PROXY_USER}:${PROXY_PASS}@proxy.example.com:3128"
curl -x "${PROXY_URL}" https://api.example.com/data

Comparison table of authentication methods:

7. Exclusions and no_proxy: how to bypass proxy for local addresses

In corporate and cloud environments, fine-tuning is often required: external traffic goes through the proxy, internal traffic goes directly. The no_proxy (or NO_PROXY) variable allows you to specify a list of exceptions.

# Basic exceptions
export no_proxy="localhost,127.0.0.1,::1"

# Domain exception (with a dot — all subdomains)
export no_proxy="localhost,127.0.0.1,.internal.company.com,.corp.local"

# IP range exception (CIDR notation may not work everywhere!)
export no_proxy="localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

# For AWS: exclude metadata endpoint and internal addresses
export no_proxy="localhost,127.0.0.1,169.254.169.254,.amazonaws.com.internal"

A practical example for a Kubernetes environment where you need to exclude the API server and internal services:

export no_proxy="localhost,127.0.0.1,10.96.0.0/12,10.244.0.0/16,.cluster.local,.svc,.default"
export NO_PROXY="${no_proxy}"

8. Proxies in CI/CD: GitHub Actions, GitLab CI, Docker

Setting up proxies in CI/CD pipelines is one of the most common tasks for DevOps engineers working in corporate networks or with limited internet access. Let’s discuss specific configurations for popular platforms.

GitHub Actions

# .github/workflows/deploy.yml
name: Deploy

on: [push]

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      http_proxy: ${{ secrets.PROXY_URL }}
      https_proxy: ${{ secrets.PROXY_URL }}
      no_proxy: "localhost,127.0.0.1,.github.com"
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Download dependencies
        run: |
          curl -x "${http_proxy}" https://external-api.example.com/config.json -o config.json
          wget -e use_proxy=yes -e http_proxy="${http_proxy}" https://releases.example.com/app-v1.0.tar.gz

GitLab CI

# .gitlab-ci.yml
variables:
  http_proxy: "http://proxy.company.com:3128"
  https_proxy: "http://proxy.company.com:3128"
  no_proxy: "localhost,127.0.0.1,.gitlab.company.com"

build:
  stage: build
  script:
    - curl -x "${http_proxy}" https://registry.npmjs.org/package -o package.json
    - wget -e use_proxy=yes -e http_proxy="${http_proxy}" https://example.com/resource.tar.gz

Docker: proxy during image builds

# Passing proxy through build args
docker build \
  --build-arg http_proxy=http://proxy.example.com:3128 \
  --build-arg https_proxy=http://proxy.example.com:3128 \
  --build-arg no_proxy=localhost,127.0.0.1 \
  -t myapp:latest .

# In Dockerfile, use ARG to get variables

# Dockerfile
FROM ubuntu:22.04

ARG http_proxy
ARG https_proxy
ARG no_proxy

ENV http_proxy=${http_proxy}
ENV https_proxy=${https_proxy}
ENV no_proxy=${no_proxy}

RUN apt-get update && apt-get install -y curl wget

RUN curl -x "${http_proxy}" https://example.com/setup.sh | bash

# Clear proxy variables in the final image (optional)
ENV http_proxy=""
ENV https_proxy=""

Global proxy settings for Docker daemon (for docker pull through proxy):

# /etc/systemd/system/docker.service.d/proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:3128"
Environment="HTTPS_PROXY=http://proxy.example.com:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.internal.registry.com"

# Restart docker
systemctl daemon-reload
systemctl restart docker

9. Proxy rotation in bash scripts

If you need to make a large number of requests to external APIs or services, proxy rotation allows you to distribute the load and avoid IP blocks. This is especially relevant for price monitoring, data collection, or testing the availability of resources from different regions.

For such tasks, data center proxies are well-suited — they provide high speed and stability for bulk requests.

#!/bin/bash
# rotate_proxy.sh — rotating proxies from a list

PROXY_LIST=(
  "http://user:[email protected]:3128"
  "http://user:[email protected]:3128"
  "http://user:[email protected]:3128"
  "http://user:[email protected]:3128"
)

URLS_FILE="urls.txt"
OUTPUT_DIR="./results"
mkdir -p "${OUTPUT_DIR}"

PROXY_COUNT=${#PROXY_LIST[@]}
INDEX=0

while IFS= read -r url; do
  PROXY="${PROXY_LIST[$INDEX]}"
  FILENAME=$(echo "${url}" | md5sum | cut -d' ' -f1)
  
  echo "Requesting: ${url} via proxy $((INDEX + 1))/${PROXY_COUNT}"
  
  curl -x "${PROXY}" \
       --max-time 30 \
       --retry 3 \
       --retry-delay 2 \
       --silent \
       --output "${OUTPUT_DIR}/${FILENAME}.html" \
       "${url}"
  
  if [ $? -eq 0 ]; then
    echo "  ✓ Success"
  else
    echo "  ✗ Failed, trying next proxy..."
    INDEX=$(( (INDEX + 1) % PROXY_COUNT ))
    curl -x "${PROXY_LIST[$INDEX]}" \
         --max-time 30 \
         --silent \
         --output "${OUTPUT_DIR}/${FILENAME}.html" \
         "${url}"
  fi
  
  # Move to the next proxy
  INDEX=$(( (INDEX + 1) % PROXY_COUNT ))
  
  # Small pause between requests
  sleep 0.5
  
done < "${URLS_FILE}"

echo "Done! Results saved to ${OUTPUT_DIR}"

A more advanced option is to check the availability of the proxies before using them:

#!/bin/bash
# check_proxy.sh — checking proxy availability

check_proxy() {
  local proxy_url="$1"
  local test_url="https://api.ipify.org"
  
  result=$(curl -x "${proxy_url}" \
                --max-time 10 \
                --silent \
                --write-out "%{http_code}" \
                --output /dev/null \
                "${test_url}")
  
  if [ "${result}" -eq 200 ]; then
    echo "ALIVE"
  else
    echo "DEAD"
  fi
}

# Get external IP through the proxy
get_proxy_ip() {
  local proxy_url="$1"
  curl -x "${proxy_url}" --max-time 10 --silent https://api.ipify.org
}

PROXY="http://user:[email protected]:3128"
STATUS=$(check_proxy "${PROXY}")
echo "Proxy status: ${STATUS}"

if [ "${STATUS}" == "ALIVE" ]; then
  EXTERNAL_IP=$(get_proxy_ip "${PROXY}")
  echo "External IP via proxy: ${EXTERNAL_IP}"
fi

10. Debugging and common errors

Working with proxies inevitably comes with errors — especially during the initial setup. Let’s discuss the most common problems and how to diagnose them.

Diagnosis using verbose mode

# Most detailed output from curl
curl -vvv -x http://proxy.example.com:3128 https://api.example.com/data 2>&1 | head -50

# What to look for in the output:
# * Connected to proxy.example.com — connection to the proxy established
# CONNECT api.example.com:443 — request for tunnel
# HTTP/1.1 200 Connection established — tunnel opened
# * SSL connection using TLS — TLS is working

# Debugging wget
wget -d -e use_proxy=yes -e http_proxy=http://proxy.example.com:3128 https://api.example.com/data

Common errors and solutions