Google Colab and Jupyter Notebook are convenient tools for quickly writing and running scraping scripts without installing Python on a local machine. However, cloud notebooks have a peculiarity that is often overlooked: all requests come from the IP addresses of Google data centers, which are well-known to anti-bot systems and get banned almost instantly. In this article, we will discuss how to properly connect proxies in Colab and Jupyter, what limitations the cloud environment imposes, and how to structure scraping so that it doesn't fail after the first 50 requests.
Why Google Colab gets banned more often than a regular PC
The main problem with Colab for scraping is the shared pool of IP addresses. Every time you launch a notebook, Google allocates a virtual machine with an IP from its cloud ranges (AS15169 and similar). These ranges have long been blacklisted by most anti-bot systems: Cloudflare, DataDome, PerimeterX, as well as the built-in protection of Wildberries, Ozon, and marketplaces recognize them in a fraction of a second. The request doesn't even reach the site — it is filtered out by the ASN (Autonomous System Number) before checking the User-Agent or behavior.
The second reason is the mass usage. Thousands of developers simultaneously run scrapers, bots, and ML scripts through Colab, so the reputation of these IPs with many target sites is already "burned" long before your request. Even if you make one careful GET request every 10 seconds, the site may return a 403 simply because someone from the same range was hammering the server with 500 threads an hour ago.
The third reason is that Jupyter running locally usually does not have this problem (if you have a regular home or office IP), but as soon as you start working through JupyterHub on a hosting server or in a cloud provider's container, the situation becomes the same as in Colab. Therefore, the question of "whether proxies are needed or not" for cloud notebooks is answered unequivocally: without them, scraping practically does not work beyond the first few requests.
What type of proxy is suitable for cloud scraping
The choice of proxy type depends on what exactly you are scraping and how aggressive the target site's protection is. For Colab and Jupyter, it is important to consider one more nuance: the notebook session is time-limited, so long-lasting sticky proxy sessions are not always critical — more often, the stability of the connection within a single run is what matters.
| Proxy Type | When to Use in Colab/Jupyter | Features |
|---|---|---|
| Data Center Proxies | Scraping sites without strict protection, open APIs, static pages | High speed, low cost, but can also be banned by ASN on aggressive sites |
| Residential Proxies | Scraping Wildberries, Ozon, Avito, Google, social networks with anti-bot protection | IP of real users, rarely get blacklisted, higher latency |
| Mobile Proxies | Scraping mobile versions of sites, TikTok, Instagram through APIs | Highest trust, but more expensive and slower rotation |
For most scraping tasks involving marketplaces and search engines from Colab, the optimal balance will be a combination of residential proxies with rotation for each request — this reduces the likelihood of being banned due to behavioral patterns, even if the request itself is executed from Google's cloud environment.
Setting up proxies in requests and urllib
The basic proxy setup in Colab is no different from a local Python environment. The only difference is that in Colab, libraries usually need to be installed via !pip install, as the standard image does not always contain everything necessary.
!pip install requests
import requests
proxy = {
"http": "http://login:[email protected]:9000",
"https": "http://login:[email protected]:9000"
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
response = requests.get(
"https://example.com/catalog",
proxies=proxy,
headers=headers,
timeout=15
)
print(response.status_code)
print(response.text[:500])
An important point: there is no persistent configuration file in Colab, so the proxy login and password have to be either hardcoded (which is unsafe when publishing the notebook), or stored in Colab's environment variables via google.colab.userdata, or connected through Colab Secrets. This is more convenient than storing the password directly in the code, especially if you are sharing the notebook with colleagues.
from google.colab import userdata
proxy_login = userdata.get('PROXY_LOGIN')
proxy_pass = userdata.get('PROXY_PASS')
proxy_url = f"http://{proxy_login}:{proxy_pass}@gate.proxycove.com:9000"
proxies = {"http": proxy_url, "https": proxy_url}
Proxies with Selenium and headless Chrome in Colab
If the site renders content through JavaScript (a typical situation for Wildberries, Ozon, Avito), simple requests are not enough — a headless browser is needed. In Colab, this is usually Selenium with Chromium, which needs to be installed manually, as it is not included in the base image.
!apt-get update
!apt install chromium-chromedriver
!pip install selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--proxy-server=http://gate.proxycove.com:9000')
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://example.com/catalog")
print(driver.page_source[:500])
driver.quit()
The problem is that the --proxy-server flag does not support login and password authentication directly. For proxies with authentication in Selenium, you either need to use IP authentication (whitelisting by the static IP of the notebook, which is unstable in Colab since the machine's IP changes with each new launch), or generate a Chrome extension "on the fly" with the proxy credentials, or use a proxy tunnel through a library like selenium-wire, which supports authenticated proxies out of the box.
!pip install selenium-wire
from seleniumwire import webdriver
proxy_options = {
'proxy': {
'http': 'http://login:[email protected]:9000',
'https': 'http://login:[email protected]:9000',
}
}
driver = webdriver.Chrome(seleniumwire_options=proxy_options, options=chrome_options)
driver.get("https://example.com/catalog")
Jupyter Notebook: locally vs in the cloud
Jupyter Notebook running on a local machine (via Anaconda or directly through pip) uses your home or office IP address — and in this case, proxies are only needed for masking or bypassing geo-restrictions, not for avoiding bans due to the data center's ASN. The setup in this case is completely identical to a regular Python script: the same requests, the same Selenium, without limitations on installing drivers and system packages.
The situation changes if you use Jupyter through a cloud service: JupyterHub on a VPS, Kaggle Notebooks, Deepnote, Binder, or a corporate cloud cluster. In these cases, the IP again belongs to the provider's data center, and all Colab issues are relevant one-to-one. The main difference from Colab is that such services often have a more stable IP throughout the session (it does not change with each kernel restart), which simplifies IP whitelisting setup if your proxy provider supports it.
Practical takeaway: if scraping is critical for business (for example, daily monitoring of competitor prices on marketplaces), it is better not to rely on free cloud notebooks as a production environment, but to use them only for developing and testing the script, moving the final scraper to a server with a static IP and configured proxies.
Limitations of cloud notebooks that break scraping
In addition to issues with IP reputation, Google Colab has a number of technical limitations that directly affect the stability of scraping:
- Inactivity timeout. The free version of Colab disconnects the notebook after ~90 minutes of inactivity and forcibly ends the session after 12 hours of continuous operation — long scraping of thousands of pages needs to be broken into parts or intermediate results saved to Google Drive.
- IP change with each new runtime. If you disconnect and reconnect the runtime, the IP address of the virtual machine changes — this breaks any setup with IP whitelisting on the proxy service side.
- Limit on outgoing connections. Google limits the number of simultaneous network connections from one virtual machine, so aggressive multithreading with dozens of parallel requests may hit not the site's block, but Colab's own limits.
- No persistent storage without mounting a disk. Cookies, browser sessions, and proxy connection cache disappear with each restart — for scraping with authentication, cookies need to be explicitly saved to a file on Google Drive.
- Unpredictable network delays. Google's cloud infrastructure sometimes causes latency spikes, which means request timeouts need to be set with a buffer (15-30 seconds instead of the standard 5-10).
These limitations make Colab an excellent tool for developing and debugging scraper logic, but poorly suited for long-term production tasks — for those, it is better to move the finished script to a VPS or dedicated server with its own IP.
Proxy rotation and error handling in a loop
To prevent scraping from failing at the first block, the code needs retry logic with proxy replacement. Below is an example of a simple loop with rotation and error handling that can be run directly in Colab.
import requests
import time
import random
proxy_pool = [
"http://login:[email protected]:9000",
"http://login:[email protected]:9001",
"http://login:[email protected]:9002",
]
def get_with_retry(url, max_retries=5):
for attempt in range(max_retries):
proxy_url = random.choice(proxy_pool)
proxies = {"http": proxy_url, "https": proxy_url}
try:
response = requests.get(url, proxies=proxies, timeout=20)
if response.status_code == 200:
return response
else:
print(f"Attempt {attempt+1}: status {response.status_code}")
except requests.exceptions.RequestException as e:
print(f"Attempt {attempt+1}: error {e}")
time.sleep(random.uniform(2, 5))
return None
result = get_with_retry("https://example.com/catalog")
if result:
print(result.status_code)
else:
print("All attempts exhausted")
For scraping large catalogs (for example, thousands of product cards on Wildberries), this logic is additionally wrapped in a task queue with pauses between requests and saving intermediate results in CSV or a database directly in Google Drive to avoid losing progress during a Colab session interruption.
Checklist before launching the scraper in Colab
- Proxies are connected and tested with a separate test request before the main loop
- Proxy login and password are stored in Colab Secrets, not in open code
- Selenium is used for JS sites with selenium-wire or an explicitly defined authorization extension
- The code includes retry logic and proxy rotation for 403/429 errors
- Intermediate results are saved to Google Drive every N records
- Request timeouts are increased considering possible delays in the cloud network
- Request parallelism is limited to avoid hitting Colab's outgoing connection limits
- For production tasks, the script is planned to be moved to a separate server with a static IP
Conclusion
Google Colab and cloud Jupyter are great for quickly developing and testing scraper logic, but due to the shared IPs of data centers, they almost always require proxies from the very first steps. For simple sites without aggressive protection, data center proxies will suffice, while for scraping marketplaces, search engines, and social networks, residential and mobile IPs work more reliably, as they are less likely to fall under ASN filters.
If you plan to regularly monitor prices on Wildberries or Ozon directly from Colab, we recommend starting with residential proxies — they provide stable access without frequent blocks, while for one-off test scripts, more budget-friendly data center proxies can suffice, connecting them in the same way through requests or Selenium.