Forum and Classifieds Scraping: How to Collect Data Without Blocks Using Proxies

Data collection from forums and classifieds is a critically important task for marketers, market analysts, and business owners. Parsing Avito for monitoring competitor prices, collecting contacts from industry forums, analyzing reviews on specialized platforms — all these tasks face one problem: websites actively block automated data collection. In this article, we will discuss how to set up stable parsing through proxies and avoid bans.

Why forums and classifieds block parsing

Website owners protect their data for several reasons. Firstly, mass parsing creates a load on servers — one parser can generate thousands of requests per hour, which is equivalent to hundreds of users visiting the site simultaneously. Secondly, the collected data is often used by competitors: prices from Avito end up in monitoring systems, contacts from forums are added to cold sales databases.

Modern protection systems analyze many parameters: request frequency from one IP, behavior patterns (the parser opens pages too quickly and sequentially), browser headers, and the presence of JavaScript. For example, Avito uses multi-layered protection: User-Agent checks, cookie analysis, browser fingerprinting, and CAPTCHAs during suspicious activity.

Typical signs that will reveal you:

• One IP address — if all requests come from one IP, it leads to an instant ban
• High request frequency — an ordinary user cannot open 10 pages per second
• Absence of cookies and JavaScript — simple scripts do not execute JS and do not save cookies
• Suspicious User-Agent — old versions of browsers or mismatched headers
• Sequential page navigation — parsing strictly in order (page 1, 2, 3...) looks unnatural

Which proxies are suitable for parsing forums

The choice of proxy type depends on the volume of data, budget, and the level of protection of the target website. Let's consider three main options and their applications for parsing.

Data center proxies — the cheapest option, suitable for simple tasks. If you need to scrape a small thematic forum or a classifieds site without serious protection, this will suffice. The speed allows processing tens of thousands of pages per hour. However, Avito, YouDo, forum.ru, and other large platforms will quickly detect such IPs and block them.

Residential proxies — the optimal balance of price and quality for most tasks. These are real IPs from home users, which websites cannot distinguish from regular visitors. For parsing Avito, Yandex.Services, and large forums, this is the standard choice. An important point: residential proxies are usually sold with traffic payment, so optimize your requests — do not load unnecessary images and scripts.

Mobile proxies — maximum reliability for complex cases. IPs from mobile operators (MTS, Beeline, Megafon) have the highest trust level, as one IP can represent thousands of real users (CGNAT technology). Use them for sites with strict protection or when you need to collect critically important data without the risk of bans.

Parsing Avito: features and setup

Avito is one of the most protected platforms in the Russian Internet. The anti-parsing system includes JavaScript checks, browser fingerprinting, behavior analysis, and CAPTCHAs at the slightest suspicion. A simple script using requests will not work — you will receive an empty page or a CAPTCHA on the third request.

What you need for stable parsing of Avito:

A typical task is monitoring competitor prices. You need to collect ads in your category every day and track changes. For a category with 500-1000 ads, about 50-100 requests will be needed (considering pagination and product cards). With the right setup, this will take 10-15 minutes and 1-2 GB of traffic from residential proxies.

Step-by-step setup of the parser for Avito:

1. Get proxies — order a pool of residential IPs with rotation. For daily monitoring of one category, 10-20 GB of traffic per month will be sufficient.
2. Set up the headless browser — use Selenium or Puppeteer. Important: enable headless mode, but add parameters to bypass detection (window.navigator.webdriver = false).
3. Configure proxies in the browser — pass the proxy data when launching the browser. For Selenium, this is the --proxy-server parameter, for Puppeteer — args in puppeteer.launch().
4. Add realistic behavior — random delays of 3-7 seconds, scrolling the page before data collection, mouse movement (for Selenium).
5. Save cookies — after the first visit, save cookies and use them in subsequent sessions. This reduces suspicion.
6. Change IP regularly — rotation every 5-10 minutes or every 20-30 requests. Do not use one IP for the entire parsing.

A critical mistake beginners make is parsing too quickly. Even with proxies, if you open pages every second, the system will detect the bot by behavior patterns. An ordinary user reads an ad for 10-30 seconds, scrolls down, and returns to the search. Your parser should imitate this: delays, scrolling, and occasionally switching to neighboring categories.

Data collection from forums: strategies and tools

Forums vary in their level of protection. Old forums on phpBB or vBulletin usually do not have serious anti-bot protection — data center proxies and a simple parser are sufficient. Modern platforms (forum.ru, specialized industry forums) use Cloudflare or their own protection systems.

Typical tasks for parsing forums:

• Contact collection — emails, phone numbers, Telegram from user signatures and messages
• Monitoring brand mentions — tracking reviews about your company or competitors
• Sentiment analysis — collecting opinions about products, services, and trends in the industry
• Lead generation — people looking for solutions to your problem (for example, contractors are sought on construction forums)

For small forums (up to 10,000 pages), ready-made tools will suffice: Octoparse, ParseHub, WebHarvy. They have a visual interface — you simply click on the elements you need to collect, and the tool creates the parser. In the settings, you specify the proxies, delays, and start the collection.

For large projects (hundreds of thousands of pages), a custom parser is needed. Popular frameworks: Scrapy (Python), Puppeteer (JavaScript), Playwright (supports all languages). They allow for flexible configuration of crawling logic, error handling, and distributed parsing through a pool of proxies.

An important point: many forums require registration to view contacts or hidden sections. Create several accounts in advance (manually, from different IPs), maintain them for 1-2 weeks, and make several posts. Use these accounts for parsing — an authorized user raises less suspicion.

IP rotation and session management

Proper IP rotation is key to long-term stable parsing. There are two main approaches: time-based rotation and request-based rotation.

Time-based rotation: change IP every N minutes. Suitable for tasks where predictability is important. For example, if you parse Avito every 5 minutes changing IP — this guarantees that you will not exceed the request limit from one address. The downside: if the parser crashes or slows down, you waste IPs.

Request-based rotation: change IP every N requests (for example, every 20-50 pages). More efficient use of proxies but requires precise counting. If the site limits 100 requests from an IP per hour, set the rotation to 80 requests — leaving a buffer for errors.

Session management: when changing IP, decide whether to reset the session (cookies, localStorage) or keep it. For authorized parsing (forums, personal accounts), keep the session but change IP less frequently — otherwise, the site may suspect that the account has been hacked (logins from different cities). For public data (Avito without authorization), reset everything when changing IP — each IP looks like a new user.

An advanced technique is sticky sessions. Some proxy providers allow you to "stick" an IP for 10-30 minutes. You get one IP, make all requests from it within a logical task (for example, parsing one category of Avito), then switch to a new IP for the next category. This is more natural than changing IPs in the middle of browsing.

Setting up popular parsers for proxies

Let's consider setting up proxies in popular parsing tools. Examples for technical specialists who write their own parsers.

Scrapy (Python): add middleware for proxy rotation. Create a list of proxies in settings.py and use RandomProxy middleware for automatic rotation with each request.

# settings.py
ROTATING_PROXY_LIST = [
    'http://user:pass@proxy1.example.com:8000',
    'http://user:pass@proxy2.example.com:8000',
    'http://user:pass@proxy3.example.com:8000',
]

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}

Puppeteer (JavaScript): pass the proxy when launching the browser. For rotation, create a pool of proxies and choose randomly for each new browser launch.

const puppeteer = require('puppeteer');

const proxyList = [
  'proxy1.example.com:8000',
  'proxy2.example.com:8000'
];

const proxy = proxyList[Math.floor(Math.random() * proxyList.length)];

const browser = await puppeteer.launch({
  args: [
    `--proxy-server=${proxy}`,
    '--no-sandbox'
  ]
});

// Proxy authentication
const page = await browser.newPage();
await page.authenticate({
  username: 'user',
  password: 'pass'
});

Selenium (Python): configure the proxy through Chrome options. For HTTP authentication, use an extension or pass credentials in the URL.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--proxy-server=http://user:pass@proxy.example.com:8000')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-blink-features=AutomationControlled')

driver = webdriver.Chrome(options=chrome_options)
driver.get('https://www.avito.ru/moskva/kvartiry')

Ready-made parsers (Octoparse, ParseHub): in the task settings, find the "Proxy" or "IP Rotation" section. Add a list of proxies in the format host:port:user:pass or specify the API URL for rotation. Enable the "Rotate on each request" or "Rotate every N minutes" option.

Techniques for bypassing anti-bot protection

Proxies solve the problem of IP blocking, but modern protection systems analyze dozens of other parameters. Here is a comprehensive set of measures to bypass anti-bot systems.

1. Realistic User-Agent and headers: use current versions of browsers. Do not set a User-Agent from Chrome 90 if Chrome 120 has just been released. Check header consistency: if the User-Agent says "Windows" but the sec-ch-ua-platform header says "Linux" — you will be detected.

# Good set of headers for 2024
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7',
    'Accept-Encoding': 'gzip, deflate, br',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

2. Bypassing detection of headless browsers: Selenium and Puppeteer by default have signs of automation (navigator.webdriver property = true). Use stealth plugins or patches to hide these signs.

// Puppeteer Stealth Plugin
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({headless: true});

3. JavaScript Fingerprinting: websites collect browser fingerprints (canvas fingerprint, WebGL, fonts, screen resolution). To bypass, use randomization of these parameters or real browser profiles. Tools: FingerprintJS Randomizer, Multilogin (a platform with ready-made profiles).

4. CAPTCHA handling: if a CAPTCHA does appear, use recognition services: 2Captcha, Anti-Captcha, CapMonster. They cost $1-3 for 1000 CAPTCHAs. Integration via API takes 10-15 minutes. For reCAPTCHA v2/v3, there are ready-made libraries.

5. Behavioral patterns: add randomness to actions. Do not open pages strictly every 5 seconds — vary between 3 to 8 seconds. Occasionally take breaks of 30-60 seconds, simulating reading a long page. On forums, sometimes visit user profiles instead of just collecting topics.

Conclusion

Parsing forums and classifieds is a task that requires a comprehensive approach. Proxies solve the problem of IP blocking, but for stable operation, proper headers, realistic behavior, bypassing fingerprinting, and smart rotation are needed. The choice of proxy type depends on the level of protection of the target website: for simple forums, data centers are sufficient, while for Avito and large platforms, residential or mobile IPs are required.

Key principles for successful parsing: slow and natural, regular IP rotation, using headless browsers for complex sites, and handling CAPTCHAs when necessary. Do not chase speed — it is better to collect 500 pages per hour steadily for months than 5000 per hour and get banned in two days.

If you plan to parse Avito, YouDo, large forums, or platforms with serious protection, we recommend using residential proxies — they provide the optimal balance of reliability and cost. For particularly protected sites or collecting critically important data, mobile proxies with the highest trust level will suffice.