Safe Twitter/X Profile Scraping: How to Parse Without Bans Using Proxies

Twitter (now X) actively fights against automated data scraping: it blocks IP addresses, limits the number of requests, and bans accounts for suspicious activity. If you are collecting data for marketing research, monitoring brand mentions, or analyzing competitors, you need the right strategy for working with proxies and anti-detect tools.

In this guide, we will discuss how to set up safe scraping of Twitter/X, which types of proxies to choose for different tasks, and how to avoid bans during mass profile scraping.

Why Twitter/X blocks scraping and how it works

After rebranding to X, the platform has significantly tightened its policy regarding automated data collection. The protection system analyzes several parameters simultaneously, and blocking can occur even with manual work if you do not take precautions.

Main triggers for Twitter/X blocks

1. Exceeding rate limits. Twitter sets strict limits on the number of requests per minute. For unauthorized users, this is about 180 requests every 15 minutes, and for authorized users, up to 900 requests. If you exceed this, you will receive a 429 error (Too Many Requests), and with systematic violations, your IP address will be banned.

2. Suspicious behavior patterns. If you open profiles at a rate of 10 per minute, scroll pages at the same speed, or perform actions without pauses, the system will recognize a bot. A real user takes breaks, reads content, and sometimes gets distracted.

3. Using one IP for multiple accounts. If 5-10 different accounts log in from one IP address in a short period, it raises a red flag. Twitter may block all accounts in a chain (chain-ban).

4. Lack of a digital fingerprint. The platform collects data about the browser: version, extensions, screen resolution, time zone, WebGL, Canvas. If this data does not match the real device or is the same for multiple accounts, it raises suspicions.

How Twitter/X detects automation

The protection system uses multi-level analysis:

• User-Agent and header analysis. If the HTTP request headers do not match a real browser or contain traces of automation (e.g., Selenium, Puppeteer in User-Agent) — the request is blocked.
• JavaScript verification. Twitter actively uses JavaScript calls to check that a real browser is opening the page, not a simple HTTP client.
• Mouse and keyboard behavior analysis. The platform tracks cursor movements, scrolling speed, and click patterns. Bots usually move in straight lines or do not move the mouse at all.
• IP address reputation. If the IP is on a blacklist (data center of a known host, proxy provider with a bad reputation) — trust in it decreases.

Which proxies are suitable for scraping Twitter/X: a comparison of types

The choice of proxy type depends on your tasks: mass scraping of public data, working with authorized accounts, or monitoring competitors. Let's break down each type and its application for Twitter/X.

Residential proxies — the optimal choice for working with accounts

Residential proxies use IP addresses of real home users provided by Internet service providers. For Twitter/X, this is the safest option, as the platform cannot distinguish this traffic from that of a regular user.

When to use residential proxies for Twitter/X:

• Working with authorized accounts (login, scraping private profiles)
• Long-term monitoring of specific accounts or hashtags
• High-intensity scraping (when you need to minimize the risk of bans)
• Collecting data from different geographical regions (e.g., for analyzing regional trends)

Advantages: Maximum level of trust from Twitter/X, low percentage of bans, ability to work with CAPTCHAs (usually do not occur), support for sticky sessions (one IP for 10-30 minutes).

Disadvantages: Higher cost (usually payment for traffic, from $7-15 per 1 GB), speed is lower than that of data centers.

Mobile proxies — for maximum account protection

Mobile proxies use IP addresses from mobile operators (4G/5G). This is the most trusted type of IP for social networks, as Twitter/X very rarely blocks mobile IPs — thousands of real users can be behind one address.

When to use mobile proxies for Twitter/X:

• Working with valuable accounts that cannot be lost
• Scraping after previous bans (when maximum protection is needed)
• Automating actions: likes, retweets, follows (although this violates Twitter's ToS)
• Bypassing strict IP bans (mobile IPs almost never get blacklisted)

Advantages: Maximum level of trust, virtually zero risk of IP bans, ability to rotate IPs through "airplane mode" (changing IP every 5-10 minutes).

Disadvantages: The most expensive type of proxy (from $50-100 per IP per month), limited number of available IPs, speed depends on mobile network quality.

Data center proxies — for mass scraping of public data

Data center proxies are IP addresses from hosting provider servers. They are fast and cheap, but Twitter/X views them with suspicion.

When to use data centers for Twitter/X:

• Scraping public profiles without authorization (limited functionality)
• One-time data collection with low intensity
• Testing scraping scripts before launching on residential proxies
• Working through the official API (if you have paid access)

Advantages: Low cost (from $1-3 per IP per month), high speed (up to 1 Gbps), stable connection.

Disadvantages: High risk of bans, often requires CAPTCHA solving, not suitable for working with authorized accounts, many IPs are already blacklisted by Twitter/X.

Comparison table of proxy types for Twitter/X

Rate limits on Twitter/X: how not to exceed request limits

Twitter/X imposes strict limits on the number of requests to prevent server overload and automated data collection. If you exceed the limits, you will receive a temporary ban (from 15 minutes to several hours) or a permanent ban on your IP/account.

Current limits on Twitter/X (2024)

After the introduction of the X Premium subscription, the limits have been divided into several categories:

Important: These limits apply not only to the API but also to regular viewing through the web interface. If you are scraping through a browser while emulating user actions, these restrictions still apply.

How to bypass rate limits while scraping

1. Rotate IP addresses. Use a pool of proxies with automatic rotation. For residential proxies, the optimal frequency of change is every 50-100 requests or every 10-15 minutes. This allows you to distribute the load among different IPs and not exceed the limits on one address.

2. Use multiple accounts. If you need to collect a large amount of data, create 5-10 Twitter/X accounts and distribute the scraping among them. Each account should operate through its unique proxy and have a separate digital fingerprint.

3. Delays between requests. Do not make requests at maximum speed. Add random pauses:

• Between viewing profiles: 3-7 seconds
• Between scrolling the feed: 2-5 seconds
• Between searching by hashtags: 5-10 seconds
• Long pauses every 50-100 actions: 30-60 seconds

4. Cache data. Do not request the same data repeatedly. Store scraping results in a database and check if the profile has already been processed.

Setting up an anti-detect browser for safe scraping

Anti-detect browsers allow you to create unique digital fingerprints for each Twitter/X account, which is critically important for safe scraping. Without this, the platform can link multiple accounts together and block them in a chain.

Popular anti-detect browsers for Twitter/X

Dolphin Anty — one of the most popular options among arbitrageurs and SMM specialists. The free plan allows you to create up to 10 profiles, which is sufficient for small scraping tasks.

AdsPower — a good balance between functionality and price. It has built-in automation through RPA (Robotic Process Automation), allowing you to set up scraping without writing code.

Multilogin — a premium solution with the highest level of protection. Used by large agencies, but it is expensive (from €99/month). Justified only for professional scraping of large volumes of data.

GoLogin — a budget option with good fingerprint quality. There is a mobile app for working with accounts on the go.

Step-by-step setup of a profile for scraping Twitter/X (using Dolphin Anty as an example)

Step 1: Create a new browser profile

• Open Dolphin Anty and click "Create profile"
• Select the operating system: Windows, macOS, or Linux (choose the one that matches your real device or is most popular among Twitter users)
• Specify the profile name: for example, "Twitter Parser US #1"

Step 2: Configure the proxy

• In the "Proxy" section, select the type: HTTP, HTTPS, or SOCKS5 (SOCKS5 is preferable for Twitter/X)
• Enter the proxy data: IP address, port, username, password
• Click "Check proxy" — make sure the status is green and the geolocation is correct
• Important: use a separate proxy for each Twitter/X account

Step 3: Configure the fingerprint

• User-Agent: select a real User-Agent of the current version of Chrome (e.g., Chrome 120 on Windows 10)
• Screen resolution: use popular resolutions (1920x1080, 1366x768, 1440x900) — do not set exotic values
• Time zone: must match the geolocation of the proxy (if the proxy is from the USA/New York — set EST)
• Browser language: matches the region (en-US for the USA, en-GB for the UK)
• WebRTC: disable or replace with the proxy IP (otherwise, the real IP may leak)
• Canvas and WebGL: use "Noise" mode (adding noise) — this creates a unique fingerprint for each profile

Step 4: Additional security settings

• Disable password autofill (Twitter may check for saved data)
• Clear cookies after each scraping session
• Do not use browser extensions — they create unique fingerprints and may reveal automation
• Enable "Do Not Track" (DNT) — many real users use it

Tools for scraping Twitter/X: from ready-made solutions to code

The choice of tool depends on your technical skills and the volume of tasks. Let's explore options from simple no-code services to professional scripts.

Ready-made scraping services for Twitter/X (no code)

Phantombuster — a cloud automation service with ready-made templates for Twitter/X. Allows scraping profiles, followers, tweets by hashtags. Setup takes 5-10 minutes: choose a template, specify parameters (hashtag, list of accounts), connect a proxy, and launch.

Apify — a marketplace for ready-made scrapers. There are several actors (ready scripts) for Twitter/X: scraping profiles, collecting tweets, monitoring mentions. Works through the cloud, supports proxies, and has a free plan with limitations.

Octoparse — a visual parser builder. You open Twitter/X in the program's interface, click on the elements you need to collect (name, bio, number of followers), and the program automatically creates a script. Suitable for simple tasks but may struggle with complex page structures.

Automation through RPA in anti-detect browsers

Some anti-detect browsers (AdsPower, Octo Browser) have built-in RPA (Robotic Process Automation) tools that allow you to record and reproduce user actions.

How it works:

• Open Twitter/X in the anti-detect browser
• Enable action recording mode
• Perform the necessary actions: open a profile, copy data, move to the next
• Stop recording — the browser has created a script
• Upload a list of accounts for scraping and run the script in a loop

Advantages: does not require programming skills, works through a real browser (high level of bypassing protection), easy to add random delays and emulate behavior.

Scraping through code: Python + Selenium/Playwright

For more complex tasks and full control over the process, use programming. Python is the most popular language for scraping due to its rich ecosystem of libraries.

Main libraries:

• Selenium — browser automation, works with Chrome/Firefox, supports proxies and user action emulation
• Playwright — a modern alternative to Selenium, faster and more stable, supports headless mode
• Tweepy — a library for working with the official Twitter API (requires paid access)
• Twint — scraping Twitter without API (note: Twitter actively blocks this method, works unstably)

Example of a basic script in Python + Selenium:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
import random

# Proxy setup
proxy = "123.45.67.89:8080"  # Replace with your proxy
chrome_options = Options()
chrome_options.add_argument(f'--proxy-server={proxy}')
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])

# Initialize browser
driver = webdriver.Chrome(options=chrome_options)

# Open Twitter profile
driver.get('https://twitter.com/elonmusk')
time.sleep(random.uniform(3, 7))  # Random delay

# Scrape profile data
try:
    name = driver.find_element(By.XPATH, '//div[@data-testid="UserName"]').text
    bio = driver.find_element(By.XPATH, '//div[@data-testid="UserDescription"]').text
    followers = driver.find_element(By.XPATH, '//a[contains(@href, "/followers")]/span').text
    
    print(f"Name: {name}")
    print(f"Bio: {bio}")
    print(f"Followers: {followers}")
except Exception as e:
    print(f"Scraping error: {e}")

driver.quit()

Important points when working with code:

• Add time.sleep(random.uniform(3, 7)) between actions — this emulates real user behavior
• Use --disable-blink-features=AutomationControlled to hide signs of automation
• Change User-Agent to a real one: chrome_options.add_argument('user-agent=Mozilla/5.0...')
• Handle errors through try/except — Twitter/X often changes page structures
• Store results in a database (SQLite, PostgreSQL) or CSV file

Warming up Twitter/X accounts before mass scraping

If you are using new Twitter/X accounts for scraping, you cannot start mass data collection immediately. The platform tracks the "age" of the account and its activity — new accounts with aggressive behavior are blocked first.

Account warming plan (7-14 days)

Day 1-3: Basic setup and minimal activity

• Complete the profile: avatar, bio, website link (use real data, do not leave the profile empty)
• Follow 5-10 popular accounts in your niche
• Scroll the feed for 2-3 minutes, like 2-3 tweets
• Make 1-2 retweets
• Do not perform more than 10 actions per day

Day 4-7: Increasing activity

• Follow another 10-15 accounts
• Post 1-2 of your own tweets (simple ones like "Hello Twitter!")
• Increase the number of likes to 5-10 per day
• Start viewing other users' profiles (5-10 profiles per day)
• Add a few tweets to bookmarks

Day 8-14: Preparing for scraping

• Follow 20-30 accounts so that the total number of follows is 50-70
• Post 1 tweet every 2-3 days
• View 10-20 profiles per day
• Use hashtag search (but do not scrape yet — just search and read)
• On days 12-14, you can start light scraping: 20-30 profiles per day with pauses

Signs of a "warmed-up" account

• Account age: at least 14 days, preferably 30+ days
• Number of follows: 50-100 (not too many, not too few)
• Number of followers: 5-20 (even a few real followers increase trust)
• Tweet history: at least 5-10 posts
• Likes and retweets: 20-50 actions in history
• Completed profile: avatar, bio, possibly a link

Safe scraping checklist: 12 rules for protection against bans

Let's compile all the recommendations into a single checklist that should be followed when scraping Twitter/X: