Proxy for Web Scraping: What Gets Blocked and What Actually Works
Your scraper handles the first few hundred pages, then the 403s start. The parser is fine. The request format is fine. What changed is how the target treats the IP your request left from.
A scraping proxy is an egress problem. The parser is rarely the part that broke. Which exit you choose, how you keep a session coherent, and what the per-GB bill looks like decide whether the crawl finishes.
What the target sees before it reads your request
Every request arrives with an IP, and that IP belongs to an Autonomous System. Comcast and Deutsche Telekom are eyeball networks. AWS, Hetzner, and DigitalOcean are hosting networks. That classification is public, and bot management stacks query it before they look at your cookies.
Alongside the ASN, the target can look at IP reputation, request rate, header order, TLS fingerprint, and whether the session makes sense. A proxy changes the first two signals. It does not repair the other four. That distinction matters because buying more IPs is the most common way to waste money on a scraper that has a fingerprint problem.
Choose the exit type before you choose a provider
There are three exit types worth comparing, and they trade trust against cost. Start with the cheapest one that clears the target, not with the most expensive one in the catalog.
| Exit type | What the target sees | Where it works | Cost shape |
|---|---|---|---|
| Datacenter | Hosting ASN | Low-protection pages, public APIs, internal mirrors | Lowest per GB or per IP |
| Residential | ISP-assigned home connection | Ecommerce, search results, sites with bot management | Per GB, higher than datacenter |
| Mobile | Carrier-grade NAT, shared by many phones | Endpoints that reject residential ranges too | Highest per GB, limited supply |
Moving everything to mobile is an easy way to turn a $50 crawl into a $500 one. Use residential where datacenter fails, and mobile only for the subset that fails residential.
Use a session, not a fresh IP per request
Two rotation strategies cover most scraping jobs. Sticky sessions hold one exit for a window, which keeps cookies, rate limits, and pagination coherent. Rotating sessions change the exit per request or per few requests, which spreads load across a pool.
Use sticky sessions for anything with state: logins, carts, paginated results, multi-step forms. Use rotation for broad crawls where each URL is independent. Rotating on every request inside a checkout flow is a reliable way to look like an account takeover.
For the mechanics behind that choice, including per-connection rotation and manual sticky rotation, see the rotating proxy guide.
With ProxyBase, country and session type are set when you open the session. They are not encoded in the SOCKS5 username, and the gateway does not parse suffix syntax. The username is the session ID and the password is your session token.
The warmup call is worth keeping in production. Log the exit IP and country on every session start, and stop if they do not match the session you requested. A wrong exit caught at startup costs one request. The same exit discovered halfway through a crawl costs you a partial dataset.
Set up SOCKS5 in Python and curl
In Python, requests needs PySocks for SOCKS proxies. Install it with pip install requests[socks], then pass the proxy URL in the same format shown above. Keep the socks5h:// prefix. Plain socks5:// resolves DNS on your machine, so your resolver sees every hostname even though the connection itself is proxied.
With curl, the equivalent flag is --socks5-hostname:
Browser automation is the exception
Playwright and Puppeteer drive Chromium, and most Selenium scraping setups do too. Chromium does not implement SOCKS5 username and password authentication. If you pass credentials in a SOCKS5 proxy URL, the browser will not send them. This is the same limitation that makes Chrome extensions and desktop clients use a local relay.
The workaround is a loopback HTTP-to-SOCKS5 bridge. The bridge listens on 127.0.0.1, accepts plain HTTP proxy requests from the browser, and performs the authenticated SOCKS5 handshake upstream. The browser only sees a local HTTP proxy.
Run one bridge per session. Sharing a single bridge across rotating sessions defeats the point, because every browser tab will exit from the same upstream.
For the Playwright, Puppeteer, and Selenium configuration details, see the browser automation proxy guide.
Do the per-GB math before you commit
Residential and mobile exits are billed by traffic. A typical HTML page is somewhere between 50 KB and 300 KB, so measure your own average instead of guessing. At 150 KB per page, 10,000 pages is about 1.5 GB. At $3 per GB, that crawl costs roughly $4.50.
The number that surprises people is retries. Failed requests, blocked responses, and assets fetched through the proxy all count. Cap retries at two or three attempts, and set the session spend cap so a loop cannot drain the balance overnight. ProxyBase lets you set spend_cap_microcredits when the session is created.
Five failure modes worth fixing before you buy more IPs
- DNS resolved locally. A plain
socks5://URL leaks every hostname to your resolver. Usesocks5h://. - Rotation on every request. New IPs mid-flow break cookies, carts, and pagination. Match the rotation strategy to the job.
- High concurrency on one exit. Twenty workers behind one IP look like an attack. Keep per-IP concurrency low or add exits.
- Blind retries on 403. Retrying the same request with the same fingerprint produces the same block and a larger bill. Change something or stop.
- No exit logging. If you cannot say which IP and country a failed request used, you cannot tell a proxy problem from a target-side rule.
What a proxy does not do
A proxy changes where a request comes from. It does not change what you are allowed to do. Respect the target's terms and robots rules, do not use a proxy to bypass authentication or a paywall, and do not collect personal data without a lawful basis. If the plan only works when the target believes you are someone else, a proxy is not the missing piece.
Frequently Asked Questions
Do I need residential proxies for web scraping?
Only when the target rejects datacenter IPs. Start with datacenter exits for low-protection pages and APIs. Move to residential for sites with bot management, and to mobile for the smaller set of endpoints that reject residential ranges as well.
How many proxy IPs do I need?
Concurrency drives the number, not the size of the site. One exit can handle a few simultaneous requests before rate limits or reputation checks kick in. If you need twenty parallel workers, plan for enough distinct exits to keep per-IP concurrency low.
What is the difference between socks5 and socks5h?
socks5 resolves the hostname on your machine and sends the resulting IP to the proxy. socks5h sends the hostname to the proxy and lets the exit node resolve it. Use socks5h for scraping so DNS resolution and connection origin stay on the same side.
Can Playwright or Puppeteer use an authenticated SOCKS5 proxy?
Not directly. Chromium does not implement SOCKS5 username and password authentication. Run a loopback HTTP-to-SOCKS5 bridge that holds the upstream credentials, then point the browser at the local HTTP proxy address.
How much does a scraping proxy cost?
Residential and mobile exits are usually billed per gigabyte. ProxyBase marketplace pricing starts at $3 per GB. Ten thousand pages at an average of 150 KB is about 1.5 GB, or roughly $4.50. Retries and failed requests consume bandwidth too.
Why does my scraper still get blocked with a proxy?
A new IP changes the ASN and reputation signal. It does not fix a stale TLS fingerprint, a missing header set, or a request rate that looks automated. Fix the request pattern first, then add exits.