Bot Blocks, CAPTCHAs, and Other Scraping “WTF” Moments: A Plain-English Glossary for Data Collectors

If you scrape the web, you know the feeling. Your script runs fine for five minutes. Then the site throws a 403, a CAPTCHA, or a blank page that looks “fine” in a browser.

NetLingo fans love quick defs, so let’s treat scraping pain the same way. This guide explains the terms you see in logs and vendor docs. It also tells you what to do next, without turning your app into a science fair.

One stat sets the mood. Imperva reported that automated traffic made up 49.6% of all internet traffic in 2023. Sites assume “bot” first, even when you act like a polite user.

Why sites fight scrapers (even the nice ones)

Sites defend uptime, ad views, and user data. They also protect prices, stock, and content from copycats. Your scraper may look like an attack, even when you just track a few SKUs.

Many teams run a WAF and bot rules by default. They tune those rules for risk, not for your use case. That mismatch causes most “but I’m harmless” blocks.

Log-speak you will meet on day one

403 Forbidden

A 403 means the site heard you and said “no.” Your IP, headers, or path tripped a rule. A WAF often makes that call in under a second.

Fixes start with basics. Match real browser headers. Keep cookies per session. Slow your hit rate before you buy more IPs.

429 Too Many Requests

A 429 means you sent too many hits in a short span. Some sites set a hard cap per IP. Others watch patterns across many IPs.

Retry with backoff. Add jitter so you avoid a neat, clock-like beat. Cache pages you already pulled, so you stop “re-asking” the same thing.

CAPTCHA

A CAPTCHA asks for human proof. Sites show it when they doubt your client. They doubt you when they spot odd mouse flow, fast page turns, or reused device traits.

CAPTCHAs cost time and money. Treat them as a signal that your client looks wrong. Do not treat them as a core workflow.

Fingerprint

A fingerprint groups signals like TLS traits, fonts, screen size, and JS quirks. A site uses it to link “new” visits to old ones. That link can beat IP rotation.

Keep your client stable per session. Do not mix device traits across many IPs. Your stack should act like one user at a time.

WAF (Web Application Firewall)

A WAF blocks bad input and bad patterns. It also blocks good input that looks risky. If you see sudden blocks across many pages, assume a WAF rule fired.

Use clean URLs. Avoid odd query spam. Send sane Accept-Language and Accept-Encoding headers.

Proxy talk, minus the hype

Datacenter proxy

Datacenter IPs come from cloud hosts. They cost less and run fast. Many sites label them as “non-user” and watch them harder.

They still work for low-risk tasks. Think public pages with light rules. They often fail on login, carts, and price pages that drive revenue.

Residential proxy

Residential IPs come from consumer ISPs. Sites often trust them more. That trust helps on pages that hate bots.

Some jobs need the same IP for days. A static residential proxy. gives you that steady “home” feel while you keep one session alive.

Rotation, pool size, and session length

Rotation means you swap IPs on a timer or per request. A pool means you spread load across many IPs. Session length means how long you keep one identity.

Match these to the site’s rules. Fast rotation can look odd on checkout flows. Long sessions can raise risk if you hit pages too fast.

Build a scraper that looks normal (and stays cheap)

Rate limits that act human

Most teams start by adding more IPs. Start with pace instead. Send fewer hits per minute, and your block rate often drops fast.

Use per-host caps. Many scrapers flood one domain while other tasks sit idle. Spread work across time, not just across IPs.

Cache like you pay for bandwidth

Caching cuts cost and cuts risk. It also helps you prove you act in good faith. You can re-use past pages for tests, QA, and replays.

Store raw HTML and key fields. Track fetch time and status code. When a site changes layout, you can debug without re-scraping.

Detect soft blocks

Some sites return a 200 and still block you. They may serve a “verify you are human” page with a normal status. Your parser then reads junk and you miss it.

Add content checks. Look for known block text, odd titles, or missing page anchors. Treat those as errors, not as data.

The rules part: stay out of trouble

Scraping raises legal and trust issues. Read the site’s terms and respect access rules. Treat robots.txt as a clear signal, even when it lacks teeth.

Never pull data you do not need. Avoid personal data unless you have a clear right and a clear use. If you store user data, lock it down and set short retention.

When you work with vendors, ask hard questions. Ask where the IPs come from and how they handle consent. Your brand owns the risk, even when your proxy bill goes to someone else.