How to Block Scraping Bots — Stopping Every Type of Automated Data Extraction

Web scraping used to require skill. A scraper developer needed to understand HTML parsing, session management, authentication circumvention, and anti-bot evasion — knowledge that kept large-scale scraping in the hands of a small group of technical operators. That barrier has collapsed. A 2026 academic study, "Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users" (Stolee et al., 2026), demonstrated that large language models have "democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts." Their benchmark across 35 sites — including sites with authentication, anti-bot, and CAPTCHA controls — showed that end-to-end AI agents required "as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows."

In other words: anyone who can type a sentence can now scrape your website. The tools do the rest. And the volume of scraping traffic reflects it — research on AI web scraper identification (Kornblum et al., 2026) found scrapers feeding language models that "are not publicly known or disclosed by the companies," meaning the scraping landscape extends well beyond the bots that honestly identify themselves.

This article covers the different types of scraping bots, what the 2026 research says about each, and how IPrating's per-category filtering stops them — not with a single blunt block, but with separate controls for each type of scraper, so you can protect your content without cutting off traffic you want to keep.

The Types of Scraping Bots

Not all scrapers are the same, and the right response is different for each type. IPrating classifies scraping traffic into distinct categories, each with its own action control:

Automation Tools — general-purpose scraping and automation bots. These include commercial scraping frameworks, headless browsers, and custom scripts designed to extract content at scale. The 2026 LLM scraping benchmark showed that AI-assisted scripting and autonomous agents have made these tools accessible to non-technical users, capable of navigating complex sites with anti-bot controls. Set the Automation Tools category to Block and the scraper traffic that identifies itself through the bot catalog or that fails the browser reality checks is filtered before it reaches your content.

AI Crawlers — bots operated by AI companies to scrape content for model training. These are the extractors that take your pages and feed them into commercial language models you do not control and are not compensated for. As we documented in our AI bot scraping article, these crawlers can be allowed or blocked per category — but the default for most publishers should be Block for training crawlers, because the content they extract becomes a competitor's product.

Archivers — bots that crawl and store web content for archival purposes. Some publishers want to allow archival services; others consider any bulk extraction of their content unacceptable. The Archivers dropdown lets you decide per your policy.

Monitors — uptime and content monitoring services that periodically check your site. These are often legitimate — your own monitoring, or a partner's — but they can also be scrapers disguising themselves as monitors. The Monitors dropdown lets you allow monitoring you trust and block the rest.

Security — security scanners and vulnerability research tools that crawl your site looking for exposed content, misconfigurations, or attack surfaces. Some are operated by legitimate security researchers; others are reconnaissance for attacks. The Security dropdown lets you control these individually.

Academic — research crawlers operated by universities and academic institutions. These are typically low-volume and well-identified, but they still extract content. The Academic dropdown lets you decide whether to support research scraping on your terms.

Why User-Agent Detection Alone Fails

The traditional approach to blocking scrapers is to check the User-Agent string against a list of known scraper signatures. This worked when scrapers identified themselves honestly. It does not work anymore. The 2026 LLM scraping benchmark explicitly tested sites with "anti-bot and CAPTCHA controls" and found that autonomous agents could complete scraping workflows across them. The canary token study (Kornblum et al., 2026) demonstrated that voluntary disclosure by scraping companies is "neither reliable nor scalable," and that undisclosed scrapers — those that forge or omit their User-Agent — are the ones doing the most extraction.

A scraper that wants to evade User-Agent detection changes one string. A scraper that wants to evade browser fingerprinting uses an antidetect browser. But a scraper cannot change the behavior of the operating system's network stack, and it cannot forge the TLS handshake's protocol characteristics. This is the same principle we described in our antidetect browser article: the network layer is the layer the attacker cannot win, because it is determined by the kernel, not by the browser or the script.

How IPrating Stops Each Type

IPrating does not rely on User-Agent strings or browser fingerprints alone. It classifies every visitor using the multi-layer detection we described in our real-time detection article — network fingerprints, browser proof of execution, TLS behavior, and the curated bot catalog — and separates scrapers into the categories above, each with its own action dropdown.

For a publisher who wants to block all scraping while keeping search engines and legitimate visitors, the configuration is:

• Automation Tools — set to Block. General-purpose scrapers and automation bots are filtered by the bot catalog and by the browser reality checks that catch headless browsers and scripting tools pretending to be real browsers.
• AI Crawlers — set to Block. Training crawlers from known AI companies are identified by the catalog; undisclosed scrapers are caught by the network-layer analysis.
• Archivers, Security, Academic — set to Block or Allow per your policy. Each category is controlled independently.
• Monitors — set to Allow if you run monitoring services, or Block if you want to stop all automated periodic access.

For scrapers that do not match any built-in category — a proprietary tool, a new entrant, a scraper using a novel User-Agent — the custom signatures system handles the long tail. Add a signature with a regex pattern and a Block action, and the scraper is caught on every subsequent request, labeled in your analytics with the name you gave it.

The Network Layer Catches What User-Agent Misses

The scrapers that defeat User-Agent detection and browser fingerprinting — the AI-assisted agents, the antidetect browser deployments, the undisclosed crawlers — all share one characteristic: they connect through infrastructure that is not residential, or they route through proxies to disguise their origin. IPrating's proxy detection — Datacenter Proxy, Public Proxy, Residential Proxy — catches the network layer that scrapers cannot forge. A scraper on a datacenter IP is flagged as a datacenter proxy. A scraper on a residential proxy network is flagged by the cross-layer correlation that compares the proxy suspicion against the browser's network-stack behavior. Set the relevant proxy dropdowns to Block or Challenge, and the scraping traffic that evades User-Agent checks is caught by the network it arrives on.

This is the combination that makes the difference: per-category bot controls for scrapers that identify themselves, and network-layer detection for scrapers that do not. See our pricing for plans that scale with your traffic volume.

Sources

• Stolee, N. et al. (2026). Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users. arXiv:2601.06301. (LLMs democratized web scraping; autonomous agents complete workflows with "a single prompt" across sites with anti-bot controls.)
• Kornblum, J. et al. (2026). Identifying AI Web Scrapers Using Canary Tokens. arXiv:2605.13706. (Voluntary disclosure unreliable; undisclosed scrapers feeding LLMs.)
• Salganik, M. et al. (2024). Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations. arXiv:2410.23432. (Platforms restricting data access as AI scraping increased.)
• Liu, Z., Shrestha, P., & Saxena, N. (2021). Gummy Browsers: Targeted Browser Spoofing against State-of-the-Art Fingerprinting Techniques. arXiv:2110.10129. (Browser-layer detection unreliable; network layer as the detection vector.)