How to Control AI Bot Scraping — Keep the Useful Traffic, Block the Rest

AI traffic to websites has grown explosively, and it has created a problem that traditional bot management was not built to solve: not all AI traffic is the same. An AI search engine that cites your content and sends visitors your way is useful. A training crawler that extracts your entire site to build someone else's commercial model is not. A research assistant browsing your pages to answer a user's question is a new kind of visitor you may want to serve. An undisclosed scraper that ignores your robots.txt and feeds your data to a model you have never heard of is stealing. Treating all AI traffic as a single block — allow all or block all — is the wrong answer, but it is the only answer most tools offer.

This article separates the categories of AI traffic, explains what the research says about the scraping problem, and shows how IPrating's per-category filtering lets you keep the useful AI visitors while blocking the extractors — with a dropdown for each.

The Three Categories of AI Traffic

IPrating classifies AI traffic into three distinct categories, each with its own policy control:

AI Crawlers are bots operated by AI companies to scrape web content for model training. They visit pages systematically, extract text and images, and feed the results into training pipelines for large language models and image generators. The content they take becomes part of a commercial product you are not paid for and do not control. This is the same pattern that ad fraud researchers have documented in a different domain — automated traffic extracting value from a site's content without delivering any in return. Academic research on generative AI training and copyright (Hug, 2025) argues that this practice "fundamentally differs" from the text-and-data-mining exceptions that AI companies rely on in Europe, and that training data memorization creates copyright issues independently of those exceptions.

AI Search bots are operated by AI-powered search engines that index your content to answer user queries with citations and links back to your site. When a user asks an AI search engine a question and the engine cites your page, that citation can drive a real visitor to your site. This is closer to traditional search engine traffic than to training scraping, and many publishers want to allow it.

AI User / Assistant traffic is generated by AI assistants — tools like ChatGPT, Gemini, and Copilot — when a user asks them to browse a page on your site to answer a question or complete a task. This is a real human directing an AI tool to your content, analogous to a visitor using a screen reader or a translation service. Whether you allow it, challenge it, or block it is a policy choice, but it is categorically different from a bot scraping your site over night to build a training corpus.

The distinction is important because the right answer is different for each. Blocking AI search means losing citations and referral traffic. Allowing AI crawlers means giving away your content for free to build a competitor's product. And the line between them is not something a simple robots.txt rule can draw — because the categories are defined by intent, not by a User-Agent string that can be changed in one line of code.

Why robots.txt Is Not Enough

The traditional approach to controlling bot access is the Robots Exclusion Protocol — a text file that tells crawlers which paths they may and may not visit. It works on the honor system: crawlers that respect robots.txt will obey; crawlers that do not will ignore it. Research on identifying AI web scrapers (Kornblum et al., 2026) demonstrated that existing mechanisms for identifying which scrapers feed which LLMs "rely on voluntary disclosure by companies, one-off experiments by researchers, or crowd-sourced reports — methods that are neither reliable nor scalable." The study used canary tokens to trace which scrapers fed which language models and found "several that are not publicly known or disclosed by the companies."

In other words: the AI scrapers that respect robots.txt and identify themselves honestly are the ones you can already control. The ones that do not — the undisclosed scrapers, the ones that forge User-Agent strings, the ones that route through residential proxies to look like real visitors — are the ones doing the most damage, and robots.txt cannot touch them. A 2025 legal-economics analysis (Lucchini Gupta et al.) noted that "legal uncertainty surrounds the large-scale scraping of copyrighted data for model training, with divergent rules across jurisdictions and limited transparency on how data is sourced."

How IPrating Separates and Controls AI Traffic

IPrating does not rely on robots.txt or User-Agent self-identification alone. It classifies every visitor in real time using the multi-layer detection we described in our real-time detection article — network fingerprints, browser proof of execution, TLS behavior, and the curated bot catalog — and it separates AI traffic into the three categories above, each with its own action dropdown.

For a publisher who wants to allow AI search but block training crawlers, the configuration is:

• AI Crawlers — set to Block. Training scrapers from known AI companies are identified by the bot catalog and blocked before they extract a single page. Undisclosed scrapers that forge their User-Agent are caught by the browser reality checks and network fingerprint analysis — the same detection that catches antidetect browsers, because a scraping tool pretending to be Chrome does not produce the network-stack behavior that real Chrome produces.
• AI Search — set to Allow. AI search engines that cite your content and send referral traffic are permitted through, and their visits appear in your analytics with the operator name and category so you can measure the value they deliver.
• AI User / Assistant — set to Allow or Challenge, depending on your policy. If you want human-directed AI browsing to work, allow it. If you want to gate it behind a challenge to confirm a human is in the loop, set it to Challenge and choose your challenge type.

For a publisher who wants to block all AI traffic — including search and assistants — set all three to Block. For a publisher who wants to allow everything and simply monitor, set all three to Allow and review the analytics to see which AI operators are visiting and how much bandwidth they consume. The point is that the choice is yours, per category, with the same four-action dropdowns: Allow, Block, Challenge, Redirect.

Custom Signatures for Undisclosed Scrapers

The bot catalog covers the known AI operators — the companies that identify their crawlers honestly. The long tail — undisclosed scrapers, new entrants, scrapers that forge User-Agent strings — is handled by the same custom signature system we described in our VPN configuration article. When you identify a scraper pattern that is not in the catalog, you add a custom signature with a regex match and a Block action. The scraper is caught on every subsequent request, and the match is labeled in your analytics with the name you gave it — so you can track how much traffic it was generating and confirm it stopped.

The Analytics You Need to Decide

Every AI visitor that IPrating classifies generates an analytics event carrying the operator name, the category, and the action taken. This means you can see, in your dashboard, exactly how much traffic came from each AI operator, broken down by category — how many pages AI crawlers attempted to scrape (and were blocked), how many referrals AI search sent your way, and how many AI assistant sessions occurred. This is the data you need to make an informed policy decision rather than a blanket guess, and it is available from the same platform that enforces the policy. See our pricing for plans that scale with your traffic volume.

Sources

• Hug, E. (2025). Generative AI Training and Copyright Law. arXiv:2502.15858. (Web scraping for AI training vs copyright; training data memorization issues; fair use vs TDM exception debate.)
• Kornblum, J. et al. (2026). Identifying AI Web Scrapers Using Canary Tokens. arXiv:2605.13706. (Voluntary disclosure unreliable; canary tokens identified undisclosed scrapers feeding LLMs.)
• Lucchini Gupta, F. et al. (2025). The Law and Economics of Generative AI and Copyright: A Primer to Core Challenges for Our Digital Future. (Legal uncertainty around large-scale scraping; divergent jurisdictional rules; limited transparency on data sourcing.)
• Salganik, M. et al. (2024). Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations. arXiv:2410.23432. (Platforms restricting data access as AI scraping increased; regulatory landscape.)