How to Prevent AI Companies from Stealing Your Data

Every time an AI training crawler visits your website, it extracts your content and feeds it into a commercial model you did not authorize, are not compensated for, and cannot control. The model that results from your content may compete with you, reproduce your work, or be sold to your competitors. And the legal system has not caught up — class action lawsuits by authors against AI companies (documented in the SMU Law Review, 2026) highlight that "authors need a solution to ensure their content is not exploited by AI companies," while copyright scholars note that "data scraping is considered an economic right of the rightholder" and scraping without consent is "deemed unlawful" under existing copyright frameworks in multiple jurisdictions.

But blocking all AI traffic is the wrong answer. AI search engines that cite your content and send visitors your way are useful. AI assistants browsing your pages at a user's direction are a new kind of visitor you may want to serve. The problem is not AI traffic — it is unauthorized training scraping — and the solution is separating the categories and controlling each one independently.

What AI Companies Are Taking

The scope of AI training scraping is vast and largely undisclosed. Research on identifying AI web scrapers using canary tokens (Kornblum et al., 2026) demonstrated that scrapers feed language models that "are not publicly known or disclosed by the companies," and that voluntary disclosure by scraping companies is "neither reliable nor scalable." The study used canary tokens embedded in website content to trace which scrapers fed which language models, revealing a scraping ecosystem that extends well beyond the bots that honestly identify themselves in their User-Agent strings.

A 2025 legal analysis of algorithmic scraping and copyright protection (SSRN, 2025) examined the fair use boundaries for text and data mining in AI training, while a study on foundation model training as data compression (Hoffmann et al., 2024) argued that model weights are "a compressed representation of the training data" and may constitute "a derivative work of a potentially protected set of works." In other words: your content does not just pass through the model during training. It becomes part of the model's weights, and those weights are a commercial product.

Lawsuits have followed. Authors have filed class actions against OpenAI and Perplexity AI for unauthorized use of their works (SMU Law Review, 2026). A study of the lawfulness of generative AI training (2024) examined these cases and concluded that data scraping for commercial AI training without the rightholder's consent is unlawful under copyright frameworks in both the US and other jurisdictions. The legal battle is ongoing, but the practical reality is clear: by the time a lawsuit is filed, your content is already in the model, and getting it out is, at best, a partial and retroactive remedy.

Why robots.txt Cannot Solve This

The traditional mechanism for controlling crawler access is the Robots Exclusion Protocol — a text file that tells crawlers which paths they may visit. It relies on voluntary compliance. AI crawlers that respect robots.txt and identify themselves honestly can be blocked with a Disallow rule. But the canary token research showed that the scrapers doing the most undisclosed extraction — the ones feeding models you have never heard of — do not identify themselves and do not respect robots.txt. They forge User-Agent strings, route through proxies, and behave like ordinary browsers to evade detection.

A robots.txt file cannot stop a scraper that does not read it. And as we documented in our AI bot scraping article, the distinction between a training crawler and an AI search engine is defined by intent, not by a User-Agent string that can be changed in one line of code. robots.txt is a request, not a control. IPrating is a control.

The Three Categories You Need to Separate

IPrating classifies AI traffic into three distinct categories, each with its own policy control. This is the separation that lets you block training scrapers while allowing AI search and user-directed agents:

AI Crawlers — training scrapers operated by AI companies to extract content for model building. These are the bots that take your pages and turn them into commercial model weights. Set AI Crawlers to Block, and every training scraper identified by the bot catalog is filtered before it extracts a single page. Undisclosed scrapers that forge their User-Agent are caught by the network-layer analysis — the same detection that catches antidetect browsers, because a scraping tool pretending to be Chrome does not produce the network-stack behavior that real Chrome produces. As we showed in our antidetect browser article, the network layer is the one the attacker cannot forge.

AI Search — bots operated by AI-powered search engines that index your content to answer user queries with citations and links back to your site. When an AI search engine cites your page, that citation can drive a real visitor to your site. This is closer to traditional search engine traffic than to training scraping. Set AI Search to Allow if you want the citation and referral value, or to Challenge if you want to gate it behind a confirmation step.

AI User / Assistant — traffic generated by AI assistants when a real human directs the tool to browse your page to answer a question or complete a task. This is a human using an AI tool to access your content, analogous to a visitor using a screen reader or a translation service. Set AI User / Assistant to Allow if you want human-directed AI browsing to work, or to Challenge if you want to confirm a human is in the loop.

The configuration for a publisher who wants to stop training scraping while keeping AI search and assistant traffic is three dropdowns: AI Crawlers to Block, AI Search to Allow, AI User / Assistant to Allow. The same four-action system — Allow, Block, Challenge, Redirect — that we described in our VPN configuration article.

Why Network-Layer Detection Catches What robots.txt Misses

The training scrapers that defeat robots.txt and User-Agent detection share one structural weakness: they connect through infrastructure that is not residential, or they route through proxies to disguise their origin. IPrating's proxy detection — Datacenter Proxy, Public Proxy, Residential Proxy — catches the network layer that scrapers cannot forge. A training crawler on a datacenter IP is flagged as a datacenter proxy. A training crawler on a residential proxy network is flagged by the cross-layer correlation that compares the proxy suspicion against the browser's network-stack behavior.

Set the relevant proxy dropdowns to Block or Challenge, and the scraping traffic that evades robots.txt is caught by the network it arrives on — not by a User-Agent string it can change, but by the packets its operating system sends before the browser is even involved.

The Analytics That Confirm It Worked

Every AI visitor that IPrating classifies generates an analytics event carrying the operator name, the category, and the action taken. You can see, in your dashboard, exactly how many pages AI crawlers attempted to scrape (and were blocked), how many referrals AI search sent your way, and how many AI assistant sessions occurred — broken down by operator. This is the evidence you need to confirm that your content is protected and that the AI traffic you chose to allow is delivering value. See our pricing for plans that scale with your traffic volume.

Sources

• SMU Law Review (2026). The Fall of Z-Library: Protection for Authors Against AI Companies. (Class action lawsuits against AI companies; "authors need a solution to ensure their content is not exploited.")
• Kornblum, J. et al. (2026). Identifying AI Web Scrapers Using Canary Tokens. arXiv:2605.13706. (Undisclosed scrapers feeding LLMs; voluntary disclosure unreliable.)
• SSRN (2025). Algorithmic Scraping vs. Copyright Protection: Defining Fair Use Boundaries for Text and Data Mining in AI Training. (Fair use boundaries for AI training scraping.)
• Hoffmann, J. et al. (2024). Training Foundation Models as Data Compression. arXiv:2407.13493. (Model weights as compressed representation of training data; derivative work implications.)
• Pratama, D. (2024). The Lawfulness of Using Inventions for Generative AI Training. (Lawsuits against OpenAI and Perplexity AI; scraping without consent deemed unlawful under copyright law.)