Identifying AI Web Scrapers Using Canary Tokens

• URL: http://arxiv.org/abs/2605.13706v1
• Date: Wed, 13 May 2026 15:53:57 GMT
• Title: Identifying AI Web Scrapers Using Canary Tokens
• Abstract summary: Large-scale web scraping to feed large language models (LLMs) can affect site stability and raise legal, privacy, or ethics concerns.<n>This paper proposes a novel technique for accurately and automatically inferring LLM-related scrapers.
• Score: 4.364248691212873
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: From pre-training to query-time augmentation, web-scraped data helps to improve the quality and contextual relevancy of content generated by large language models (LLMs). However, large-scale web scraping to feed LLMs can affect site stability and raise legal, privacy, or ethics concerns. If website owners wish to limit LLM-related web scraping on their site, due to these or other concerns, they may turn to scraper access control mechanisms like the Robots Exclusion Protocol. To be most effective, such mechanisms require site owners to first identify the scrapers that they wish to restrict (e.g., via User-Agent strings). Existing mechanisms to identify LLM-related scrapers rely on voluntary disclosure by companies, one-off experiments by researchers, or crowd-sourced reports -- methods that are neither reliable nor scalable. This paper proposes a novel technique for accurately and automatically inferring LLM-related scrapers. We host dynamic websites that serve unique canary tokens to each visiting scraper, then prompt LLMs for information about our sites. If an LLM consistently generates outputs containing tokens unique to a scraper, it provides evidence of exposure to that scraper. Via experiments across 22 production LLM systems, we demonstrate that our approach can reliably identify which scrapers feed which LLM, including several that are not publicly known or disclosed by the companies. Our approach provides a promising avenue for unprivileged third parties to infer which scrapers serve data to which LLMs, potentially enabling better control over unwanted scraping.

Related papers

• DeGenTWeb: A First Look at LLM-dominant Websites [4.7586268097101945]
  We describe DeGenTWeb which systematically identifies LLM-dominant websites.<n>We show how to adapt detectors of LLM-generated text for use on web pages.<n>We find that LLM-dominant sites are highly prevalent both in data from Common Crawl and in Bing's search results.
  arXiv  Detail & Related papers  (2026-04-30T17:54:35Z)
• Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users [5.7578515237305625]
  Large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts.<n>We show that without extensive manual effort, current LLM-based benchmarks allow novice users to scrape websites that would otherwise be inaccessible.
  arXiv  Detail & Related papers  (2026-01-09T20:34:28Z)
• Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions [60.43398881149664]
  We introduce LOS-Net, a lightweight attention-based architecture trained on an efficient encoding of the LLM Output Signature.<n>It achieves superior performance across diverse benchmarks and LLMs, while maintaining extremely low detection latency.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models [52.439289085318634]
  We show how to identify training data known to proprietary large language models (LLMs) by using information-guided probes.<n>Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes.
  arXiv  Detail & Related papers  (2025-03-15T10:19:15Z)
• WebWalker: Benchmarking LLMs in Web Traversal [64.48425443951749]
  We introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal.<n>We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm.
  arXiv  Detail & Related papers  (2025-01-13T18:58:07Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation [54.17246674188208]
  Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website. We introduce the paradigm of generating web scrapers with large language models (LLMs) and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently.
  arXiv  Detail & Related papers  (2024-04-19T09:59:44Z)
• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs [59.596335292426105]
  This paper collects the first open-source dataset to evaluate safeguards in large language models. We train several BERT-like classifiers to achieve results comparable with GPT-4 on automatic safety evaluation.
  arXiv  Detail & Related papers  (2023-08-25T14:02:12Z)
• Detecting Phishing Sites Using ChatGPT [2.3999111269325266]
  We propose a novel system called ChatPhishDetector that utilizes Large Language Models (LLMs) to detect phishing sites.<n>Our system involves leveraging a web crawler to gather information from websites, generating prompts for LLMs based on the crawled data, and then retrieving the detection results from the responses generated by the LLMs.<n>The experimental results using GPT-4V demonstrated outstanding performance, with a precision of 98.7% and a recall of 99.6%, outperforming the detection results of other LLMs and existing systems.
  arXiv  Detail & Related papers  (2023-06-09T11:30:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.