Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research

• URL: http://arxiv.org/abs/2606.14525v1
• Date: Fri, 12 Jun 2026 14:59:11 GMT
• Title: Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research
• Abstract summary: We conduct a measurement study of 10,000 websites across four browser configurations to quantify detection prevalence and employed techniques.<n>We develop a taxonomy of bot detection techniques and measure how often they appear in practice.<n>These findings demonstrate that bot detection creates systematic, provider-correlated sample loss that the web measurement community neither measures nor reports.
• Score: 0.5352699766206808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Browser automation frameworks are essential tools for security and privacy research on the web, yet bot detection scripts increasingly probe their artifacts, threatening measurement validity as automated browsers may be blocked or served different content. Prior work measures detection deployment, while we measure blocking-induced sample loss. Through a literature survey of top-tier security, privacy, and web measurement venues, we find that 83% of papers omit any discussion of bot detection blocking. To address this gap, we conduct a measurement study of 10,000 websites across four browser configurations (40K page visits in total) to quantify detection prevalence and employed techniques. Using custom instrumentation to detect when sites probe for automation, we develop a taxonomy of bot detection techniques and measure how often they appear in practice. Chromium headless encounters a 15% soft block rate compared to 7% for other configurations. Across all conditions, 82% of blocks are attributable to bot detection (59% vendor-confirmed, 23% inferred from condition-dependent blocking), predominantly by providers with integrated bot detection such as Cloudflare (37% block rate) and Akamai (26%). A header spoofing experiment establishes that 75% of Chromium-headless-only blocks are caused by header-level signals alone, yet JavaScript-based environment probing is more extensive than current blocking rates suggest. These findings demonstrate that bot detection creates systematic, provider-correlated sample loss that the web measurement community neither measures nor reports. The downstream effect on specific measurement outcomes remains future work.

Related papers

• Shy Guys: A Light-Weight Approach to Detecting Robots on Websites [1.5036109852211983]
  This paper proposes a lightweight, passive approach to bot detection, operating entirely on standard web server logs with no client-side interaction.<n>We evaluate the method on over 4.6 million requests containing 54,945 unique user-agent strings collected from website hosted all around the earth.<n>Our approach detects 67.7% of bot traffic while maintaining a false-positive rate of 3%, outperforming state of the art (less than 20%)
  arXiv  Detail & Related papers  (2026-03-30T15:07:00Z)
• RoBCtrl: Attacking GNN-Based Social Bot Detectors via Reinforced Manipulation of Bots Control Interaction [51.46634975923564]
  This paper proposes the first adversarial multi-agent Reinforcement learning framework for social Bot control attacks (RoBCtrl)<n> Specifically, we use a diffusion model to generate high-fidelity bot accounts by reconstructing existing account data with minor modifications.<n>We then employ a Multi-Agent Reinforcement Learning (MARL) method to simulate bots adversarial behavior.
  arXiv  Detail & Related papers  (2025-10-16T02:41:49Z)
• Characterizing Phishing Pages by JavaScript Capabilities [77.64740286751834]
  This paper aims to aid researchers and analysts by automatically differentiating groups of phishing pages based on the underlying kit.<n>For kit detection, our system has an accuracy of 97% on a ground-truth dataset of 548 kit families deployed across 4,562 phishing URLs.<n>We find that UI interactivity and basic fingerprinting are universal techniques, present in 90% and 80% of the clusters.
  arXiv  Detail & Related papers  (2025-09-16T15:39:23Z)
• BOTracle: A framework for Discriminating Bots and Humans [5.3248028128815434]
  Bots constitute a significant portion of Internet traffic and are a source of various issues across multiple domains.<n>We analyze the challenge of bot detection in high-traffic scenarios by analyzing three distinct detection methods.<n>Our performance metrics, including precision, recall, and AUC, reach 98 percent or higher, surpassing Botcha.
  arXiv  Detail & Related papers  (2024-12-03T08:38:30Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable [70.77600345240867]
  A novel arbitrary-in-arbitrary-out (AIAO) strategy makes watermarks resilient to fine-tuning-based removal. Unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO.
  arXiv  Detail & Related papers  (2024-05-01T12:03:39Z)
• My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection [69.99192868521564]
  Social platforms such as Twitter are under siege from a multitude of fraudulent users. Due to the structure of social networks, the majority of methods are based on the graph neural network(GNN), which is susceptible to attacks. We propose a node injection-based adversarial attack method designed to deceive bot detection models.
  arXiv  Detail & Related papers  (2023-10-11T03:09:48Z)
• An Unforgeable Publicly Verifiable Watermark for Large Language Models [84.2805275589553]
  Current watermark detection algorithms require the secret key used in the watermark generation process, making them susceptible to security breaches and counterfeiting during public detection. We propose an unforgeable publicly verifiable watermark algorithm named UPV that uses two different neural networks for watermark generation and detection, instead of using the same key at both stages.
  arXiv  Detail & Related papers  (2023-07-30T13:43:27Z)
• PhishSim: Aiding Phishing Website Detection with a Feature-Free Tool [12.468922937529966]
  We propose a feature-free method for detecting phishing websites using the Normalized Compression Distance (NCD) This measure computes the similarity of two websites by compressing them, thus eliminating the need to perform any feature extraction. We use the Furthest Point First algorithm to perform phishing prototype extractions, in order to select instances that are representative of a cluster of phishing webpages.
  arXiv  Detail & Related papers  (2022-07-13T20:44:03Z)
• An Adversarial Attack Analysis on Malicious Advertisement URL Detection Framework [22.259444589459513]
  Malicious advertisement URLs pose a security risk since they are the source of cyber-attacks. Existing malicious URL detection techniques are limited and to handle unseen features as well as generalize to test data. In this study, we extract a novel set of lexical and web-scrapped features and employ machine learning technique to set up system for fraudulent advertisement URLs detection.
  arXiv  Detail & Related papers  (2022-04-27T20:06:22Z)
• Phishing URL Detection Through Top-level Domain Analysis: A Descriptive Approach [3.494620587853103]
  This study aims to develop a machine-learning model to detect fraudulent URLs which can be used within the Splunk platform. Inspired from similar approaches in the literature, we trained the SVM and Random Forests algorithms using malicious and benign datasets. We evaluated the algorithms' performance with precision and recall, reaching up to 85% precision and 87% recall in the case of Random Forests.
  arXiv  Detail & Related papers  (2020-05-13T21:41:29Z)
• Learning to Evaluate Perception Models Using Planner-Centric Metrics [104.33349410009161]
  We propose a principled metric for 3D object detection specifically for the task of self-driving. We find that our metric penalizes many of the mistakes that other metrics penalize by design. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.
  arXiv  Detail & Related papers  (2020-04-19T02:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.