Shy Guys: A Light-Weight Approach to Detecting Robots on Websites

• URL: http://arxiv.org/abs/2603.28546v2
• Date: Tue, 31 Mar 2026 12:51:55 GMT
• Title: Shy Guys: A Light-Weight Approach to Detecting Robots on Websites
• Abstract summary: This paper proposes a lightweight, passive approach to bot detection, operating entirely on standard web server logs with no client-side interaction.<n>We evaluate the method on over 4.6 million requests containing 54,945 unique user-agent strings collected from website hosted all around the earth.<n>Our approach detects 67.7% of bot traffic while maintaining a false-positive rate of 3%, outperforming state of the art (less than 20%)
• Score: 1.5036109852211983
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automated bots now account for roughly half of all web requests, and an increasing number deliberately spoof their identity to either evade detection or to not respect robots.txt. Existing countermeasures are either resource-intensive (JavaScript challenges, CAPTCHAs), cost-prohibitive (commercial solutions), or degrade the user experience. This paper proposes a lightweight, passive approach to bot detection that combines user-agent string analysis with favicon-based heuristics, operating entirely on standard web server logs with no client-side interaction. We evaluate the method on over 4.6 million requests containing 54,945 unique user-agent strings collected from website hosted all around the earth. Our approach detects 67.7% of bot traffic while maintaining a false-positive rate of 3%, outperforming state of the art (less than 20%). This method can serve as a first line of defence, routing only genuinely ambiguous requests to active challenges and preserving the experience of legitimate users.

Related papers

• Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents [0.10467550794914121]
  We evaluate the resilience of challenge-based defenses and non-interactive trust-based defenses against two attacker classes: commercial Captcha-solving services and LLM-based browser agents.<n>Our results show that challenge-based defenses are broadly ineffective against commercial solvers, which achieve near-perfect bypass at negligible cost.<n>Non-interactive defenses such as reCaptcha v3 exhibit stronger resistance, but our analysis reveals that this resilience does not reflect a fundamental security property.
  arXiv  Detail & Related papers  (2026-07-21T03:02:57Z)
• On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting [1.4726407245786044]
  Since 2023, a new class of bots has emerged: Web Agents.<n>They can automate complex tasks on the Web, going beyond traditional browser automation tools such as Selenium, Puppeteer, or Playwright.<n>It is becoming increasingly difficult for website administrators to detect and block these LLM-based bots.
  arXiv  Detail & Related papers  (2026-06-29T10:51:13Z)
• Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research [0.5352699766206808]
  We conduct a measurement study of 10,000 websites across four browser configurations to quantify detection prevalence and employed techniques.<n>We develop a taxonomy of bot detection techniques and measure how often they appear in practice.<n>These findings demonstrate that bot detection creates systematic, provider-correlated sample loss that the web measurement community neither measures nor reports.
  arXiv  Detail & Related papers  (2026-06-12T14:59:11Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• MolmoWeb: Open Visual Web Agent and Open Data for the Open Web [60.29597961827816]
  MolmoWebMix is a mixture of browser task demonstrations and web-GUI perception data.<n>MolmoWeb-8B is a family of fully open multimodal web agents.<n>We will release model checkpoints, training data, code, and a unified evaluation harness to enable open research on web agents.
  arXiv  Detail & Related papers  (2026-04-09T17:54:02Z)
• Software Testing at the Network Layer: Automated HTTP API Quality Assessment and Security Analysis of Production Web Applications [1.9537983097153042]
  We present an automated software testing framework that captures and analyzes the complete HTTP traffic of 18 production websites.<n> minimalist server-rendered sites achieve perfect scores of 100, while content-heavy commercial sites score as low as 56.8.<n>We identify redundant API calls and missing cache headers as the two most pervasive anti-patterns, each affecting 67% of sites.
  arXiv  Detail & Related papers  (2026-02-09T03:39:45Z)
• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents [52.81924177620322]
  Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.<n>Their reliance on dynamic web content makes them vulnerable to prompt injection attacks: adversarial instructions hidden in interface elements that persuade the agent to divert from its original task.<n>We introduce the Task-Redirecting Agent Persuasion Benchmark (TRAP), an evaluation for studying how persuasion techniques misguide autonomous web agents on realistic tasks.
  arXiv  Detail & Related papers  (2025-12-29T01:09:10Z)
• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks [51.803138848305814]
  We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks.<n>We identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs.<n>Our findings surface both the diversity and brittleness of current web agents.
  arXiv  Detail & Related papers  (2025-10-02T15:22:21Z)
• Characterizing Phishing Pages by JavaScript Capabilities [77.64740286751834]
  This paper aims to aid researchers and analysts by automatically differentiating groups of phishing pages based on the underlying kit.<n>For kit detection, our system has an accuracy of 97% on a ground-truth dataset of 548 kit families deployed across 4,562 phishing URLs.<n>We find that UI interactivity and basic fingerprinting are universal techniques, present in 90% and 80% of the clusters.
  arXiv  Detail & Related papers  (2025-09-16T15:39:23Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)
• BOTracle: A framework for Discriminating Bots and Humans [5.3248028128815434]
  Bots constitute a significant portion of Internet traffic and are a source of various issues across multiple domains.<n>We analyze the challenge of bot detection in high-traffic scenarios by analyzing three distinct detection methods.<n>Our performance metrics, including precision, recall, and AUC, reach 98 percent or higher, surpassing Botcha.
  arXiv  Detail & Related papers  (2024-12-03T08:38:30Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection [69.99192868521564]
  Social platforms such as Twitter are under siege from a multitude of fraudulent users. Due to the structure of social networks, the majority of methods are based on the graph neural network(GNN), which is susceptible to attacks. We propose a node injection-based adversarial attack method designed to deceive bot detection models.
  arXiv  Detail & Related papers  (2023-10-11T03:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.