The Liabilities of Robots.txt

• URL: http://arxiv.org/abs/2503.06035v2
• Date: Wed, 27 Aug 2025 09:19:32 GMT
• Title: The Liabilities of Robots.txt
• Abstract summary: We argue that civil doctrines offer a more balanced and sustainable framework for regulating web robot behavior.<n>We argue that clarifying liability for "robots.txt" violations is essential to addressing the growing fragmentation of the internet.
• Score: 9.246300945280142
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper explores the legal implications of violating "robots.txt", a technical standard widely used by webmasters to communicate restrictions on automated access to website content. Although historically regarded as a voluntary guideline, the rise of generative AI and large-scale web scraping has amplified the consequences of disregarding "robots.txt" directives. While previous legal discourse has largely focused on criminal or copyright-based remedies, we argue that civil doctrines, particularly in contract and tort law, offer a more balanced and sustainable framework for regulating web robot behavior in common law jurisdictions. Under certain conditions, "robots.txt" can give rise to a unilateral contract or serve as a form of notice sufficient to establish tortious liability, including trespass to chattels and negligence. Ultimately, we argue that clarifying liability for "robots.txt" violations is essential to addressing the growing fragmentation of the internet. By restoring balance and accountability in the digital ecosystem, our proposed framework helps preserve the internet's open and cooperative foundations. Through this lens, "robots.txt" can remain an equitable and effective tool for digital governance in the age of AI.

Related papers

• Operational Agency: A Permeable Legal Fiction for Tracing Culpability in AI Systems [0.31061678033205636]
  This Article introduces Operational Agency (OA)-a permeable legal fiction structured as an ex post evidentiary framework-and Operational Agency Graph (OAG)<n>OA evaluates an AI's observable operational characteristics.<n>OAG operationalizes that analysis by embedding these characteristics in a causal graph to trace and apportion culpability among developers, fine-tuners, deployers, and users.
  arXiv  Detail & Related papers  (2026-02-20T01:49:03Z)
• What hackers talk about when they talk about AI: Early-stage diffusion of a cybercrime innovation [0.0]
  The rapid expansion of artificial intelligence (AI) is raising concerns about its potential to transform cybercrime.<n>This paper examines the evolving relationship between cybercriminals and AI using a unique dataset from a cyber threat intelligence platform.
  arXiv  Detail & Related papers  (2026-02-16T14:31:33Z)
• In the Mood to Exclude: Revitalizing Trespass to Chattels in the Era of GenAI Scraping [1.8256490853231881]
  This paper argues that website owners have the right to exclude others from their websites.<n>If the scraping leads to a decrease in the website's value, then trespass to chattels should apply.
  arXiv  Detail & Related papers  (2025-10-16T18:33:48Z)
• A roadmap for AI in robotics [55.87087746398059]
  We are witnessing growing excitement in robotics at the prospect of leveraging the potential of AI to tackle some of the outstanding barriers to the full deployment of robots in our daily lives.<n>This article offers an assessment of what AI for robotics has achieved since the 1990s and proposes a short- and medium-term research roadmap listing challenges and promises.
  arXiv  Detail & Related papers  (2025-07-26T15:18:28Z)
• Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics [55.05920313034645]
  We introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control.<n>Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions.<n>Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks.
  arXiv  Detail & Related papers  (2025-05-29T16:41:12Z)
• Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study [4.68008217188575]
  We conduct the first large-scale study of web scraper compliance with robots.txt directives using anonymized web logs from our institution.<n>We find that bots are less likely to comply with stricter robots.txt directives, and that certain categories of bots, including AI search crawlers, rarely check robots.txt at all.<n>These findings suggest that relying on robots.txt to prevent unwanted scraping is risky and highlight the need for alternative approaches.
  arXiv  Detail & Related papers  (2025-05-27T20:22:45Z)
• ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet [44.29685364907017]
  We introduce ai.txt, a domain-specific language designed to regulate interactions between AI models, agents, and web content.<n>Our approach aims to aid the governance of AI-Internet interactions, promoting responsible AI use in digital ecosystems.
  arXiv  Detail & Related papers  (2025-05-02T00:33:00Z)
• Generating Robot Constitutions & Benchmarks for Semantic Safety [22.889717765617394]
  We release the ASIMOV Benchmark for evaluating semantic safety of robot brains. We develop a framework to automatically generate robot constitutions from real-world data. We propose a novel auto-amending process that is able to introduce nuances in written rules of behavior.
  arXiv  Detail & Related papers  (2025-03-11T17:50:47Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Consent in Crisis: The Rapid Decline of the AI Data Commons [74.68176012363253]
  General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data. We conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
  arXiv  Detail & Related papers  (2024-07-20T16:50:18Z)
• Robotic Control via Embodied Chain-of-Thought Reasoning [86.6680905262442]
  Key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models can substantially improve their robustness and generalization ability. We introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features before predicting the robot action.
  arXiv  Detail & Related papers  (2024-07-11T17:31:01Z)
• LLM Granularity for On-the-Fly Robot Control [3.5015824313818578]
  In circumstances where visuals become unreliable or unavailable, can we rely solely on language to control robots? This work takes the initial steps to answer this question by: 1) evaluating the responses of assistive robots to language prompts of varying granularities; and 2) exploring the necessity and feasibility of controlling the robot on-the-fly.
  arXiv  Detail & Related papers  (2024-06-20T18:17:48Z)
• Debunking Robot Rights Metaphysically, Ethically, and Legally [0.10241134756773229]
  We argue that machines are not the kinds of things that may be denied or granted rights. From a legal perspective, the best analogy to robot rights is not human rights but corporate rights.
  arXiv  Detail & Related papers  (2024-04-15T18:23:58Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Learning Vision-based Pursuit-Evasion Robot Policies [54.52536214251999]
  We develop a fully-observable robot policy that generates supervision for a partially-observable one. We deploy our policy on a physical quadruped robot with an RGB-D camera on pursuit-evasion interactions in the wild.
  arXiv  Detail & Related papers  (2023-08-30T17:59:05Z)
• Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations [66.47064743686953]
  Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation. Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation. In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
  arXiv  Detail & Related papers  (2023-07-12T07:04:53Z)
• REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer [57.045140028275036]
  We consider the problem of transferring a policy across two different robots with significantly different parameters such as kinematics and morphology. Existing approaches that train a new policy by matching the action or state transition distribution, including imitation learning methods, fail due to optimal action and/or state distribution being mismatched in different robots. We propose a novel method named $REvolveR$ of using continuous evolutionary models for robotic policy transfer implemented in a physics simulator.
  arXiv  Detail & Related papers  (2022-02-10T18:50:25Z)
• Federated Continual Learning for Socially Aware Robotics [4.224305864052757]
  Social robots do not adapt their behavior to new users, and they do not provide sufficient privacy protections. We propose a decentralized learning alternative that improves the privacy and personalization of social robots. We show that decentralized learning is a viable alternative to centralized learning in a proof-of-concept Socially-Aware Navigation domain.
  arXiv  Detail & Related papers  (2022-01-14T15:54:51Z)
• A New Paradigm of Threats in Robotics Behaviors [4.873362301533825]
  We identify a new paradigm of security threats in the next generation of robots. These threats fall beyond the known hardware or network-based ones. We provide a taxonomy of attacks that exploit these vulnerabilities with realistic examples.
  arXiv  Detail & Related papers  (2021-03-24T15:33:49Z)
• Fault-Aware Robust Control via Adversarial Reinforcement Learning [35.16413579212691]
  We propose an adversarial reinforcement learning framework, which significantly increases robot fragility over joint damage cases. We validate our algorithm on a three-fingered robot hand and a quadruped robot. Our algorithm can be trained only in simulation and directly deployed on a real robot without any fine-tuning.
  arXiv  Detail & Related papers  (2020-11-17T16:01:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.