Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers

• URL: http://arxiv.org/abs/2607.14447v2
• Date: Sun, 19 Jul 2026 00:51:32 GMT
• Title: Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
• Abstract summary: AI assistants increasingly retrieve web content at inference time to provide fresh and grounded answers.<n>It remains unclear whether these search-augmented capabilities respect website-owner restrictions expressed through robots$.$txt.<n>We present a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities.
• Score: 0.3499870393443268
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: AI assistants increasingly retrieve web content at inference time to provide fresh and grounded answers, yet it remains unclear whether these search-augmented capabilities respect website-owner restrictions expressed through robots$.$txt. We present a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities. For each assistant, we first identify a configuration that actually produces observable web-browsing behavior and record the user-agent exposed during retrieval. We then evaluate compliance with controlled robots$.$txt rules across four complementary conditions: allowed for all user-agents, disallowed for all user-agents, allowed only for the assistant-specific user-agent, and disallowed only for that user-agent. Using server-side logs and secret codes embedded in target pages, we distinguish actual page access from user-visible answer correctness across 200 trials. Our results show substantial variation across assistants. Some systems followed the expected allowed/disallowed access pattern, whereas others accessed restricted resources without requesting robots$.$txt or used generic user-agents that complicated attribution. We also find that retrieval behavior and answer correctness can diverge: assistants may access pages without surfacing the retrieved content, or fail to access even allowed resources. These findings raise broader legal and governance concerns about whether AI-assisted web access adequately respects content owners' rights and restrictions. Furthermore, our observations provide valuable insight into the growing erosion of traditional web governance protocols, highlighting the urgent need for updated, enforceable standards that guarantee publisher autonomy in the age of search-augmented AI assistants.

Related papers

• AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence [2.303636351420161]
  We estimate the causal effects of AI Overviews and AI Mode on user behavior, perceptions, and publisher traffic on Google Search.<n>We show that removing AI Overviews and AI Mode increases click-through rates to publishers, while an AI Mode-only experience reduces click-through rates and erodes user experience and trust in information found on Google.
  arXiv  Detail & Related papers  (2026-08-18T22:08:10Z)
• Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals [0.0]
  The rapid adoption of AI coding agents and AI assistant web services is fundamentally changing how developers discover, consume, and interact with technical documentation.<n>This paper studies that transformation across three interconnected dimensions: documentation accessibility, content analytics, and feedback systems.<n>We present an empirical study of HTTP request fingerprints from nine AI coding agents (Aider, Antigravity, Claude Code, Cline, Cursor, Junie, OpenCode, VS Code, and Windsurf) and six AI assistant services (ChatGPT, Claude, Google Gemini, Google NotebookLM, MistralAI, and Perplexity) accessing a live developer documentation endpoint.
  arXiv  Detail & Related papers  (2026-04-02T21:54:07Z)
• Nested Browser-Use Learning for Agentic Information Seeking [60.775556172513014]
  Information-seeking (IS) agents have achieved strong performance across a range of wide and deep search tasks, yet their tool use remains largely restricted to API-level snippet retrieval and URL-based page fetching.<n>We propose Nested Browser-Use Learning (NestBrowse), which introduces a minimal and complete browser-action framework that decouples interaction control from page exploration through a nested structure.
  arXiv  Detail & Related papers  (2025-12-29T17:59:14Z)
• BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions [48.194688161526756]
  BrowserAgent operates directly on raw web pages via Playwright through a set of predefined browser actions.<n>We introduce an explicit memory mechanism to store key conclusions across steps, further enhancing the model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T15:43:37Z)
• Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web [2.370481325034444]
  We investigate whether reputable news websites and misinformation sites differ in how they configure robots.txt files.<n>We find that 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files.<n>Our findings highlight a growing asymmetry in content accessibility that may shape the training data available to Large Language Models.
  arXiv  Detail & Related papers  (2025-10-11T18:47:14Z)
• AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code [11.11923891120399]
  We introduce a novel taxonomy classifying Web accessibility violations into three key categories - Syntactic, Semantic, and Layout.<n>We propose a novel method, AccessGuru, which combines existing accessibility testing tools and Large Language Models (LLMs) to detect violations.<n>Our benchmark quantifies syntactic and layout compliance and judges semantic accuracy through comparative analysis with human expert corrections.
  arXiv  Detail & Related papers  (2025-07-24T17:59:30Z)
• Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistants [10.38225151849918]
  Generative AI (GenAI) browser assistants integrate powerful capabilities of GenAI in web browsers to provide rich experiences such as question answering, content summarization, and agentic navigation.<n>These assistants, available today as browser extensions, can not only track detailed browsing activity such as search and click data, but can also autonomously perform tasks such as filling forms, raising significant privacy concerns.<n>We study their ability to profile users and personalize their responses based on explicit or inferred demographic attributes and interests of users.
  arXiv  Detail & Related papers  (2025-03-20T16:21:47Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• Beyond Browsing: API-Based Web Agents [58.39129004543844]
  API-Based Agents outperform web Browsing Agents in experiments on WebArena.<n>Hybrid Agents out-perform both others nearly uniformly across tasks.<n>Results strongly suggest that when APIs are available, they present an attractive alternative to relying on web browsing alone.
  arXiv  Detail & Related papers  (2024-10-21T19:46:06Z)
• Consent in Crisis: The Rapid Decline of the AI Data Commons [74.68176012363253]
  General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data. We conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
  arXiv  Detail & Related papers  (2024-07-20T16:50:18Z)
• User Attitudes to Content Moderation in Web Search [49.1574468325115]
  We examine the levels of support for different moderation practices applied to potentially misleading and/or potentially offensive content in web search. We find that the most supported practice is informing users about potentially misleading or offensive content, and the least supported one is the complete removal of search results. More conservative users and users with lower levels of trust in web search results are more likely to be against content moderation in web search.
  arXiv  Detail & Related papers  (2023-10-05T10:57:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.