Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

• URL: http://arxiv.org/abs/2510.10315v3
• Date: Thu, 16 Oct 2025 17:26:58 GMT
• Title: Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
• Abstract summary: We investigate whether reputable news websites and misinformation sites differ in how they configure robots.txt files.<n>We find that 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files.<n>Our findings highlight a growing asymmetry in content accessibility that may shape the training data available to Large Language Models.
• Score: 2.370481325034444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files. Reputable sites forbid an average of 15.5 AI user agents, while misinformation sites prohibit fewer than one. We then measure active blocking behavior, where websites refuse to return content when HTTP requests include AI crawler user agents, and reveal that both categories of websites utilize it. Notably, the behavior of reputable news websites in this regard aligns more closely with their declared robots.txt directive than that of misinformation websites. Finally, our longitudinal analysis reveals that this gap has widened over time, with AI-blocking by reputable sites rising from 23% in September 2023 to nearly 60% by May 2025. Our findings highlight a growing asymmetry in content accessibility that may shape the training data available to LLMs, raising essential questions for web transparency, data ethics, and the future of AI training practices.

Related papers

• Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers [0.3499870393443268]
  AI assistants increasingly retrieve web content at inference time to provide fresh and grounded answers.<n>It remains unclear whether these search-augmented capabilities respect website-owner restrictions expressed through robots$.$txt.<n>We present a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities.
  arXiv  Detail & Related papers  (2026-07-16T00:45:06Z)
• Shy Guys: A Light-Weight Approach to Detecting Robots on Websites [1.5036109852211983]
  This paper proposes a lightweight, passive approach to bot detection, operating entirely on standard web server logs with no client-side interaction.<n>We evaluate the method on over 4.6 million requests containing 54,945 unique user-agent strings collected from website hosted all around the earth.<n>Our approach detects 67.7% of bot traffic while maintaining a false-positive rate of 3%, outperforming state of the art (less than 20%)
  arXiv  Detail & Related papers  (2026-03-30T15:07:00Z)
• How do data owners say no? A case study of data consent mechanisms in web-scraped vision-language AI training datasets [2.790280665168348]
  We aim to reveal information about data owners' consent to AI scraping and training, and study how it's expressed in DataComp.<n>We estimate at least 122M of samples exhibit some indication of copyright notice in CommonPool.<n>Our holistic methods and findings show that data owners rely on various channels to convey data consent, of which current AI data collection pipelines do not entirely respect.
  arXiv  Detail & Related papers  (2025-11-10T05:41:02Z)
• A roadmap for AI in robotics [55.87087746398059]
  We are witnessing growing excitement in robotics at the prospect of leveraging the potential of AI to tackle some of the outstanding barriers to the full deployment of robots in our daily lives.<n>This article offers an assessment of what AI for robotics has achieved since the 1990s and proposes a short- and medium-term research roadmap listing challenges and promises.
  arXiv  Detail & Related papers  (2025-07-26T15:18:28Z)
• Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study [4.68008217188575]
  We conduct the first large-scale study of web scraper compliance with robots.txt directives using anonymized web logs from our institution.<n>We find that bots are less likely to comply with stricter robots.txt directives, and that certain categories of bots, including AI search crawlers, rarely check robots.txt at all.<n>These findings suggest that relying on robots.txt to prevent unwanted scraping is risky and highlight the need for alternative approaches.
  arXiv  Detail & Related papers  (2025-05-27T20:22:45Z)
• The Liabilities of Robots.txt [9.246300945280142]
  We argue that civil doctrines offer a more balanced and sustainable framework for regulating web robot behavior.<n>We argue that clarifying liability for "robots.txt" violations is essential to addressing the growing fragmentation of the internet.
  arXiv  Detail & Related papers  (2025-03-08T03:16:17Z)
• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  This study systematically evaluations twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset.<n>Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)
• Consent in Crisis: The Rapid Decline of the AI Data Commons [74.68176012363253]
  General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data. We conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
  arXiv  Detail & Related papers  (2024-07-20T16:50:18Z)
• Towards Generalist Robot Learning from Internet Video: A Survey [56.621902345314645]
  This survey systematically examines the emerging field of learning from videos (LfV)<n>We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data.<n>Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training.
  arXiv  Detail & Related papers  (2024-04-30T15:57:41Z)
• Learning to Defer in Congested Systems: The AI-Human Interplay [4.324474867341765]
  High-stakes applications rely on combining Artificial Intelligence (AI) and humans for responsive and reliable decision making.<n>In this paper, we introduce a model to capture such an AI-human interplay.<n>We propose a near-optimal learning algorithm that carefully balances the classification loss from a selectively sampled dataset.
  arXiv  Detail & Related papers  (2024-02-19T15:47:47Z)
• Detecting The Corruption Of Online Questionnaires By Artificial Intelligence [1.9458156037869137]
  This study tested if text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. Humans were able to correctly identify authorship of text above chance level. But their performance was still below what would be required to ensure satisfactory data quality.
  arXiv  Detail & Related papers  (2023-08-14T23:47:56Z)
• GNM: A General Navigation Model to Drive Any Robot [67.40225397212717]
  General goal-conditioned model for vision-based navigation can be trained on data obtained from many distinct but structurally similar robots. We analyze the necessary design decisions for effective data sharing across robots. We deploy the trained GNM on a range of new robots, including an under quadrotor.
  arXiv  Detail & Related papers  (2022-10-07T07:26:41Z)
• A Survey of Machine Unlearning [56.017968863854186]
  Recent regulations now require that, on request, private information about a user must be removed from computer systems. ML models often remember' the old data. Recent works on machine unlearning have not been able to completely solve the problem.
  arXiv  Detail & Related papers  (2022-09-06T08:51:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.