Attacks by Content: Automated Fact-checking is an AI Security Issue

• URL: http://arxiv.org/abs/2510.11238v1
• Date: Mon, 13 Oct 2025 10:18:48 GMT
• Title: Attacks by Content: Automated Fact-checking is an AI Security Issue
• Abstract summary: Previous research has studied indirect prompt injection, where the attacker injects malicious instructions.<n>We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information.<n>Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content.
• Score: 1.5770741892463622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When AI agents retrieve and reason over external documents, adversaries can manipulate the data they receive to subvert their behaviour. Previous research has studied indirect prompt injection, where the attacker injects malicious instructions. We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information. We term this an attack by content. Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content. To defend themselves and their users, agents must critically evaluate retrieved information, corroborating claims with external evidence and evaluating source trustworthiness. We argue that this is analogous to an existing NLP task, automated fact-checking, which we propose to repurpose as a cognitive self-defense tool for agents.

Related papers

• Agent Data Injection Attacks are Realistic Threats to AI Agents [15.288128487641464]
  We introduce a new category of IPI, agent data injection attacks (ADI)<n>ADI injects malicious data disguised as trusted data, such as security-critical metadata or agent context data.<n>ADI has similar attack impacts as instruction injection attacks, because it causes agents to misbehave and execute unintended actions.
  arXiv  Detail & Related papers  (2026-07-06T14:07:49Z)
• An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments [3.275342475988451]
  This paper studies a privacy-leakage attack chain based on indirect prompt injection.<n>We first analyze how an attacker can hijack an agent's intended task by crafting external content that appears benign to the victim.<n>We then evaluate a new prompt-language technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page.
  arXiv  Detail & Related papers  (2026-05-18T09:38:18Z)
• To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack [14.333336222782856]
  AI agents automate vulnerability discovery and exploitation across thousands of targets.<n>Current developers focus on preventing misuse through data filtering, safety alignment, and output guardrails.<n>We argue that AI-agent-driven cyber attacks are inevitable, requiring a fundamental shift in defensive strategy.
  arXiv  Detail & Related papers  (2026-02-01T12:37:55Z)
• Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods [95.54363609024847]
  Large language models (LLMs) are vulnerable to prompt injection attacks.<n>In this paper, we explore more vicious attacks that nullify the prompt injection defense methods.<n> backdoor-powered prompt injection attacks are more harmful than previous prompt injection attacks.
  arXiv  Detail & Related papers  (2025-10-04T07:11:11Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)
• Cuckoo Attack: Stealthy and Persistent Attacks Against AI-IDE [64.47951172662745]
  Cuckoo Attack is a novel attack that achieves stealthy and persistent command execution by embedding malicious payloads into configuration files.<n>We formalize our attack paradigm into two stages, including initial infection and persistence.<n>We contribute seven actionable checkpoints for vendors to evaluate their product security.
  arXiv  Detail & Related papers  (2025-09-19T04:10:52Z)
• The Hidden Dangers of Browsing AI Agents [0.0]
  This paper presents a comprehensive security evaluation of such agents, focusing on systemic vulnerabilities across multiple architectural layers.<n>Our work outlines the first end-to-end threat model for browsing agents and provides actionable guidance for securing their deployment in real-world environments.
  arXiv  Detail & Related papers  (2025-05-19T13:10:29Z)
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks [36.97842000562324]
  We introduce WASP -- a new benchmark for end-to-end evaluation of Web Agent Security against Prompt injection attacks.<n>We show that even top-tier AI models, including those with advanced reasoning capabilities, can be deceived by simple, low-effort human-written injections.<n>Our end-to-end evaluation reveals a previously unobserved insight: while attacks partially succeed in up to 86% of the case, even state-of-the-art agents often struggle to fully complete the attacker goals.
  arXiv  Detail & Related papers  (2025-04-22T17:51:03Z)
• In-Context Defense in Computer Agents: An Empirical Study [19.734768644310414]
  We introduce textbfin-context defense, leveraging in-context learning and chain-of-thought reasoning to counter such attacks.<n>Our approach involves augmenting the agent's context with a small set of carefully curated exemplars containing both malicious environments and corresponding defensive responses.<n>Experiments demonstrate the effectiveness of our method, reducing attack success rates by 91.2% on pop-up window attacks, 74.6% on average on environment injection attacks, while achieving 100% successful defenses against distracting advertisements.
  arXiv  Detail & Related papers  (2025-03-12T10:38:15Z)
• MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents [60.30753230776882]
  LLM agents are vulnerable to indirect prompt injection (IPI) attacks, where malicious tasks embedded in tool-retrieved information can redirect the agent to take unauthorized actions.<n>We present MELON, a novel IPI defense that detects attacks by re-executing the agent's trajectory with a masked user prompt modified through a masking function.
  arXiv  Detail & Related papers  (2025-02-07T18:57:49Z)
• Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges [52.96987928118327]
  We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to content injection attacks.<n>We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance.<n>Our study systematically examines the factors that influence an attack's success, such as the placement of injected content and the balance between relevant and non-relevant material.
  arXiv  Detail & Related papers  (2025-01-30T18:02:15Z)
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents [27.701301913159067]
  We introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. AgentDojo is not a static test suite, but rather an environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate AgentDojo with 97 realistic tasks, 629 security test cases, and various attack and defense paradigms from the literature.
  arXiv  Detail & Related papers  (2024-06-19T08:55:56Z)
• Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks [76.35478518372692]
  We introduce epsilon-illusory, a novel form of adversarial attack on sequential decision-makers. Compared to existing attacks, we empirically find epsilon-illusory to be significantly harder to detect with automated methods. Our findings suggest the need for better anomaly detectors, as well as effective hardware- and system-level defenses.
  arXiv  Detail & Related papers  (2022-07-20T19:49:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.