Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior

• URL: http://arxiv.org/abs/2508.19287v1
• Date: Mon, 25 Aug 2025 05:20:11 GMT
• Title: Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior
• Authors: Zhuotao Lian, Weiyu Wang, Qingkui Zeng, Toru Nakanishi, Teruaki Kitasuka, Chunhua Su,
• Abstract summary: Large Language Models (LLMs) are widely deployed in applications that accept user-submitted content.<n>In this paper, we identify a new class of attacks, prompt in content injection, where adversarial instructions are embedded in seemingly benign inputs.
• Score: 3.715687323488774
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) are widely deployed in applications that accept user-submitted content, such as uploaded documents or pasted text, for tasks like summarization and question answering. In this paper, we identify a new class of attacks, prompt in content injection, where adversarial instructions are embedded in seemingly benign inputs. When processed by the LLM, these hidden prompts can manipulate outputs without user awareness or system compromise, leading to biased summaries, fabricated claims, or misleading suggestions. We demonstrate the feasibility of such attacks across popular platforms, analyze their root causes including prompt concatenation and insufficient input isolation, and discuss mitigation strategies. Our findings reveal a subtle yet practical threat in real-world LLM workflows.

Related papers

• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
  Large language models (LLMs) are vulnerable to prompt injection attacks.<n>We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-29T07:13:53Z)
• Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injection [12.565784666173277]
  This report presents a real-world case study demonstrating how prompt injection can attack large language model platforms such as ChatGPT.<n>We show how adversarial prompts can be injected via user inputs, web-based retrieval, and system-level agent instructions.
  arXiv  Detail & Related papers  (2025-04-20T05:59:00Z)
• Compromising Embodied Agents with Contextual Backdoor Attacks [69.71630408822767]
  Large language models (LLMs) have transformed the development of embodied intelligence. This paper uncovers a significant backdoor security threat within this process. By poisoning just a few contextual demonstrations, attackers can covertly compromise the contextual environment of a black-box LLM.
  arXiv  Detail & Related papers  (2024-08-06T01:20:12Z)
• Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context [49.13497493053742]
  This research explores converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs.
  arXiv  Detail & Related papers  (2024-07-19T19:47:26Z)
• Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications [0.0]
  This paper introduces the 'Signed-Prompt' method as a novel solution for prompt injection attacks. The study involves signing sensitive instructions within command segments by authorized users, enabling the LLM to discern trusted instruction sources. Experiments demonstrate the effectiveness of the Signed-Prompt method, showing substantial resistance to various types of prompt injection attacks.
  arXiv  Detail & Related papers  (2024-01-15T11:44:18Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition [8.560772603154545]
  Large Language Models are vulnerable to prompt injection and jailbreaking. We launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2023-10-24T18:18:11Z)
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
  arXiv  Detail & Related papers  (2023-08-17T06:21:50Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.