Defending Against Prompt Injection with DataFilter

• URL: http://arxiv.org/abs/2510.19207v1
• Date: Wed, 22 Oct 2025 03:30:49 GMT
• Title: Defending Against Prompt Injection with DataFilter
• Authors: Yizhu Wang, Sizhe Chen, Raghad Alkhudair, Basel Alomair, David Wagner,
• Abstract summary: Large language model (LLM) agents are increasingly deployed to automate tasks and interact with untrusted external data.<n>By injecting malicious instructions into the data that LLMs access, an attacker can arbitrarily override the original user task and redirect the agent toward unintended, potentially harmful actions.<n>We propose DataFilter, a test-time model-agnostic defense that removes malicious instructions from the data before it reaches the backend LLM.
• Score: 7.1507566575747346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When large language model (LLM) agents are increasingly deployed to automate tasks and interact with untrusted external data, prompt injection emerges as a significant security threat. By injecting malicious instructions into the data that LLMs access, an attacker can arbitrarily override the original user task and redirect the agent toward unintended, potentially harmful actions. Existing defenses either require access to model weights (fine-tuning), incur substantial utility loss (detection-based), or demand non-trivial system redesign (system-level). Motivated by this, we propose DataFilter, a test-time model-agnostic defense that removes malicious instructions from the data before it reaches the backend LLM. DataFilter is trained with supervised fine-tuning on simulated injections and leverages both the user's instruction and the data to selectively strip adversarial content while preserving benign information. Across multiple benchmarks, DataFilter consistently reduces the prompt injection attack success rates to near zero while maintaining the LLMs' utility. DataFilter delivers strong security, high utility, and plug-and-play deployment, making it a strong practical defense to secure black-box commercial LLMs against prompt injection. Our DataFilter model is released at https://huggingface.co/JoyYizhu/DataFilter for immediate use, with the code to reproduce our results at https://github.com/yizhu-joy/DataFilter.

Related papers

• Better Privilege Separation for Agents by Restricting Data Types [6.028799607869068]
  We propose type-directed privilege separation for large language models (LLMs)<n>We restrict the ability of an LLM to interact with third-party data by converting untrusted content to a curated set of data types.<n>Unlike raw strings, each data type is limited in scope and content, eliminating the possibility for prompt injections.
  arXiv  Detail & Related papers  (2025-09-30T08:20:50Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks [87.66245688589977]
  LLM-integrated applications and agents are vulnerable to prompt injection attacks.<n>A detection method aims to determine whether a given input is contaminated by an injected prompt.<n>We propose DataSentinel, a game-theoretic method to detect prompt injection attacks.
  arXiv  Detail & Related papers  (2025-04-15T16:26:21Z)
• Defeating Prompt Injections by Design [79.00910871948787]
  CaMeL is a robust defense that creates a protective system layer around the Large Language Models.<n>To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query.<n>To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows.
  arXiv  Detail & Related papers  (2025-03-24T15:54:10Z)
• Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment [35.344406718760574]
  A prompt injection attack aims to make an Large Language Model follow an injected prompt to perform an attacker-chosen task.<n>Existing attacks primarily focus on crafting these injections at inference time, treating the LLM itself as a static target.<n>In this work, we introduce a more foundational attack vector: poisoning the LLM's alignment process to amplify the success of future prompt injection attacks.
  arXiv  Detail & Related papers  (2024-10-18T18:52:16Z)
• SecAlign: Defending Against Prompt Injection with Preference Optimization [52.48001255555192]
  Adversarial prompts can be injected into external data sources to override the system's intended instruction and execute a malicious instruction.<n>We propose a new defense called SecAlign based on the technique of preference optimization.<n>Our method reduces the success rates of various prompt injections to 10%, even against attacks much more sophisticated than ones seen during training.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game [86.66627242073724]
  This paper presents a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking.
  arXiv  Detail & Related papers  (2023-11-02T06:13:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.