Related papers: PromptShield: Deployable Detection for Prompt Injection Attacks

Related papers

LongPIBench: A Long-Context Benchmark for Prompt Injection [52.0498674763688]
LongPIBench is a benchmark for evaluating prompt injection defenses in long-context scenarios.<n>We show that even simple prompt injection attacks achieve high success rates and frequently bypass state-of-the-art defenses.
arXiv Detail & Related papers (2026-08-28T15:00:33Z)
Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening [101.09427874773901]
We present the first systematic study of prompt-injection attacks in a widely used application: LLM-based resume screening.<n>Our analysis is based on approximately 200K real-world resumes collected over multiple years by hireEZ.
arXiv Detail & Related papers (2026-05-27T18:56:19Z)
ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
arXiv Detail & Related papers (2026-01-15T08:23:38Z)
Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs [27.395843922014294]
Large Language Models (LLMs) have been integrated into many applications (e.g., web agents) to perform more sophisticated tasks.<n>This paper presents Rennervate, a defense framework to detect and prevent IPI attacks.
arXiv Detail & Related papers (2025-12-09T09:44:13Z)
PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features [33.95073302161128]
Existing prompt injection detection methods often have sub-optimal performance and/or high computational overhead.<n>We propose PIShield, a detection method that is both effective and efficient.<n>We show that PIShield is both highly effective and efficient, substantially outperforming existing methods.
arXiv Detail & Related papers (2025-10-15T18:34:49Z)
PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance [10.105673138616483]
Large Language Models (LLMs) are increasingly integrated into real-world applications, from virtual assistants to autonomous agents.<n>As attackers evolve with paraphrased, obfuscated, and even multi-task injection strategies, existing benchmarks are no longer sufficient to capture the full spectrum of emerging threats.<n>We propose PromptSleuth, a semantic-oriented defense framework that detects prompt injection by reasoning over task-level intent rather than surface features.
arXiv Detail & Related papers (2025-08-28T15:19:07Z)
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition [71.81906608221038]
Large language models (LLMs) are vulnerable to indirect prompt injection attacks.<n>We propose TopicAttack, which prompts the LLM to generate a fabricated transition prompt that gradually shifts the topic toward the injected instruction.<n>We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.
arXiv Detail & Related papers (2025-07-18T06:23:31Z)
Enhancing Security in LLM Applications: A Performance Evaluation of Early Detection Systems [1.03590082373586]
In prompt injection attacks, an attacker maliciously manipulates the system instructions, violating the system's confidentiality.<n>We investigated the capabilities of early prompt injection detection systems, focusing specifically on the detection performance of techniques implemented in various open-source solutions.<n>Our study presents analyzes of distinct prompt leakage detection techniques, and a comparative analysis of several detection solutions, which implement those techniques.
arXiv Detail & Related papers (2025-06-23T20:39:43Z)
CAPTURE: Context-Aware Prompt Injection Testing and Robustness Enhancement [0.34530027457862006]
We introduce CAPTURE, a novel context-aware benchmark assessing both attack detection and over-defense tendencies.<n>Our experiments reveal that current prompt injection guardrail models suffer from high false negatives in adversarial cases and excessive false positives in benign scenarios.<n>This new model drastically reduces both false negative and false positive rates on our context-aware datasets.
arXiv Detail & Related papers (2025-05-18T11:14:14Z)
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
Large language models (LLMs) are vulnerable to prompt injection attacks. We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
arXiv Detail & Related papers (2025-04-29T07:13:53Z)
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks [101.52204404377039]
LLM-integrated applications and agents are vulnerable to prompt injection attacks. A detection method aims to determine whether a given input is contaminated by an injected prompt. We propose DataSentinel, a game-theoretic method to detect prompt injection attacks.
arXiv Detail & Related papers (2025-04-15T16:26:21Z)
Can Indirect Prompt Injection Attacks Be Detected and Removed? [68.6543680065379]
We investigate the feasibility of detecting and removing indirect prompt injection attacks. For detection, we assess the performance of existing LLMs and open-source detection models. For removal, we evaluate two intuitive methods: (1) the segmentation removal method, which segments the injected document and removes parts containing injected instructions, and (2) the extraction removal method, which trains an extraction model to identify and remove injected instructions.
arXiv Detail & Related papers (2025-02-23T14:02:16Z)
Prompt Inject Detection with Generative Explanation as an Investigative Tool [0.0]
Large Language Models (LLMs) are vulnerable to adversarial prompt based injects. This research explores the use of a text generation capabilities of LLM to detect prompt injects.
arXiv Detail & Related papers (2025-02-16T06:16:00Z)
Attention Tracker: Detecting Prompt Injection Attacks in LLMs [62.247841717696765]
Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks. We introduce the concept of the distraction effect, where specific attention heads shift focus from the original instruction to the injected instruction. We propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks.
arXiv Detail & Related papers (2024-11-01T04:05:59Z)
Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection [6.269725911814401]
Large language models (LLMs) are becoming a popular tool as they have significantly advanced in their capability to tackle a wide range of language-based tasks. However, LLMs applications are highly vulnerable to prompt injection attacks, which poses a critical problem. This project explores the security vulnerabilities in relation to prompt injection attacks.
arXiv Detail & Related papers (2024-10-28T00:36:21Z)
PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs [2.4117856871959953]
Large Language Models (LLMs) have gained widespread use in various applications due to their powerful capability to generate human-like text. prompt injection attacks involve overwriting a model's original instructions with malicious prompts to manipulate the generated text. We propose PROMPTFUZZ, a novel testing framework that leverages fuzzing techniques to assess the robustness of LLMs against prompt injection attacks.
arXiv Detail & Related papers (2024-09-23T06:08:32Z)
Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations [76.19419888353586]
Large language models (LLMs) are susceptible to a variety of risks, from non-faithful output to biased and toxic generations. We present our efforts to create and deploy a library of detectors: compact and easy-to-build classification models that provide labels for various harms.
arXiv Detail & Related papers (2024-03-09T21:07:16Z)
Formalizing and Benchmarking Prompt Injection Attacks and Defenses [59.57908526441172]
We propose a framework to formalize prompt injection attacks. Based on our framework, we design a new attack by combining existing ones. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses.
arXiv Detail & Related papers (2023-10-19T15:12:09Z)
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
arXiv Detail & Related papers (2023-08-17T06:21:50Z)
Red Teaming Language Model Detectors with Language Models [114.36392560711022]
Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
arXiv Detail & Related papers (2023-05-31T10:08:37Z)
Large Language Models can be Guided to Evade AI-Generated Text Detection [40.7707919628752]
Large language models (LLMs) have shown remarkable performance in various tasks and have been extensively utilized by the public. We equip LLMs with prompts, rather than relying on an external paraphraser, to evaluate the vulnerability of these detectors. We propose a novel Substitution-based In-Context example optimization method (SICO) to automatically construct prompts for evading the detectors.
arXiv Detail & Related papers (2023-05-18T10:03:25Z)
Adversarially Robust One-class Novelty Detection [83.1570537254877]
We show that existing novelty detectors are susceptible to adversarial examples. We propose a defense strategy that manipulates the latent space of novelty detectors to improve the robustness against adversarial examples.
arXiv Detail & Related papers (2021-08-25T10:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.