Related papers: Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators

Related papers

When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems [0.5161531917413708]
Large language models are increasingly integrated into autonomous robotic systems for task planning and control.<n>This integration exposes them to prompt injection attacks that can lead to unsafe decisions and physical harm.<n>In this paper, we evaluate prompt injection attacks against an LLM-based multi-agent robotic system.
arXiv Detail & Related papers (2026-08-01T16:31:00Z)
Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models [57.870323273127234]
Motivated by the observation, we propose textbfAttention Hijacking, a novel adversarial attack that steers internal attention toward a persistent image-dominant pattern.<n>Our method reduces the dependence of the manipulated output on the specific wording of the query.
arXiv Detail & Related papers (2026-05-17T08:02:27Z)
When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis [0.0]
This paper provides the first systematic empirical test of the assumption that models will reliably maintain their assigned roles.<n>We measure role fidelity across 60 political statements using four metrics: Role Drift Index (RDI), Drift Distance (EDD), Directional Drift Index (DDI), and Entropy-based Role Stability (ERS)<n>We identify two failure modes - the Epistemic Floor Effect (fact-check results create an absolute lower bound below which the legitimizing role cannot be maintained) and Role-Prior Conflict (training-time knowledge overrides role instructions for factually unambiguous statements)
arXiv Detail & Related papers (2026-04-29T21:50:28Z)
Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
arXiv Detail & Related papers (2026-04-21T15:05:58Z)
Prompt Injection as Role Confusion [6.379494871147752]
Language models remain vulnerable to prompt injection attacks despite extensive safety training.<n>We design novel role probes to capture how models internally identify "who is speaking"<n>These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority.
arXiv Detail & Related papers (2026-02-22T18:43:34Z)
How role-play shapes relevance judgment in zero-shot LLM rankers [15.11127856890218]
Large Language Models (LLMs) have emerged as promising zero-shot rankers.<n>Their performance is highly sensitive to prompt formulation.<n>In particular, role-play prompts, where the model is assigned a functional role or identity, often give more robust and accurate relevance rankings.
arXiv Detail & Related papers (2025-10-20T13:39:48Z)
D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models [62.83226685925107]
Deceptive Reasoning Exposure Suite (D-REX) is a novel dataset designed to evaluate the discrepancy between a model's internal reasoning process and its final output.<n>Each sample in D-REX contains the adversarial system prompt, an end-user's test query, the model's seemingly innocuous response, and, crucially, the model's internal chain-of-thought.<n>We demonstrate that D-REX presents a significant challenge for existing models and safety mechanisms.
arXiv Detail & Related papers (2025-09-22T15:59:40Z)
Can an Individual Manipulate the Collective Decisions of Multi-Agents? [53.01767232004823]
M-Spoiler is a framework that simulates agent interactions within a multi-agent system to generate adversarial samples.<n>M-Spoiler introduces a stubborn agent that actively aids in optimizing adversarial samples.<n>Our findings confirm the risks posed by the knowledge of an individual agent in multi-agent systems.
arXiv Detail & Related papers (2025-09-20T01:54:20Z)
Preliminary Investigation into Uncertainty-Aware Attack Stage Classification [81.28215542218724]
This work addresses the problem of attack stage inference under uncertainty.<n>We propose a classification approach based on Evidential Deep Learning (EDL), which models predictive uncertainty by outputting parameters of a Dirichlet distribution over possible stages.<n>Preliminary experiments in a simulated environment demonstrate that the proposed model can accurately infer the stage of an attack with confidence.
arXiv Detail & Related papers (2025-08-01T06:58:00Z)
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition [71.81906608221038]
Large language models (LLMs) are vulnerable to indirect prompt injection attacks.<n>We propose TopicAttack, which prompts the LLM to generate a fabricated transition prompt that gradually shifts the topic toward the injected instruction.<n>We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.
arXiv Detail & Related papers (2025-07-18T06:23:31Z)
Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Study [5.843063647136238]
This study systematically evaluates five leading LALMs across four attack scenarios.<n>No single model consistently outperforms others across all attack types.<n>A negative correlation between instruction-following capability and robustness suggests models adhering strictly to instructions may be more susceptible.
arXiv Detail & Related papers (2025-05-26T07:08:38Z)
The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) [15.48684126686974]
We show that fine-tuned large language models often rely on two proxies for role identification. We propose reinforcing emphinvariant signals that mark role boundaries by adjusting token-wise cues in the model's input encoding.
arXiv Detail & Related papers (2025-05-01T16:06:16Z)
Towards Robust Dialogue Breakdown Detection: Addressing Disruptors in Large Language Models with Self-Guided Reasoning [30.13634341221476]
Large language models (LLMs) are rapidly changing various domains. This paper addresses the challenge of detecting and mitigating dialogue breakdowns within LLM-driven systems. We propose an approach that combines specialized fine-tuning with advanced prompting strategies.
arXiv Detail & Related papers (2025-04-26T07:51:05Z)
Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots [6.302734977940601]
We introduce Role-Modality Attacks (RMA), a class of adversarial attacks that exploit role confusion between the user and assistant.<n>RMAs manipulate the input structure without altering the query itself.<n>We propose an adversarial training approach that makes the model robust against input prompt perturbations.
arXiv Detail & Related papers (2025-04-01T03:54:36Z)
MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks [85.3303135160762]
MIRAGE is a novel framework that exploits narrative-driven context and role immersion to circumvent safety mechanisms in Multimodal Large Language Models.<n>It achieves state-of-the-art performance, improving attack success rates by up to 17.5% over the best baselines.<n>We demonstrate that role immersion and structured semantic reconstruction can activate inherent model biases, facilitating the model's spontaneous violation of ethical safeguards.
arXiv Detail & Related papers (2025-03-24T20:38:42Z)
ASIDE: Architectural Separation of Instructions and Data in Language Models [87.16417239344285]
We propose a method, ASIDE, that allows the model to clearly separate between instructions and data on the level of embeddings. ASIDE applies a fixed rotation to the embeddings of data tokens, thus creating distinct representations of instructions and data tokens without introducing any additional parameters. We demonstrate the effectiveness of our method by instruct-tuning LLMs with ASIDE and showing (1) highly increased instruction-data separation scores without a loss in model capabilities and (2) competitive results on prompt injection benchmarks, even without dedicated safety training.
arXiv Detail & Related papers (2025-03-13T17:17:17Z)
A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition [71.61103962200666]
Zero-shot named entity recognition (NER) aims to develop entity recognition systems from unannotated text corpora.<n>Recent work has adapted large language models (LLMs) for zero-shot NER by crafting specialized prompt templates.<n>We introduce the cooperative multi-agent system (CMAS), a novel framework for zero-shot NER.
arXiv Detail & Related papers (2025-02-25T23:30:43Z)
Coherence-Driven Multimodal Safety Dialogue with Active Learning for Embodied Agents [23.960719833886984]
M-CoDAL is a multimodal-dialogue system specifically designed for embodied agents to better understand and communicate in safety-critical situations.<n>Our approach is evaluated using a newly created multimodal dataset comprising 1K safety violations extracted from 2K Reddit images.<n>Results with this dataset demonstrate that our approach improves resolution of safety situations, user sentiment, as well as safety of the conversation.
arXiv Detail & Related papers (2024-10-18T03:26:06Z)
Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas [7.677029165197536]
Role-play in large language models (LLMs) enhances their ability to generate contextually relevant and high-quality responses.<n>We demonstrate that autotuning, a method used to auto-select models' roles based on the question, can lead to the generation of harmful outputs.
arXiv Detail & Related papers (2024-09-21T02:09:13Z)
Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models.<n>We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks.<n>We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
arXiv Detail & Related papers (2024-05-28T19:16:17Z)
Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
arXiv Detail & Related papers (2024-04-08T14:15:56Z)
Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks [72.03945355787776]
We advocate MDP, a lightweight, pluggable, and effective defense for PLMs as few-shot learners. We show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness.
arXiv Detail & Related papers (2023-09-23T04:41:55Z)
Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
arXiv Detail & Related papers (2021-06-17T14:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.