RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

• URL: http://arxiv.org/abs/2409.16727v1
• Date: Wed, 25 Sep 2024 08:23:46 GMT
• Title: RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems
• Abstract summary: Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. These systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework.
• Score: 20.786294377706717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.

Related papers

• Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization [62.82109000444373]
  Psy-CoT is a chain-of-thought framework that decomposes pre-response reasoning into three role-specific steps.<n>Experiments on CoSER, CharacterBench, and CharacterEval demonstrate that Psy-CoT outperforms existing role-playing CoT methods.
  arXiv  Detail & Related papers  (2026-06-25T13:34:47Z)
• The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMs [1.3112847740767446]
  Narrative conventions embedded in human-authored text are rarely examined as a source of systematic behavioral influence.<n>This paper considers whether the storytelling patterns inherent in published human writing, including archetypal roles such as protagonist, antagonist, and underdog, are absorbed during training.<n>The findings reveal three key patterns. First, LLMs reproduce statistical patterns from their training data rather than reasoning independently. Second, measurable latent traits, including sycophancy and deceptiveness, emerge reliably across unrelated prompts. Third, fine-tuning on a narrow narrative task can produce unintended behavioral changes well beyond that task.
  arXiv  Detail & Related papers  (2026-05-13T22:22:04Z)
• Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning [57.22705949022221]
  RoleJudge is an evaluation framework that leverages audio large language models to assess the alignment between speech and character.<n>We introduce RoleChat, the first voice role-playing evaluation dataset enriched with chain-of-thought reasoning annotations.
  arXiv  Detail & Related papers  (2026-04-15T12:39:03Z)
• A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents [14.1714863221125]
  We present the first systematic study of cross-feature stylistic side effects.<n>We quantify how prompting for one style feature causally affects others via a pairwise Large Language Model evaluation framework.<n>Our results reveal consistent and structured side effects, such as prompting for conciseness significantly reduces perceived expertise.
  arXiv  Detail & Related papers  (2026-01-15T19:18:18Z)
• Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels [50.43968216132018]
  We present an end-to-end system that transforms any literary work into an immersive, multi-character conversational experience.<n>This system is designed to solve two fundamental challenges for LLM-driven characters.
  arXiv  Detail & Related papers  (2025-12-08T11:57:46Z)
• Review of Hallucination Understanding in Large Language and Vision Models [65.29139004945712]
  We present a framework for characterizing both image and text hallucinations across diverse applications.<n>Our investigations reveal that hallucinations often stem from predictable patterns in data distributions and inherited biases.<n>This survey provides a foundation for developing more robust and effective solutions to hallucinations in real-world generative AI systems.
  arXiv  Detail & Related papers  (2025-09-26T09:23:08Z)
• SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents [52.29009595100625]
  Role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance.<n>Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios.<n>We construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations.
  arXiv  Detail & Related papers  (2025-08-04T03:18:36Z)
• HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models [30.596530112268848]
  We present the first systematic study of hallucinations in large language models performing long-horizon tasks under scene-task inconsistencies.<n>Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond.
  arXiv  Detail & Related papers  (2025-06-18T02:13:41Z)
• MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks [85.3303135160762]
  MIRAGE is a novel framework that exploits narrative-driven context and role immersion to circumvent safety mechanisms in Multimodal Large Language Models. It achieves state-of-the-art performance, improving attack success rates by up to 17.5% over the best baselines. We demonstrate that role immersion and structured semantic reconstruction can activate inherent model biases, facilitating the model's spontaneous violation of ethical safeguards.
  arXiv  Detail & Related papers  (2025-03-24T20:38:42Z)
• Towards Enhanced Immersion and Agency for LLM-based Interactive Drama [55.770617779283064]
  This paper begins with understanding interactive drama from two aspects: Immersion, the player's feeling of being present in the story, and Agency. To enhance these two aspects, we first propose Playwriting-guided Generation, a novel method that helps LLMs craft dramatic stories with substantially improved structures and narrative quality.
  arXiv  Detail & Related papers  (2025-02-25T06:06:16Z)
• Eliciting Language Model Behaviors with Investigator Agents [93.34072434845162]
  Language models exhibit complex, diverse behaviors when prompted with free-form text. We study the problem of behavior elicitation, where the goal is to search for prompts that induce specific target behaviors. We train investigator models to map randomly-chosen target behaviors to a diverse distribution of outputs that elicit them.
  arXiv  Detail & Related papers  (2025-02-03T10:52:44Z)
• CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds [74.02480671181685]
  Role-playing is a crucial capability of Large Language Models (LLMs) Current evaluation methods fall short of adequately capturing the nuanced character traits and behaviors essential for authentic role-playing. We propose CharacterBox, a simulation sandbox designed to generate situational fine-grained character behavior trajectories.
  arXiv  Detail & Related papers  (2024-12-07T12:09:35Z)
• SHARP: Unlocking Interactive Hallucination via Stance Transfer in Role-Playing Agents [12.990119925990477]
  We propose a generalizable, explicit and effective paradigm to unlock the interactive patterns in diverse worldviews. Specifically, we define the interactive hallucination based on stance transfer and construct a benchmark, SHARP, by extracting relations from a general commonsense knowledge graph. Our findings explore the factors influencing these metrics and discuss the trade-off between blind loyalty to roles and adherence to facts in RPAs.
  arXiv  Detail & Related papers  (2024-11-12T17:41:16Z)
• Mitigating Hallucination in Fictional Character Role-Play [19.705708068900076]
  We focus on the evaluation and mitigation of hallucination in fictional character role-play. We introduce a dataset with over 2,000 characters and 72,000 interviews, including 18,000 adversarial questions. We propose RoleFact, a role-playing method that mitigates hallucination by modulating the influence of parametric knowledge.
  arXiv  Detail & Related papers  (2024-06-25T03:56:33Z)
• TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models [55.51648393234699]
  We introduce TimeChara, a new benchmark designed to evaluate point-in-time character hallucination in role-playing LLMs. We propose Narrative-Experts, a method that decomposes the reasoning steps and utilizes narrative experts to reduce point-in-time character hallucinations effectively.
  arXiv  Detail & Related papers  (2024-05-28T10:19:18Z)
• A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generation [51.53917938874146]
  We propose a possible solution for alleviating the hallucination in KGD by exploiting the dialogue-knowledge interaction. Experimental results of our example implementation show that this method can reduce hallucination without disrupting other dialogue performance.
  arXiv  Detail & Related papers  (2024-04-04T14:45:26Z)
• Affective and Dynamic Beam Search for Story Generation [50.3130767805383]
  We propose Affective Story Generator (AffGen) for generating interesting narratives. AffGen employs two novel techniques-Dynamic Beam Sizing and Affective Reranking.
  arXiv  Detail & Related papers  (2023-10-23T16:37:14Z)
• Investigating Human-Identifiable Features Hidden in Adversarial Perturbations [54.39726653562144]
  Our study explores up to five attack algorithms across three datasets. We identify human-identifiable features in adversarial perturbations. Using pixel-level annotations, we extract such features and demonstrate their ability to compromise target models.
  arXiv  Detail & Related papers  (2023-09-28T22:31:29Z)
• Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing [19.589945994234075]
  We revisit a widely used conceptualization of framing from the communication sciences which explicitly captures elements of narratives. We adapt an effective annotation paradigm that breaks a complex annotation task into a series of simpler binary questions. We explore automatic multi-label prediction of our frames with supervised and semi-supervised approaches.
  arXiv  Detail & Related papers  (2023-06-03T08:50:13Z)
• M-SENSE: Modeling Narrative Structure in Short Personal Narratives Using Protagonist's Mental Representations [14.64546899992196]
  We propose the task of automatically detecting prominent elements of the narrative structure by analyzing the role of characters' inferred mental state. We introduce a STORIES dataset of short personal narratives containing manual annotations of key elements of narrative structure, specifically climax and resolution. Our model is able to achieve significant improvements in the task of identifying climax and resolution.
  arXiv  Detail & Related papers  (2023-02-18T20:48:02Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)
• Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News [57.9843300852526]
  We introduce the more realistic and challenging task of defending against machine-generated news that also includes images and captions. To identify the possible weaknesses that adversaries can exploit, we create a NeuralNews dataset composed of 4 different types of generated articles. In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies.
  arXiv  Detail & Related papers  (2020-09-16T14:13:15Z)
• Once Upon A Time In Visualization: Understanding the Use of Textual Narratives for Causality [21.67542584041709]
  Causality visualization can help people understand temporal chains of events. But as the scale and complexity of these event sequences grows, even these visualizations can become overwhelming to use. We propose the use of textual narratives as a data-driven storytelling method to augment causality visualization.
  arXiv  Detail & Related papers  (2020-09-06T05:46:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.