Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing

• URL: http://arxiv.org/abs/2409.16913v1
• Date: Wed, 25 Sep 2024 13:18:12 GMT
• Title: Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing
• Authors: Wenhao Liu, Siyu An, Junru Lu, Muling Wu, Tianlong Li, Xiaohua Wang, Xiaoqing Zheng, Di Yin, Xing Sun, Xuanjing Huang,
• Abstract summary: We develop an evaluation benchmark that includes contextual knowledge conflicting requests, parametric knowledge conflicting requests, and non-conflicting requests. We find that most RPAs behave significant performance gaps toward different conflict requests. We introduce a lightweight representation editing approach that conveniently shifts conflicting requests to the rejection region.
• Score: 54.098203568194606
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Role-Playing Agents (RPAs) have shown remarkable performance in various applications, yet they often struggle to recognize and appropriately respond to hard queries that conflict with their role-play knowledge. To investigate RPAs' performance when faced with different types of conflicting requests, we develop an evaluation benchmark that includes contextual knowledge conflicting requests, parametric knowledge conflicting requests, and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer appropriately without over-refusing. Through extensive evaluation, we find that most RPAs behave significant performance gaps toward different conflict requests. To elucidate the reasons, we conduct an in-depth representation-level analysis of RPAs under various conflict scenarios. Our findings reveal the existence of rejection regions and direct response regions within the model's forwarding representation, and thus influence the RPA's final response behavior. Therefore, we introduce a lightweight representation editing approach that conveniently shifts conflicting requests to the rejection region, thereby enhancing the model's refusal accuracy. The experimental results validate the effectiveness of our editing method, improving RPAs' refusal ability of conflicting requests while maintaining their general role-playing capabilities.

Related papers

• Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models [5.10832476049103]
  We identify three common scenarios-unanswerable, adversarial, conflicting-where retrieved document sets can confuse RALM with plausible real-world examples. We propose a new adversarial attack method, Generative model-based ADVersarial attack (GenADV) and a novel metric Robustness under Additional Document (RAD) Our findings reveal that RALMs often fail to identify the unanswerability or contradiction of a document set, which frequently leads to hallucinations.
  arXiv  Detail & Related papers  (2024-10-19T13:40:33Z)
• ECon: On the Detection and Resolution of Evidence Conflicts [56.89209046429291]
  The rise of large language models (LLMs) has significantly influenced the quality of information in decision-making systems. This study introduces a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios.
  arXiv  Detail & Related papers  (2024-10-05T07:41:17Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. Our research identifies two critical latent factors affecting RAG's confidence in its predictions. We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)
• Causal Influence in Federated Edge Inference [34.487472866247586]
  In this paper, we consider a setting where heterogeneous agents with connectivity are performing inference using unlabeled streaming data. In order to overcome the uncertainty, agents cooperate with each other by exchanging their local inferences with and through a fusion center. Various scenarios reflecting different agent participation patterns and fusion center policies are investigated.
  arXiv  Detail & Related papers  (2024-05-02T13:06:50Z)
• Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests [23.405917899107767]
  Transformer-based models fail to learn good policies for when to ask Instruction CRs. We discuss the shortcomings of the data-driven paradigm for learning meta-communication acts.
  arXiv  Detail & Related papers  (2024-01-30T14:18:31Z)
• Rehearsal: Simulating Conflict to Teach Conflict Resolution [54.32934135393982]
  Rehearsal is a system that allows users to rehearse conflicts with a believable simulated interlocutor. Users can utilize Rehearsal to practice handling a variety of predefined conflict scenarios. Rehearsal uses IRP to generate utterances grounded in conflict resolution theory.
  arXiv  Detail & Related papers  (2023-09-21T17:59:20Z)
• Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise [14.38859858538404]
  In a retrieved document set, even the "relevant" documents may contain misleading or incorrect information. Our work investigates a more challenging scenario in which even the "relevant" documents may contain misleading or incorrect information. We propose approaches for handling knowledge conflicts among retrieved documents by explicitly fine-tuning a discriminator or prompting GPT-3.5 to elicit its discriminative capability.
  arXiv  Detail & Related papers  (2023-05-02T16:28:10Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)
• Counterfactual Off-Policy Training for Neural Response Generation [94.76649147381232]
  We propose to explore potential responses by counterfactual reasoning. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model.
  arXiv  Detail & Related papers  (2020-04-29T22:46:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.