KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models

• URL: http://arxiv.org/abs/2408.03297v2
• Date: Mon, 19 Aug 2024 10:38:45 GMT
• Title: KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models
• Authors: Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, Yasha Wang,
• Abstract summary: We propose a Knowledge-aware Preference Optimization strategy, dubbed KnowPO, aimed at achieving adaptive knowledge selection. We show that KnowPO outperforms previous methods for handling knowledge conflicts by over 37%.
• Score: 14.057527352653787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By integrating external knowledge, Retrieval-Augmented Generation (RAG) has become an effective strategy for mitigating the hallucination problems that large language models (LLMs) encounter when dealing with knowledge-intensive tasks. However, in the process of integrating external non-parametric supporting evidence with internal parametric knowledge, inevitable knowledge conflicts may arise, leading to confusion in the model's responses. To enhance the knowledge selection of LLMs in various contexts, some research has focused on refining their behavior patterns through instruction-tuning. Nonetheless, due to the absence of explicit negative signals and comparative objectives, models fine-tuned in this manner may still exhibit undesirable behaviors such as contextual ignorance and contextual overinclusion. To this end, we propose a Knowledge-aware Preference Optimization strategy, dubbed KnowPO, aimed at achieving adaptive knowledge selection based on contextual relevance in real retrieval scenarios. Concretely, we proposed a general paradigm for constructing knowledge conflict datasets, which comprehensively cover various error types and learn how to avoid these negative signals through preference optimization methods. Simultaneously, we proposed a rewriting strategy and data ratio optimization strategy to address preference imbalances. Experimental results show that KnowPO outperforms previous methods for handling knowledge conflicts by over 37\%, while also exhibiting robust generalization across various out-of-distribution datasets.

Related papers

• Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment [56.87031484108484]
  Large Language Models (LLMs) are increasingly recognized for their practical applications. Retrieval-Augmented Generation (RAG) tackles this challenge and has shown a significant impact on LLMs. By minimizing retrieval requests that yield neutral or harmful results, we can effectively reduce both time and computational costs.
  arXiv  Detail & Related papers  (2024-11-09T15:12:28Z)
• Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuning [14.403812623299027]
  Retrieval-Augmented Generation (RAG) offers an effective solution to the issues faced by Large Language Models (LLMs) in hallucination generation and knowledge obsolescence. We propose Parenting, a novel framework that decouples, identifies, and purposefully optimize parameter subspaces related to adherence and robustness.
  arXiv  Detail & Related papers  (2024-10-14T10:26:57Z)
• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Pattern based learning and optimisation through pricing for bin packing problem [50.83768979636913]
  We argue that when problem conditions such as the distributions of random variables change, the patterns that performed well in previous circumstances may become less effective. We propose a novel scheme to efficiently identify patterns and dynamically quantify their values for each specific condition. Our method quantifies the value of patterns based on their ability to satisfy constraints and their effects on the objective value.
  arXiv  Detail & Related papers  (2024-08-27T17:03:48Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questions [0.46873264197900916]
  This paper examines the role of cognitive biases in the decision-making processes of large language models (LLMs) We show that certain cognitive biases when properly balanced, can enhance decision-making efficiency through rational deviations and shortcuts.
  arXiv  Detail & Related papers  (2024-06-16T16:25:22Z)
• READ: Improving Relation Extraction from an ADversarial Perspective [33.44949503459933]
  We propose an adversarial training method specifically designed for relation extraction (RE) Our approach introduces both sequence- and token-level perturbations to the sample and uses a separate perturbation vocabulary to improve the search for entity and context perturbations.
  arXiv  Detail & Related papers  (2024-04-02T16:42:44Z)
• Generative Intrinsic Optimization: Intrinsic Control with Model Learning [5.439020425819001]
  Future sequence represents the outcome after executing the action into the environment. Explicit outcomes may vary across state, return, or trajectory serving different purposes such as credit assignment or imitation learning. We propose a policy scheme that seamlessly incorporates the mutual information, ensuring convergence to the optimal policy.
  arXiv  Detail & Related papers  (2023-10-12T07:50:37Z)
• Confounder Identification-free Causal Visual Feature Learning [84.28462256571822]
  We propose a novel Confounder Identification-free Causal Visual Feature Learning (CICF) method, which obviates the need for identifying confounders. CICF models the interventions among different samples based on front-door criterion, and then approximates the global-scope intervening effect upon the instance-level interventions. We uncover the relation between CICF and the popular meta-learning strategy MAML, and provide an interpretation of why MAML works from the theoretical perspective.
  arXiv  Detail & Related papers  (2021-11-26T10:57:47Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.