Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

• URL: http://arxiv.org/abs/2505.20871v1
• Date: Tue, 27 May 2025 08:21:21 GMT
• Title: Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG
• Authors: Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Weiqiang Wang, Zilei Wang, Liang Wang,
• Abstract summary: We propose Divide-Then-Align (DTA) to endow RAG systems with the ability to respond with "I don't know" when the query is out of the knowledge boundary.<n>DTA balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems.
• Score: 51.120170062795566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources, enabling more accurate and contextually rich responses. To improve the robustness of such systems against noisy retrievals, Retrieval-Augmented Fine-Tuning (RAFT) has emerged as a widely adopted method. However, RAFT conditions models to generate answers even in the absence of reliable knowledge. This behavior undermines their reliability in high-stakes domains, where acknowledging uncertainty is critical. To address this issue, we propose Divide-Then-Align (DTA), a post-training approach designed to endow RAG systems with the ability to respond with "I don't know" when the query is out of the knowledge boundary of both the retrieved passages and the model's internal knowledge. DTA divides data samples into four knowledge quadrants and constructs tailored preference data for each quadrant, resulting in a curated dataset for Direct Preference Optimization (DPO). Experimental results on three benchmark datasets demonstrate that DTA effectively balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems.

Related papers

• KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG [63.82127103851471]
  Retrieval-Augmented Generation (RAG) enables large language models to access broader knowledge sources.<n>We demonstrate that enhancing generative models' capacity to process noisy content is equally critical for robust performance.<n>We present KARE-RAG, which improves knowledge utilization through three key innovations.
  arXiv  Detail & Related papers  (2025-06-03T06:31:17Z)
• After Retrieval, Before Generation: Enhancing the Trustworthiness of Large Language Models in RAG [13.603907803297561]
  RAG systems face challenges in balancing internal (parametric) and external (retrieved) knowledge.<n>We propose the BRIDGE framework, which dynamically determines a comprehensive response strategy of large language models.<n>Experiments show BRIDGE outperforms baselines by 5-15% in accuracy while maintaining balanced performance across all scenarios.
  arXiv  Detail & Related papers  (2025-05-21T16:29:19Z)
• Know3-RAG: A Knowledge-aware RAG Framework with Adaptive Retrieval, Generation, and Filtering [9.814926166669366]
  We propose Know3-RAG, a knowledge-aware RAG framework that leverages structured knowledge from knowledge graphs (KGs) to guide three core stages of the RAG process, including retrieval, generation, and filtering.<n> Experiments on multiple open-domain QA benchmarks demonstrate that Know3-RAG consistently outperforms strong baselines, significantly reducing hallucinations and enhancing answer reliability.
  arXiv  Detail & Related papers  (2025-05-19T03:25:18Z)
• Trustworthy Answers, Messier Data: Bridging the Gap in Low-Resource Retrieval-Augmented Generation for Domain Expert Systems [7.76315323320043]
  We introduce a data generation pipeline that transforms raw multi-modal data into structured corpus and Q&A pairs.<n>Our system improves factual correctness (+1.94), informativeness (+1.16), and helpfulness (+1.67) over a non-RAG baseline.<n>Results highlight the effectiveness of our approach across distinct aspects, with strong answer grounding and transparency.
  arXiv  Detail & Related papers  (2025-02-26T22:20:08Z)
• ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation [91.20492150248106]
  We investigate the internal mechanisms behind unfaithful generation and identify a subset of mid-to-deep feed-forward networks (FFNs) that are disproportionately activated in such cases.<n>We propose Parametric Knowledge Muting through FFN Suppression (ParamMute), a framework that improves contextual faithfulness by suppressing the activation of unfaithfulness-associated FFNs.<n> Experimental results show that ParamMute significantly enhances faithfulness across both CoFaithfulQA and the established ConFiQA benchmark, achieving substantial reductions in reliance on parametric memory.
  arXiv  Detail & Related papers  (2025-02-21T15:50:41Z)
• RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects [12.5122702720856]
  We propose Robust Fine-Tuning (RbFT) to enhance the resilience of large language models against retrieval defects.<n> Experimental results demonstrate that RbFT significantly improves the robustness of RAG systems across diverse retrieval conditions.
  arXiv  Detail & Related papers  (2025-01-30T14:15:09Z)
• RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation [33.85528514353727]
  We introduce the Retrieval Preference Optimization (RPO) to adaptively leverage multi-source knowledge based on retrieval relevance.<n>RPO is the only RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training.<n>Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component.
  arXiv  Detail & Related papers  (2025-01-23T14:58:56Z)
• Semantic Tokens in Retrieval Augmented Generation [0.0]
  I propose a novel Comparative RAG system that introduces an evaluator module to bridge the gap between probabilistic RAG systems and deterministically verifiable responses.<n>This framework paves the way for more reliable and scalable question-answering applications in domains requiring high precision and verifiability.
  arXiv  Detail & Related papers  (2024-12-03T16:52:06Z)
• Trustworthiness in Retrieval-Augmented Generation Systems: A Survey [59.26328612791924]
  Retrieval-Augmented Generation (RAG) has quickly grown into a pivotal paradigm in the development of Large Language Models (LLMs) We propose a unified framework that assesses the trustworthiness of RAG systems across six key dimensions: factuality, robustness, fairness, transparency, accountability, and privacy.
  arXiv  Detail & Related papers  (2024-09-16T09:06:44Z)
• Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation [64.7982176398485]
  Retrieval-augmented generation (RAG) has demonstrated effectiveness in mitigating the hallucination problem of large language models (LLMs) We propose DPA-RAG, a universal framework designed to align diverse knowledge preferences within RAG systems.
  arXiv  Detail & Related papers  (2024-06-26T18:26:53Z)
• Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment [77.56326872997407]
  Pretrained language models (PLMs) based knowledge-grounded dialogue systems are prone to generate responses that are factually inconsistent with the provided knowledge source. Inspired by previous work which identified that feed-forward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability.
  arXiv  Detail & Related papers  (2023-10-12T14:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.