Agentic Verification for Ambiguous Query Disambiguation

• URL: http://arxiv.org/abs/2502.10352v1
• Date: Fri, 14 Feb 2025 18:31:39 GMT
• Title: Agentic Verification for Ambiguous Query Disambiguation
• Authors: Youngwon Lee, Seung-won Hwang, Ruofan Wu, Feng Yan, Danmei Xu, Moutasem Akkad, Zhewei Yao, Yuxiong He,
• Abstract summary: We tackle the challenge of disambiguating queries in retrieval-augmented generation (RAG) to diverse yet answerable interpretations. We propose a joint approach to unify diversification with verification by incorporating feedback from retriever and generator early on. We validate the efficiency and effectiveness of our method on the widely adopted ASQA benchmark to achieve diverse yet verifiable interpretations.
• Score: 42.238086712267396
• License:
• Abstract: In this work, we tackle the challenge of disambiguating queries in retrieval-augmented generation (RAG) to diverse yet answerable interpretations. State-of-the-arts follow a Diversify-then-Verify (DtV) pipeline, where diverse interpretations are generated by an LLM, later used as search queries to retrieve supporting passages. Such a process may introduce noise in either interpretations or retrieval, particularly in enterprise settings, where LLMs -- trained on static data -- may struggle with domain-specific disambiguations. Thus, a post-hoc verification phase is introduced to prune noises. Our distinction is to unify diversification with verification by incorporating feedback from retriever and generator early on. This joint approach improves both efficiency and robustness by reducing reliance on multiple retrieval and inference steps, which are susceptible to cascading errors. We validate the efficiency and effectiveness of our method, Verified-Diversification with Consolidation (VERDICT), on the widely adopted ASQA benchmark to achieve diverse yet verifiable interpretations. Empirical results show that VERDICT improves grounding-aware F1 score by an average of 23% over the strongest baseline across different backbone LLMs.

Related papers

• Inference Scaling for Bridging Retrieval and Augmented Generation [47.091086803980765]
  Retrieval-augmented generation (RAG) has emerged as a popular approach to steering the output of a large language model (LLM) We show such bias can be mitigated, from inference scaling, aggregating inference calls from the permuted order of retrieved contexts. We showcase the effectiveness of MOI on diverse RAG tasks, improving ROUGE-L on MS MARCO and EM on HotpotQA benchmarks by 7 points.
  arXiv  Detail & Related papers  (2024-12-14T05:06:43Z)
• Invar-RAG: Invariant LLM-aligned Retrieval for Better Generation [43.630437906898635]
  We propose a novel two-stage fine-tuning architecture called Invar-RAG. In the retrieval stage, an LLM-based retriever is constructed by integrating LoRA-based representation learning. In the generation stage, a refined fine-tuning method is employed to improve LLM accuracy in generating answers based on retrieved information.
  arXiv  Detail & Related papers  (2024-11-11T14:25:37Z)
• Retrieving, Rethinking and Revising: The Chain-of-Verification Can Improve Retrieval Augmented Generation [38.80878966092216]
  Recent Retrieval Augmented Generation (RAG) aims to enhance Large Language Models (LLMs) We propose the chain-of-verification (CoV-RAG) to enhance the external retrieval correctness and internal generation consistency.
  arXiv  Detail & Related papers  (2024-10-08T08:34:54Z)
• Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification [52.095460362197336]
  Large language models (LLMs) struggle with consistent and accurate reasoning. LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors. We propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification.
  arXiv  Detail & Related papers  (2024-10-05T05:21:48Z)
• Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering [45.154063285999015]
  The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems. RAG retrieves passages that cover all plausible interpretations and generates comprehensive responses. However, a single retrieval process often suffers from low quality results. We propose a diversify-verify-adapt (DIVA) framework to address this problem.
  arXiv  Detail & Related papers  (2024-09-04T01:14:04Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs [80.74263278847063]
  The integration of retrieved passages and large language models (LLMs) has significantly contributed to improving open-domain question answering. This paper investigates different methods of combining retrieved passages with LLMs to enhance answer generation.
  arXiv  Detail & Related papers  (2023-08-24T05:26:54Z)
• RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought [56.558892336235914]
  Reversing Chain-of-Thought (RCoT) is a novel method to improve large language models' reasoning abilities. RCoT automatically detects and rectifys factual inconsistency in generated solutions. We show that manually written fine-grained feedback can dramatically improve LLMs' reasoning abilities.
  arXiv  Detail & Related papers  (2023-05-19T08:02:52Z)
• LoL: A Comparative Regularization Loss over Query Reformulation Losses for Pseudo-Relevance Feedback [70.44530794897861]
  Pseudo-relevance feedback (PRF) has proven to be an effective query reformulation technique to improve retrieval accuracy. Existing PRF methods independently treat revised queries originating from the same query but using different numbers of feedback documents. We propose the Loss-over-Loss (LoL) framework to compare the reformulation losses between different revisions of the same query during training.
  arXiv  Detail & Related papers  (2022-04-25T10:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.