ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labeling

• URL: http://arxiv.org/abs/2402.13542v2
• Date: Tue, 4 Jun 2024 05:17:24 GMT
• Title: ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labeling
• Authors: Lingxi Zhang, Yue Yu, Kuan Wang, Chao Zhang,
• Abstract summary: ARL2 is a retriever learning technique that harnesses large language models as labelers. ARL2 uses an adaptive self-training strategy for curating high-quality and diverse relevance data. Experiments demonstrate the effectiveness of ARL2, achieving accuracy improvements of 5.4% on NQ and 4.6% on MMLU.
• Score: 20.022332182475672
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation enhances large language models (LLMs) by incorporating relevant information from external knowledge sources. This enables LLMs to adapt to specific domains and mitigate hallucinations in knowledge-intensive tasks. However, existing retrievers are often misaligned with LLMs due to their separate training processes and the black-box nature of LLMs. To address this challenge, we propose ARL2, a retriever learning technique that harnesses LLMs as labelers. ARL2 leverages LLMs to annotate and score relevant evidence, enabling learning the retriever from robust LLM supervision. Furthermore, ARL2 uses an adaptive self-training strategy for curating high-quality and diverse relevance data, which can effectively reduce the annotation cost. Extensive experiments demonstrate the effectiveness of ARL2, achieving accuracy improvements of 5.4% on NQ and 4.6% on MMLU compared to the state-of-the-art methods. Additionally, ARL2 exhibits robust transfer learning capabilities and strong zero-shot generalization abilities. Our code will be published at \url{https://github.com/zhanglingxi-cs/ARL2}.

Related papers

• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning [50.419872452397684]
  Search-R1 is an extension of reinforcement learning for reasoning frameworks. It generates search queries during step-by-step reasoning with real-time retrieval. It improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines.
  arXiv  Detail & Related papers  (2025-03-12T16:26:39Z)
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning [87.30285670315334]
  textbfR1-Searcher is a novel two-stage outcome-based RL approach designed to enhance the search capabilities of Large Language Models. Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start. Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
  arXiv  Detail & Related papers  (2025-03-07T17:14:44Z)
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers [86.54316283425001]
  Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. LLMs offer better efficiency, but they often fail to generalize effectively with limited supervised fine-tuning data. We introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers.
  arXiv  Detail & Related papers  (2025-02-25T18:59:07Z)
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution [46.5893728376551]
  This paper introduces SWE-RL, the first approach to scale RL-based large language models (LLMs) for real-world software engineering. Llama3-SWE-RL-70B achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills.
  arXiv  Detail & Related papers  (2025-02-25T18:45:04Z)
• Reinforcement Learning Enhanced LLMs: A Survey [45.57586245741664]
  We will make a systematic review of the most up-to-date state of knowledge on RL-enhanced large language models (LLMs) Specifically, we detail the basics of RL; (2) introduce popular RL-enhanced LLMs; (3) review researches on two widely-used reward model-based RL techniques: Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF)
  arXiv  Detail & Related papers  (2024-12-05T16:10:42Z)
• Invar-RAG: Invariant LLM-aligned Retrieval for Better Generation [43.630437906898635]
  We propose a novel two-stage fine-tuning architecture called Invar-RAG. In the retrieval stage, an LLM-based retriever is constructed by integrating LoRA-based representation learning. In the generation stage, a refined fine-tuning method is employed to improve LLM accuracy in generating answers based on retrieved information.
  arXiv  Detail & Related papers  (2024-11-11T14:25:37Z)
• Fine-Grained Guidance for Retrievers: Leveraging LLMs' Feedback in Retrieval-Augmented Generation [20.420575358183687]
  Retrieval-Augmented Generation (RAG) has proven to be an effective method for mitigating hallucination issues inherent in large language models (LLMs) Previous approaches typically train retrievers based on semantic similarity, lacking optimization for RAG. We propose a novel framework, FiGRet, which leverages the language capabilities of LLMs to construct examples from a more granular, information-centric perspective.
  arXiv  Detail & Related papers  (2024-11-06T14:42:39Z)
• Dynamic Uncertainty Ranking: Enhancing In-Context Learning for Long-Tail Knowledge in LLMs [50.29035873837]
  Large language models (LLMs) can learn vast amounts of knowledge from diverse domains during pre-training. Long-tail knowledge from specialized domains is often scarce and underrepresented, rarely appearing in the models' memorization. We propose a reinforcement learning-based dynamic uncertainty ranking method for ICL that accounts for the varying impact of each retrieved sample on LLM predictions.
  arXiv  Detail & Related papers  (2024-10-31T03:42:17Z)
• Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval [7.441679541836913]
  textit Intermediate Distillation treats large language models as black boxes and distills their knowledge via an innovative LLM-ranker-retriever pipeline. Our proposed method can significantly improve the performance of retriever models with only 1,000 training instances.
  arXiv  Detail & Related papers  (2024-06-18T00:41:41Z)
• Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation [128.01050030936028]
  We propose an information refinement training method named InFO-RAG. InFO-RAG is low-cost and general across various tasks. It improves the performance of LLaMA2 by an average of 9.39% relative points.
  arXiv  Detail & Related papers  (2024-02-28T08:24:38Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation [66.01754585188739]
  Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence.
  arXiv  Detail & Related papers  (2024-02-18T04:57:19Z)
• The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models [2.5721733711031978]
  We review research studies that combine Reinforcement Learning (RL) and Large Language Models (LLMs) We propose a novel taxonomy of three main classes based on the way that the two model types interact with each other.
  arXiv  Detail & Related papers  (2024-02-02T20:01:15Z)
• Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Case Study [1.3597551064547502]
  We employ a teacher-student learning framework to tackle problems of Large Language Models (LLMs) and reinforcement learning (RL) models. Within this framework, the LLM acts as a teacher, while the RL model acts as a student. We propose a practical algorithm to address the problem and conduct empirical experiments to evaluate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-01-12T14:35:57Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.