Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2501.15228v1
• Date: Sat, 25 Jan 2025 14:24:50 GMT
• Title: Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
• Authors: Yiqun Chen, Lingyong Yan, Weiwei Sun, Xinyu Ma, Yi Zhang, Shuaiqiang Wang, Dawei Yin, Yiming Yang, Jiaxin Mao,
• Abstract summary: Retrieval-augmented generation (RAG) is extensively utilized to incorporate external, current knowledge into large language models. A standard RAG pipeline may comprise several components, such as query rewriting, document retrieval, document filtering, and answer generation. To overcome these challenges, we propose treating the RAG pipeline as a multi-agent cooperative task, with each component regarded as an RL agent.
• Score: 51.54046200512198
• License:
• Abstract: Retrieval-augmented generation (RAG) is extensively utilized to incorporate external, current knowledge into large language models, thereby minimizing hallucinations. A standard RAG pipeline may comprise several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically optimized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual modules and the overarching aim of generating accurate answers in question-answering (QA) tasks. Although recent efforts have explored reinforcement learning (RL) to optimize specific RAG components, these approaches often focus on overly simplistic pipelines with only two components or do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these challenges, we propose treating the RAG pipeline as a multi-agent cooperative task, with each component regarded as an RL agent. Specifically, we present MMOA-RAG, a Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent reinforcement learning to harmonize all agents' goals towards a unified reward, such as the F1 score of the final answer. Experiments conducted on various QA datasets demonstrate that MMOA-RAG improves the overall pipeline performance and outperforms existing baselines. Furthermore, comprehensive ablation studies validate the contributions of individual components and the adaptability of MMOA-RAG across different RAG components and datasets. The code of MMOA-RAG is on https://github.com/chenyiqun/MMOA-RAG.

Related papers

• C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation [13.120930059424975]
  C-3PO is a proxy-centric framework that facilitates communication between retrievers and large language models. Our framework implements three specialized agents that collaboratively optimize the entire RAG pipeline.
  arXiv  Detail & Related papers  (2025-02-10T07:04:32Z)
• Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering [47.29580414645626]
  RopMura is a novel multi-agent system that integrates multiple knowledge bases into a unified RAG-based agent. RopMura features two key components: a router that intelligently selects the most relevant agents based on knowledge boundaries and a planner that decomposes complex multi-hop queries into manageable steps.
  arXiv  Detail & Related papers  (2025-01-14T03:25:26Z)
• MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation [34.66546005629471]
  Large Language Models (LLMs) are essential tools for various natural language processing tasks but often suffer from generating outdated or incorrect information. Retrieval-Augmented Generation (RAG) addresses this issue by incorporating external, real-time information retrieval to ground LLM responses. To tackle this problem, we propose Multi-Agent Filtering Retrieval-Augmented Generation (MAIN-RAG) MAIN-RAG is a training-free RAG framework that leverages multiple LLM agents to collaboratively filter and score retrieved documents.
  arXiv  Detail & Related papers  (2024-12-31T08:07:26Z)
• mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA [78.45521005703958]
  multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge. We propose a novel framework called textbfRetrieval-textbfReftextbfAugmented textbfGeneration (mR$2$AG) which achieves adaptive retrieval and useful information localization. mR$2$AG significantly outperforms state-of-the-art MLLMs on INFOSEEK and Encyclopedic-VQA
  arXiv  Detail & Related papers  (2024-11-22T16:15:50Z)
• RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards [78.74923079748521]
  Retrieval-Augmented Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) Current approaches use instruction tuning to optimize LLMs, improving their ability to utilize retrieved knowledge. We propose a Differentiable Data Rewards ( DDR) method, which trains RAG systems by aligning data preferences between different RAG modules.
  arXiv  Detail & Related papers  (2024-10-17T12:53:29Z)
• Self-adaptive Multimodal Retrieval-Augmented Generation [0.0]
  We propose a new approach called Self-adaptive Multimodal Retrieval-Augmented Generation (SAM-RAG) SAM-RAG not only dynamically filters relevant documents based on the input query, including image captions when needed, but also verifies the quality of both the retrieved documents and the output. Extensive experimental results show that SAM-RAG surpasses existing state-of-the-art methods in both retrieval accuracy and response generation.
  arXiv  Detail & Related papers  (2024-10-15T06:39:35Z)
• Multi-Head RAG: Solving Multi-Aspect Problems with LLMs [13.638439488923671]
  Retrieval Augmented Generation (RAG) enhances the abilities of Large Language Models (LLMs) Existing RAG solutions do not focus on queries that may require fetching multiple documents with substantially different contents. This paper introduces Multi-Head RAG (MRAG), a novel scheme designed to address this gap with a simple yet powerful idea.
  arXiv  Detail & Related papers  (2024-06-07T16:59:38Z)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity [59.57065228857247]
  Retrieval-augmented Large Language Models (LLMs) have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA) We propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs based on the query complexity. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems.
  arXiv  Detail & Related papers  (2024-03-21T13:52:30Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• Plug-and-Play Regulators for Image-Text Matching [76.28522712930668]
  Exploiting fine-grained correspondence and visual-semantic alignments has shown great potential in image-text matching. We develop two simple but quite effective regulators which efficiently encode the message output to automatically contextualize and aggregate cross-modal representations. Experiments on MSCOCO and Flickr30K datasets validate that they can bring an impressive and consistent R@1 gain on multiple models.
  arXiv  Detail & Related papers  (2023-03-23T15:42:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.