Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation

• URL: http://arxiv.org/abs/2405.03085v1
• Date: Mon, 6 May 2024 00:18:43 GMT
• Title: Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation
• Authors: Kaize Shi, Xueyao Sun, Qing Li, Guandong Xu,
• Abstract summary: Large Language Models (LLMs) have made significant strides in information acquisition. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. We propose a novel concept-based RAG framework with the Abstract Representation (AMR)-based concept distillation algorithm.
• Score: 17.156915103545728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have made significant strides in information acquisition. However, their overreliance on potentially flawed parametric knowledge leads to hallucinations and inaccuracies, particularly when handling long-tail, domain-specific queries. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. Nevertheless, the retrieved long-context documents often contain noisy, irrelevant information alongside vital knowledge, negatively diluting LLMs' attention. Inspired by the supportive role of essential concepts in individuals' reading comprehension, we propose a novel concept-based RAG framework with the Abstract Meaning Representation (AMR)-based concept distillation algorithm. The proposed algorithm compresses the cluttered raw retrieved documents into a compact set of crucial concepts distilled from the informative nodes of AMR by referring to reliable linguistic features. The concepts explicitly constrain LLMs to focus solely on vital information in the inference process. We conduct extensive experiments on open-domain question-answering datasets to empirically evaluate the proposed method's effectiveness. The results indicate that the concept-based RAG framework outperforms other baseline methods, particularly as the number of supporting documents increases, while also exhibiting robustness across various backbone LLMs. This emphasizes the distilled concepts are informative for augmenting the RAG process by filtering out interference information. To the best of our knowledge, this is the first work introducing AMR to enhance the RAG, presenting a potential solution to augment inference performance with semantic-based context compression.

Related papers

• Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation [72.70046559930555]
  We propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes. In addition, we employ an adaptive, note-based stop-exploration strategy to decide "what to retrieve and when to stop" to encourage sufficient knowledge exploration.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• How Much Can RAG Help the Reasoning of LLM? [9.601957219734683]
  Retrieval-Augmented Generation (RAG) has gained significant popularity in modern Large Language Models (LLMs) How does RAG help the reasoning process and can RAG help improve the reasoning capability remains question.
  arXiv  Detail & Related papers  (2024-10-03T09:48:09Z)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting [68.90949377014742]
  Speculative RAG is a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. It notably enhances accuracy by up to 12.97% while reducing latency by 51% compared to conventional RAG systems on PubHealth.
  arXiv  Detail & Related papers  (2024-07-11T06:50:19Z)
• Entropy-Based Decoding for Retrieval-Augmented Large Language Models [43.93281157539377]
  Augmenting Large Language Models with retrieved external knowledge has proven effective for improving the factual accuracy of generated responses. We introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue.
  arXiv  Detail & Related papers  (2024-06-25T12:59:38Z)
• Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning [49.3242278912771]
  We introduce a novel multimodal RAG framework named RMR (Retrieval Meets Reasoning) The RMR framework employs a bi-modal retrieval module to identify the most relevant question-answer pairs. It significantly boosts the performance of various vision-language models across a spectrum of benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-31T14:23:49Z)
• IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues [10.280113107290067]
  The IM-RAG approach integrates Information Retrieval systems with Large Language Models (LLMs) to support multi-round RAG. The entire IM process is optimized via Reinforcement Learning (RL) where a Progress Tracker is incorporated to provide mid-step rewards. The results show that our approach achieves state-of-the-art (SOTA) performance while providing high flexibility in integrating IR modules.
  arXiv  Detail & Related papers  (2024-05-15T12:41:20Z)
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering [122.62012375722124]
  In existing methods, large language models (LLMs) cannot precisely assess the relevance of retrieved documents. We propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA)
  arXiv  Detail & Related papers  (2024-02-27T13:22:51Z)
• Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition [57.08108545219043]
  Recognizing novel sub-categories with scarce samples is an essential and challenging research topic in computer vision. Existing literature addresses this challenge by employing local-based representation approaches. This article proposes a novel model, Robust Saliency-aware Distillation (RSaD), for few-shot fine-grained visual recognition.
  arXiv  Detail & Related papers  (2023-05-12T00:13:17Z)
• Entity Concept-enhanced Few-shot Relation Extraction [35.10974511223129]
  Few-shot relation extraction (FSRE) is of great importance in long-tail distribution problem. Most existing FSRE algorithms fail to accurately classify the relations merely based on the information of the sentences together with the recognized entity pairs. We propose a novel entity-enhanced FEw-shot Relation Extraction scheme (ConceptFERE), which introduces the inherent concepts of entities to provide clues for relation prediction.
  arXiv  Detail & Related papers  (2021-06-04T10:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.