Meta-prompting Optimized Retrieval-augmented Generation

• URL: http://arxiv.org/abs/2407.03955v1
• Date: Thu, 4 Jul 2024 14:20:12 GMT
• Title: Meta-prompting Optimized Retrieval-augmented Generation
• Authors: João Rodrigues, António Branco,
• Abstract summary: Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. We propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization.
• Score: 1.3996171129586734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible dispersion of its parts, or their out of focus range may happen nevertheless to eventually have a detrimental rather than an incremental effect. To mitigate this issue and improve retrieval-augmented generation, we propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization. Put to empirical test with the demanding multi-hop question answering task from the StrategyQA dataset, the evaluation results indicate that this method outperforms a similar retrieval-augmented system but without this method by over 30%.

Related papers

• Unsupervised Query Routing for Retrieval Augmented Generation [64.47987041500966]
  We introduce a novel unsupervised method that constructs the "upper-bound" response to evaluate the quality of retrieval-augmented responses. This evaluation enables the decision of the most suitable search engine for a given query. By eliminating manual annotations, our approach can automatically process large-scale real user queries and create training data.
  arXiv  Detail & Related papers  (2025-01-14T02:27:06Z)
• SUGAR: Leveraging Contextual Confidence for Smarter Retrieval [28.552283701883766]
  We introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR) We leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.
  arXiv  Detail & Related papers  (2025-01-09T01:24:59Z)
• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Decompose, Enrich, and Extract! Schema-aware Event Extraction using LLMs [45.83950260830323]
  This work focuses on harnessing Large Language Models for automated Event Extraction. It introduces a new method to address hallucination by decomposing the task into Event Detection and Event Argument Extraction.
  arXiv  Detail & Related papers  (2024-06-03T06:55:10Z)
• xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token [108.7069350303884]
  xRAG is an innovative context compression method tailored for retrieval-augmented generation. xRAG seamlessly integrates document embeddings into the language model representation space. Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2024-05-22T16:15:17Z)
• ExcluIR: Exclusionary Neural Information Retrieval [74.08276741093317]
  We present ExcluIR, a set of resources for exclusionary retrieval. evaluation benchmark includes 3,452 high-quality exclusionary queries. training set contains 70,293 exclusionary queries, each paired with a positive document and a negative document.
  arXiv  Detail & Related papers  (2024-04-26T09:43:40Z)
• Corrective Retrieval Augmented Generation [36.04062963574603]
  Retrieval-augmented generation (RAG) relies heavily on relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. We propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. CRAG is plug-and-play and can be seamlessly coupled with various RAG-based approaches.
  arXiv  Detail & Related papers  (2024-01-29T04:36:39Z)
• Noisy Self-Training with Synthetic Queries for Dense Retrieval [49.49928764695172]
  We introduce a novel noisy self-training framework combined with synthetic queries. Experimental results show that our method improves consistently over existing methods. Our method is data efficient and outperforms competitive baselines.
  arXiv  Detail & Related papers  (2023-11-27T06:19:50Z)
• End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems [7.2949782290577945]
  We consider a novel approach that learns the smart reply task end-to-end from a dataset of (message, reply set) pairs obtained via bootstrapping. Empirical results show this method consistently outperforms a range of state-of-the-art baselines across three datasets.
  arXiv  Detail & Related papers  (2023-10-29T09:56:17Z)
• Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs [80.74263278847063]
  The integration of retrieved passages and large language models (LLMs) has significantly contributed to improving open-domain question answering. This paper investigates different methods of combining retrieved passages with LLMs to enhance answer generation.
  arXiv  Detail & Related papers  (2023-08-24T05:26:54Z)
• AugTriever: Unsupervised Dense Retrieval and Domain Adaptation by Scalable Data Augmentation [44.93777271276723]
  We propose two approaches that enable annotation-free and scalable training by creating pseudo querydocument pairs. The query extraction method involves selecting salient spans from the original document to generate pseudo queries. The transferred query generation method utilizes generation models trained for other NLP tasks, such as summarization, to produce pseudo queries.
  arXiv  Detail & Related papers  (2022-12-17T10:43:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.