LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

• URL: http://arxiv.org/abs/2404.05825v1
• Date: Mon, 8 Apr 2024 19:29:07 GMT
• Title: LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding
• Authors: Mingrui Wu, Sheng Cao,
• Abstract summary: This paper introduces a model-agnostic doc-level embedding framework through large language model augmentation. We have been able to significantly improve the effectiveness of widely-used retriever models.
• Score: 2.0257616108612373
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently embedding-based retrieval or dense retrieval have shown state of the art results, compared with traditional sparse or bag-of-words based approaches. This paper introduces a model-agnostic doc-level embedding framework through large language model (LLM) augmentation. In addition, it also improves some important components in the retrieval model training process, such as negative sampling, loss function, etc. By implementing this LLM-augmented retrieval framework, we have been able to significantly improve the effectiveness of widely-used retriever models such as Bi-encoders (Contriever, DRAGON) and late-interaction models (ColBERTv2), thereby achieving state-of-the-art results on LoTTE datasets and BEIR datasets.

Related papers

• Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists [41.94295877935867]
  We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science. We demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-10-30T17:59:01Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild [84.57103623507082]
  This paper introduces Model-GLUE, a holistic Large Language Models scaling guideline. Our work starts with a benchmarking of existing LLM scaling techniques, especially selective merging, and variants of mixture. Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters through a model mixture.
  arXiv  Detail & Related papers  (2024-10-07T15:55:55Z)
• Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies [8.822087602255504]
  Applying large language models to the clinical domain is challenging due to the context-heavy nature of processing medical records. This paper explores how different embedding models and pooling methods affect information retrieval for the clinical domain.
  arXiv  Detail & Related papers  (2024-09-23T16:16:08Z)
• The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction [22.659005954676598]
  We show that it is possible to significantly improve the performance of Large Language Models by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed. We show extensive experiments demonstrating the generality of this finding across language models and datasets.
  arXiv  Detail & Related papers  (2023-12-21T03:51:08Z)
• RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models [57.12888828853409]
  RAVEN is a model that combines retrieval-augmented masked language modeling and prefix language modeling. Fusion-in-Context Learning enables the model to leverage more in-context examples without requiring additional training. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning.
  arXiv  Detail & Related papers  (2023-08-15T17:59:18Z)
• Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models [59.89454513692417]
  Tabular data is often hidden in text, particularly in medical diagnostic reports. We propose a novel, simple, and effective methodology for extracting structured tabular data from textual medical reports, called TEMED-LLM. We demonstrate that our approach significantly outperforms state-of-the-art text classification models in medical diagnostics.
  arXiv  Detail & Related papers  (2023-06-08T09:12:28Z)
• DORE: Document Ordered Relation Extraction based on Generative Framework [56.537386636819626]
  This paper investigates the root cause of the underwhelming performance of the existing generative DocRE models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.
  arXiv  Detail & Related papers  (2022-10-28T11:18:10Z)
• Augmenting Interpretable Models with LLMs during Training [73.40079895413861]
  We propose Augmented Interpretable Models (Aug-imodels) to build efficient and interpretable models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency. We explore two instantiations of Aug-imodels in natural-language processing: (i) Aug-GAM, which augments a generalized additive model with decoupled embeddings from an LLM and (ii) Aug-Tree, which augments a decision tree with LLM feature expansions.
  arXiv  Detail & Related papers  (2022-09-23T18:36:01Z)
• Curriculum Learning for Dense Retrieval Distillation [20.25741148622744]
  We propose a generic curriculum learning based optimization framework called CL-DRD. CL-DRD controls the difficulty level of training data produced by the re-ranking (teacher) model. Experiments on three public passage retrieval datasets demonstrate the effectiveness of our proposed framework.
  arXiv  Detail & Related papers  (2022-04-28T17:42:21Z)
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval [11.38022203865326]
  SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. We modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. Overall, SPLADE is considerably improved with more than $9$% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
  arXiv  Detail & Related papers  (2021-09-21T10:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.