MURR: Model Updating with Regularized Replay for Searching a Document Stream

• URL: http://arxiv.org/abs/2504.10250v1
• Date: Mon, 14 Apr 2025 14:13:03 GMT
• Title: MURR: Model Updating with Regularized Replay for Searching a Document Stream
• Authors: Eugene Yang, Nicola Tonellotto, Dawn Lawrie, Sean MacAvaney, James Mayfield, Douglas W. Oard, Scott Miller,
• Abstract summary: Internet produces a continuous stream of new documents and user-generated queries.<n>Neural retrieval models that were trained once on a fixed set of query-document pairs will quickly start misrepresenting newly-created content.<n>We propose MURR, a model updating strategy with regularized replay, to ensure the model can still faithfully search existing documents without reprocessing.
• Score: 32.0637790321157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Internet produces a continuous stream of new documents and user-generated queries. These naturally change over time based on events in the world and the evolution of language. Neural retrieval models that were trained once on a fixed set of query-document pairs will quickly start misrepresenting newly-created content and queries, leading to less effective retrieval. Traditional statistical sparse retrieval can update collection statistics to reflect these changes in the use of language in documents and queries. In contrast, continued fine-tuning of the language model underlying neural retrieval approaches such as DPR and ColBERT creates incompatibility with previously-encoded documents. Re-encoding and re-indexing all previously-processed documents can be costly. In this work, we explore updating a neural dual encoder retrieval model without reprocessing past documents in the stream. We propose MURR, a model updating strategy with regularized replay, to ensure the model can still faithfully search existing documents without reprocessing, while continuing to update the model for the latest topics. In our simulated streaming environments, we show that fine-tuning models using MURR leads to more effective and more consistent retrieval results than other strategies as the stream of documents and queries progresses.

Related papers

• Cognitive-Aligned Document Selection for Retrieval-augmented Generation [2.9060210098040855]
  We propose GGatrieval to dynamically update queries and filter high-quality, reliable retrieval documents. We parse the user query into its syntactic components and perform fine-grained grounded alignment with the retrieved documents. Our approach introduces a novel criterion for filtering retrieved documents, closely emulating human strategies for acquiring targeted information.
  arXiv  Detail & Related papers  (2025-02-17T13:00:15Z)
• MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models [22.50450558103786]
  In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts. We propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.
  arXiv  Detail & Related papers  (2024-08-30T07:57:30Z)
• Query-oriented Data Augmentation for Session Search [71.84678750612754]
  We propose query-oriented data augmentation to enrich search logs and empower the modeling. We generate supplemental training pairs by altering the most important part of a search context. We develop several strategies to alter the current query, resulting in new training data with varying degrees of difficulty.
  arXiv  Detail & Related papers  (2024-07-04T08:08:33Z)
• Language Modeling with Editable External Knowledge [90.7714362827356]
  This paper introduces ERASE, which improves model behavior when new documents are acquired. It incrementally deletes or rewriting other entries in the knowledge base each time a document is added. It improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute.
  arXiv  Detail & Related papers  (2024-06-17T17:59:35Z)
• IncDSI: Incrementally Updatable Document Retrieval [35.5697863674097]
  IncDSI is a method to add documents in real time without retraining the model on the entire dataset. We formulate the addition of documents as a constrained optimization problem that makes minimal changes to the network parameters. Our approach is competitive with re-training the model on the whole dataset.
  arXiv  Detail & Related papers  (2023-07-19T07:20:30Z)
• DSI++: Updating Transformer Memory with New Documents [95.70264288158766]
  We introduce DSI++, a continual learning challenge for DSI to incrementally index new documents. We show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task.
  arXiv  Detail & Related papers  (2022-12-19T18:59:34Z)
• CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion [68.19934563919192]
  We propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.
  arXiv  Detail & Related papers  (2022-12-18T15:57:46Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.