Noise-Robust Dense Retrieval via Contrastive Alignment Post Training

• URL: http://arxiv.org/abs/2304.03401v2
• Date: Mon, 10 Apr 2023 14:22:32 GMT
• Title: Noise-Robust Dense Retrieval via Contrastive Alignment Post Training
• Authors: Daniel Campos, ChengXiang Zhai, and Alessandro Magnani
• Abstract summary: Contrastive Alignment POst Training (CAPOT) is a highly efficient finetuning method that improves model robustness without requiring index regeneration. CAPOT enables robust retrieval by freezing the document encoder while the query encoder learns to align noisy queries with their unaltered root. We evaluate CAPOT noisy variants of MSMARCO, Natural Questions, and Trivia QA passage retrieval, finding CAPOT has a similar impact as data augmentation with none of its overhead.
• Score: 89.29256833403167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of contextual word representations and advances in neural information retrieval have made dense vector-based retrieval a standard approach for passage and document ranking. While effective and efficient, dual-encoders are brittle to variations in query distributions and noisy queries. Data augmentation can make models more robust but introduces overhead to training set generation and requires retraining and index regeneration. We present Contrastive Alignment POst Training (CAPOT), a highly efficient finetuning method that improves model robustness without requiring index regeneration, the training set optimization, or alteration. CAPOT enables robust retrieval by freezing the document encoder while the query encoder learns to align noisy queries with their unaltered root. We evaluate CAPOT noisy variants of MSMARCO, Natural Questions, and Trivia QA passage retrieval, finding CAPOT has a similar impact as data augmentation with none of its overhead.

Related papers

• A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks [81.2624272756733]
  In dense retrieval, deep encoders provide embeddings for both inputs and targets. We train a small parametric corrector network that adjusts stale cached target embeddings. Our approach matches state-of-the-art results even when no target embedding updates are made during training.
  arXiv  Detail & Related papers  (2024-09-03T13:29:13Z)
• Corrective Retrieval Augmented Generation [36.04062963574603]
  Retrieval-augmented generation (RAG) relies heavily on relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. We propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. CRAG is plug-and-play and can be seamlessly coupled with various RAG-based approaches.
  arXiv  Detail & Related papers  (2024-01-29T04:36:39Z)
• Noisy Pair Corrector for Dense Retrieval [59.312376423104055]
  We propose a novel approach called Noisy Pair Corrector (NPC) NPC consists of a detection module and a correction module. We conduct experiments on text-retrieval benchmarks Natural Question and TriviaQA, code-search benchmarks StaQC and SO-DS.
  arXiv  Detail & Related papers  (2023-11-07T08:27:14Z)
• Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization [76.57699934689468]
  We propose a fine-grained Token-level retrieval-augmented mechanism (Tram) on the decoder side to enhance the performance of neural models. To overcome the challenge of token-level retrieval in capturing contextual code semantics, we also propose integrating code semantics into individual summary tokens.
  arXiv  Detail & Related papers  (2023-05-18T16:02:04Z)
• Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders [89.29256833403169]
  We introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods. KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
  arXiv  Detail & Related papers  (2023-03-31T15:44:13Z)
• ConTextual Mask Auto-Encoder for Dense Passage Retrieval [49.49460769701308]
  CoT-MAE is a simple yet effective generative pre-training method for dense passage retrieval. It learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines.
  arXiv  Detail & Related papers  (2022-08-16T11:17:22Z)
• Pattern-aware Data Augmentation for Query Rewriting in Voice Assistant Systems [10.332550622090718]
  We propose an augmentation framework that learns patterns from existing training pairs and generates rewrite candidates from rewrite labels inversely to compensate for insufficient QR training data. Our experimental results show its effectiveness compared with a fully trained QR baseline and demonstrate its potential application in boosting the QR performance on low-resource domains or locales.
  arXiv  Detail & Related papers  (2020-12-21T16:36:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.