NV-Retriever: Improving text embedding models with effective hard-negative mining

• URL: http://arxiv.org/abs/2407.15831v2
• Date: Fri, 07 Feb 2025 15:17:18 GMT
• Title: NV-Retriever: Improving text embedding models with effective hard-negative mining
• Authors: Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, Even Oldridge,
• Abstract summary: We introduce a family of positive-aware mining methods that use the positive relevance score as an anchor for effective false negative removal.<n>We demonstrate the efficacy of our proposed mining methods at scale with the NV-Retriever-v1 model.
• Score: 1.8448587047759064
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text embedding models have been popular for information retrieval applications such as semantic search and Question-Answering systems based on Retrieval-Augmented Generation (RAG). Those models are typically Transformer models that are fine-tuned with contrastive learning objectives. One of the challenging aspects of fine-tuning embedding models is the selection of high quality hard-negative passages for contrastive learning. In this paper we introduce a family of positive-aware mining methods that use the positive relevance score as an anchor for effective false negative removal, leading to faster training and more accurate retrieval models. We provide an ablation study on hard-negative mining methods over their configurations, exploring different teacher and base models. We further demonstrate the efficacy of our proposed mining methods at scale with the NV-Retriever-v1 model, which scores 60.9 on MTEB Retrieval (BEIR) benchmark and placed 1st when it was published to the MTEB Retrieval on July, 2024.

Related papers

• Enhancing Retrieval Performance: An Ensemble Approach For Hard Negative Mining [0.0]
  This study focuses on explaining the crucial role of hard negatives in the training process of cross-encoder models. We have developed a robust hard negative mining technique for efficient training of cross-encoder re-rank models on an enterprise dataset.
  arXiv  Detail & Related papers  (2024-10-18T05:23:39Z)
• Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models [2.0962367975513496]
  Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. Existing unlearning methods rely solely on negative feedback to suppress responses related to the forget set. We propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set.
  arXiv  Detail & Related papers  (2024-09-20T13:05:07Z)
• Conan-embedding: General Text Embedding with More and Better Negative Samples [30.571206231457932]
  We propose a conan-embedding model, which maximizes the utilization of more and higher-quality negative examples. Our approach effectively enhances the capabilities of embedding models, currently ranking first on the Chinese leaderboard of Massive text embedding benchmark.
  arXiv  Detail & Related papers  (2024-08-28T11:18:06Z)
• Self-Taught Evaluators [77.92610887220594]
  We present an approach that aims to im-proves without human annotations, using synthetic training data only. Our Self-Taught Evaluator can improve a strong LLM from 75.4 to 88.3 on RewardBench.
  arXiv  Detail & Related papers  (2024-08-05T17:57:02Z)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models [38.41524186248607]
  We introduce NV-Embed, incorporating architectural designs, training procedures, and curated datasets. For model architecture, we propose a latent attention layer to obtain pooled embeddings. For training algorithm, we introduce a two-stage contrastive instruction-tuning method.
  arXiv  Detail & Related papers  (2024-05-27T17:59:45Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)
• GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning [0.0]
  GISTEmbed is a novel strategy that enhances in-batch negative selection during contrastive training through a guide model. Benchmarked against the Massive Text Embedding Benchmark (MTEB), GISTEmbed showcases consistent performance improvements across various model sizes.
  arXiv  Detail & Related papers  (2024-02-26T18:55:15Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration [79.04007257606862]
  This paper introduces an innovative method termed 'learning from history', which dynamically generates negative samples from the target model itself. Our approach, named Model Contrastive Learning for Image Restoration (MCLIR), rejuvenates latency models as negative models, making it compatible with diverse image restoration tasks.
  arXiv  Detail & Related papers  (2023-09-12T07:50:54Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• PartMix: Regularization Strategy to Learn Part Discovery for Visible-Infrared Person Re-identification [76.40417061480564]
  We present a novel data augmentation technique, dubbed PartMix, for part-based Visible-Infrared person Re-IDentification (VI-ReID) models. We synthesize the augmented samples by mixing the part descriptors across the modalities to improve the performance of part-based VI-ReID models.
  arXiv  Detail & Related papers  (2023-04-04T05:21:23Z)
• Semi-Supervised Learning Based on Reference Model for Low-resource TTS [32.731900584216724]
  We propose a semi-supervised learning method for neural TTS in which labeled target data is limited. Experimental results show that our proposed semi-supervised learning scheme with limited target data significantly improves the voice quality for test data to achieve naturalness and robustness in speech synthesis.
  arXiv  Detail & Related papers  (2022-10-25T07:48:07Z)
• WSLRec: Weakly Supervised Learning for Neural Sequential Recommendation Models [24.455665093145818]
  We propose a novel model-agnostic training approach called WSLRec, which adopts a three-stage framework: pre-training, top-$k$ mining, intrinsic and fine-tuning. WSLRec resolves the incompleteness problem by pre-training models on extra weak supervisions from model-free methods like BR and ItemCF, while resolving the inaccuracy problem by leveraging the top-$k$ mining to screen out reliable user-item relevance from weak supervisions for fine-tuning.
  arXiv  Detail & Related papers  (2022-02-28T08:55:12Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Positive-Congruent Training: Towards Regression-Free Model Updates [87.25247195148187]
  In image classification, sample-wise inconsistencies appear as "negative flips" A new model incorrectly predicts the output for a test sample that was correctly classified by the old (reference) model. We propose a simple approach for PC training, Focal Distillation, which enforces congruence with the reference model.
  arXiv  Detail & Related papers  (2020-11-18T09:00:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.