Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs

• URL: http://arxiv.org/abs/2505.16967v2
• Date: Sat, 18 Oct 2025 13:57:35 GMT
• Title: Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs
• Abstract summary: Training robust and reranker models typically rely on large-scale datasets.<n>We identify and relabel false negatives in training datasets.<n>Our training dataset and code are publicly available.
• Score: 53.78110936809744
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training robust retrieval and reranker models typically relies on large-scale retrieval datasets; for example, the BGE collection contains 1.6 million query-passage pairs sourced from various data sources. However, we find that certain datasets can negatively impact model effectiveness -- pruning 8 out of 15 datasets from the BGE collection, reduces the training set size by 2.35$\times$, surprisingly increases nDCG@10 on BEIR by 1.0 point. This motivates a deeper examination of training data quality, with a particular focus on "false negatives", where relevant passages are incorrectly labeled as irrelevant. We utilize LLMs as a simple, cost-effective approach to identify and relabel false negatives in training datasets. Experimental results show that relabeling false negatives as true positives improves both E5 (base) and Qwen2.5-7B retrieval models by 0.7$\unicode{x2013}$1.4 points on BEIR and by 1.7$\unicode{x2013}$1.8 points at nDCG@10 on zero-shot AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of LLMs to identify false negatives is supported by human annotation results. Our training dataset and code are publicly available.

Related papers

• Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning [21.70706473875226]
  We propose Reinforcement Distillation (REDI), a two-stage framework.<n> Stage 1 learns from positive traces via Supervised Fine-Tuning (SFT)<n>Stage 2 further refines the model using both positive and negative traces through our proposed REDI objective.<n>Our empirical evaluations demonstrate REDI's superiority over baseline Rejection Sampling SFT or SFT combined with DPO/SimPO on mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2025-05-30T17:47:17Z)
• Evaluating the Impact of Data Cleaning on the Quality of Generated Pull Request Descriptions [2.2134505920972547]
  Pull Requests (PRs) are central to collaborative coding.<n>Many PRs are incomplete, uninformative, or have out-of-context content.<n>This study examines the prevalence of "noisy" PRs and evaluates their impact on description generation models.
  arXiv  Detail & Related papers  (2025-05-02T08:58:42Z)
• SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement [100.85923086072204]
  ThinkLite-VL improves the average performance of Qwen2.5-VL-7B-Instruct by 7%, using only 11k training samples with no knowledge distillation.<n>Our code, data, and model are available at https://github.com/si0wang/ThinkLite-VL.
  arXiv  Detail & Related papers  (2025-04-10T17:49:05Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Learning to Verify Summary Facts with Fine-Grained LLM Feedback [15.007479147796403]
  Training automatic summary fact verifiers often faces the challenge of a lack of human-labeled data.<n>We introduce FineSumFact, a large-scale dataset containing fine-grained factual feedback on summaries.
  arXiv  Detail & Related papers  (2024-12-14T05:28:44Z)
• Segmentation Dataset for Reinforced Concrete Construction [4.32009010195029]
  This paper provides a dataset of 14,805 RGB images with segmentation labels for autonomous robotic inspection of reinforced concrete defects.<n>YOLOv8L-seg performs best, achieving a validation mIOU score of up to 0.59.<n>Lack of publicly available data is identified as a significant contributor to false negatives.
  arXiv  Detail & Related papers  (2024-07-12T15:53:15Z)
• SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness [9.080678336379528]
  We introduce Scaled Activation Projection (SAP), a novel SVD-based corrective machine unlearning algorithm.<n>SAP mitigates label noise by identifying a small subset of trusted samples using cross-entropy loss.<n>We observe generalization improvements of 2.31% for a Vision Transformer model trained on naturally corrupted Clothing1M.
  arXiv  Detail & Related papers  (2024-03-13T15:32:08Z)
• CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? [72.19502317793133]
  We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP) We present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases.
  arXiv  Detail & Related papers  (2024-03-07T14:43:17Z)
• BaSAL: Size-Balanced Warm Start Active Learning for LiDAR Semantic Segmentation [2.9290232815049926]
  Existing active learning methods overlook the severe class imbalance inherent in LiDAR semantic segmentation datasets. We propose BaSAL, a size-balanced warm start active learning model, based on the observation that each object class has a characteristic size. Results show that we are able to improve the performance of the initial model by a large margin.
  arXiv  Detail & Related papers  (2023-10-12T05:03:19Z)
• gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling [67.71952251641545]
  We show that models trained with negative sampling tend to overestimate the probabilities of positive interactions. We propose a novel Generalised Binary Cross-Entropy Loss function (gBCE) and theoretically prove that it can mitigate overconfidence. We show through detailed experiments on three datasets that gSASRec does not exhibit the overconfidence problem.
  arXiv  Detail & Related papers  (2023-08-14T14:56:40Z)
• Rethinking InfoNCE: How Many Negative Samples Do You Need? [54.146208195806636]
  We study how many negative samples are optimal for InfoNCE in different scenarios via a semi-quantitative theoretical framework. We estimate the optimal negative sampling ratio using the $K$ value that maximizes the training effectiveness function.
  arXiv  Detail & Related papers  (2021-05-27T08:38:29Z)
• On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks [94.23884467360521]
  We show that state-of-the art models trained on QQP and WikiQA each have only $2.4%$ average precision when evaluated on realistically imbalanced test data. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to $32.5%$ on QQP and $20.1%$ on WikiQA.
  arXiv  Detail & Related papers  (2020-10-10T21:56:27Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)
• Reinforced Negative Sampling over Knowledge Graph for Recommendation [106.07209348727564]
  We develop a new negative sampling model, Knowledge Graph Policy Network (kgPolicy), which works as a reinforcement learning agent to explore high-quality negatives. kgPolicy navigates from the target positive interaction, adaptively receives knowledge-aware negative signals, and ultimately yields a potential negative item to train the recommender.
  arXiv  Detail & Related papers  (2020-03-12T12:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.