ECI: Effective Contrastive Information to Evaluate Hard-Negatives

• URL: http://arxiv.org/abs/2603.20990v1
• Date: Sun, 22 Mar 2026 00:21:05 GMT
• Title: ECI: Effective Contrastive Information to Evaluate Hard-Negatives
• Abstract summary: ECI: Effective Contrastive Information is a metric grounded in Information Theory and Information Retrieval principles.<n>We evaluate ECI across hard-negative sets mined or generated using BM25, cross-encoders, and large language models.
• Score: 0.3674670871245149
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Hard negatives play a critical role in training and fine-tuning dense retrieval models, as they are semantically similar to positive documents yet non-relevant, and correctly distinguishing them is essential for improving retrieval accuracy. However, identifying effective hard negatives typically requires extensive ablation studies involving repeated fine-tuning with different negative sampling strategies and hyperparameters, resulting in substantial computational cost. In this paper, we introduce ECI: Effective Contrastive Information , a theoretically grounded metric grounded in Information Theory and Information Retrieval principles that enables practitioners to assess the quality of hard negatives prior to model fine-tuning. ECI evaluates negatives by optimizing the trade-off between Information Capacity the logarithmic bound on mutual information determined by set size and Discriminative Efficiency, a harmonic balance of Signal Magnitude (Hardness) and Safety (Max-Margin). Unlike heuristic approaches, ECI strictly penalizes unsafe, false-positive negatives prevalent in generative methods. We evaluate ECI across hard-negative sets mined or generated using BM25, cross-encoders, and large language models. Our results demonstrate that ECI accurately predicts downstream retrieval performance, identifying that hybrid strategies (BM25+Cross-Encoder) offer the optimal balance of volume and reliability, significantly reducing the need for costly end-to-end ablation studies.

Related papers

• EERLoss: A Novel Loss Function for Training Deep Biometric Models. A Case Study in Keystroke Dynamics [6.150574406751438]
  This paper introduces EERLoss: a subdifferentiable, arbitrarily accurate approximation to EER for training deep biometric models.<n>EERLoss is evaluated on a demanding behavioral biometric modality: keystroke dynamics verification.<n> Experiments are conducted on the large-scale KVC-onGoing benchmark, incorporating data from over 185,000 subjects.
  arXiv  Detail & Related papers  (2026-06-23T13:47:42Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Quantifying Adversarial Uncertainty in Evidential Deep Learning using Conflict Resolution [2.321323878201932]
  Conflict-aware Evidential Deep Learning (C-EDL) is a lightweight post-hoc uncertainty quantification approach.<n>C-EDL generates diverse, task-preserving transformations per input and quantifies disagreement to calibrate uncertainty estimates.<n>Our experimental evaluation shows that C-EDL significantly outperforms state-of-the-art EDL variants and competitive baselines.
  arXiv  Detail & Related papers  (2025-06-06T10:06:23Z)
• Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study [91.78803511141975]
  This work focuses on the roles of positive and negative samples in scaling reinforcement learning.<n>We identify substantial data inefficiency in group relative policy optimization, where over half of the samples yield zero advantage.<n>We investigate unstable performance across various reasoning models and benchmarks, attributing instability to uncertain problems with ambiguous outcomes.
  arXiv  Detail & Related papers  (2025-06-05T11:47:10Z)
• The role of data partitioning on the performance of EEG-based deep learning models in supervised cross-subject analysis: a preliminary study [37.69303106863453]
  Deep learning is advancing the analysis of electroencephalography (EEG) data by effectively discovering highly nonlinear patterns.<n>No comprehensive guidelines for proper data partitioning and cross-validation exist in the domain.<n>This paper thoroughly investigates the role of data partitioning and cross-validation in evaluating EEG deep learning models.
  arXiv  Detail & Related papers  (2025-05-19T12:05:28Z)
• How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence [23.019102917957152]
  Kernel Divergence Score (KDS) is a novel method that evaluates dataset contamination by computing the divergence between the kernel similarity matrix of sample embeddings.<n>KDS demonstrates a near-perfect correlation with contamination levels and outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-02-02T05:50:39Z)
• Dynamic Perturbation-Adaptive Adversarial Training on Medical Image Classification [9.039586043401972]
  adversarial examples (AEs) exhibited imperceptible similarity with raw data, raising serious concerns on network robustness. In this paper, we propose a dynamic perturbation-adaptive adversarial training (DPAAT) method, which placed AT in a dynamic learning environment to generate adaptive data-level perturbations. Comprehensive testing on dermatology HAM10000 dataset showed that the DPAAT not only achieved better robustness improvement and generalization preservation but also significantly enhanced mean average precision and interpretability.
  arXiv  Detail & Related papers  (2024-03-11T15:16:20Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)
• ELMV: an Ensemble-Learning Approach for Analyzing Electrical Health Records with Significant Missing Values [4.9810955364960385]
  We propose a novel Ensemble-Learning for Missing Value (ELMV) framework, which introduces an effective approach to construct multiple subsets of the original EHR data with a much lower missing rate. ELMV has been evaluated on a real-world healthcare data for critical feature identification as well as a batch of simulation data with different missing rates for outcome prediction.
  arXiv  Detail & Related papers  (2020-06-25T06:29:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.