Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval

• URL: http://arxiv.org/abs/2604.04734v1
• Date: Mon, 06 Apr 2026 15:02:35 GMT
• Title: Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
• Abstract summary: We propose a Stratified Sampling strategy that uniformly covers the entire score spectrum.<n>Experiments confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline.
• Score: 26.186599286240362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.

Related papers

• Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers [0.34998703934432673]
  We introduce an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training.<n>The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix.<n>Our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD.
  arXiv  Detail & Related papers  (2026-07-30T06:20:37Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
  Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
  arXiv  Detail & Related papers  (2026-05-06T15:31:50Z)
• REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency [0.0]
  We introduce REDistill, a principled framework grounded in robust statistics.<n>Redistill replaces the standard KD objective with a power divergence loss, a generalization of KL divergence.<n>Experiments on CIFAR-100 and ImageNet-1k demonstrate that REDistill consistently improves student accuracy in diverse teacher-student architectures.
  arXiv  Detail & Related papers  (2026-02-04T15:50:53Z)
• Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning [48.041170200238206]
  We introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model.<n>It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation.
  arXiv  Detail & Related papers  (2026-01-14T02:43:17Z)
• SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines [82.00660447875266]
  Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilistic outputs.<n>We rigorously analyze the convergence behavior of students trained with Gradient Descent (SGD)<n>Our analysis shows that learning from BCPs yields variance reduction and removes neighborhood terms in the convergence bounds compared to one-hot supervision.<n>Motivated by these insights, we advocate the use of Bayesian deep learning models, which typically provide improved estimates of the BCPs, as teachers in KD.
  arXiv  Detail & Related papers  (2026-01-04T11:09:49Z)
• Enriching Knowledge Distillation with Intra-Class Contrastive Learning [40.40889547725741]
  We propose incorporating an intra-class contrastive loss during teacher training to enrich the intra-class information contained in soft labels.<n>It has been proved that the intra-class contrastive loss can enrich the intra-class diversity.
  arXiv  Detail & Related papers  (2025-09-26T08:35:34Z)
• Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias [5.434571018755813]
  Group Relative Knowledge Distillation (GRKD) is a novel framework that distills teacher knowledge by learning the relative ranking among classes.<n>Experiments on classification benchmarks demonstrate GRKD achieves superior generalization compared to existing methods.
  arXiv  Detail & Related papers  (2025-04-29T07:23:22Z)
• Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning [2.395881636777087]
  We observe that certain weak classes consistently exist even for class-balanced learning.<n>The inherent inter-class accuracy discrepancy can reach over 36.9% for federated learning on the FashionMNIST and CIFAR-10 datasets.<n>A partial knowledge distillation (PKD) method is proposed to improve the model's classification accuracy for weak classes.
  arXiv  Detail & Related papers  (2024-11-23T01:16:46Z)
• Unilaterally Aggregated Contrastive Learning with Hierarchical Augmentation for Anomaly Detection [64.50126371767476]
  We propose Unilaterally Aggregated Contrastive Learning with Hierarchical Augmentation (UniCon-HA) We explicitly encourage the concentration of inliers and the dispersion of virtual outliers via supervised and unsupervised contrastive losses. Our method is evaluated under three AD settings including unlabeled one-class, unlabeled multi-class, and labeled multi-class.
  arXiv  Detail & Related papers  (2023-08-20T04:01:50Z)
• Adam: Dense Retrieval Distillation with Adaptive Dark Examples [104.01735794498767]
  We propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-12-20T12:03:19Z)
• Teacher's pet: understanding and mitigating biases in distillation [61.44867470297283]
  Several works have shown that distillation significantly boosts the student's overall performance. However, are these gains uniform across all data subgroups? We show that distillation can harm performance on certain subgroups. We present techniques which soften the teacher influence for subgroups where it is less reliable.
  arXiv  Detail & Related papers  (2021-06-19T13:06:25Z)
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation [54.49894381464853]
  We propose to leverage both labeled and unlabeled data for instance segmentation with improved accuracy by knowledge distillation. We propose a novel Mask-guided Mean Teacher framework with Perturbation-sensitive Sample Mining. Experiments show that the proposed method improves the performance significantly compared with the supervised method learned from labeled data only.
  arXiv  Detail & Related papers  (2020-07-21T13:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.