Prediction-Only Distillation in Linear and Logistic Regression

• URL: http://arxiv.org/abs/2607.15450v1
• Date: Thu, 16 Jul 2026 20:45:56 GMT
• Title: Prediction-Only Distillation in Linear and Logistic Regression
• Abstract summary: Self-distillation is typically studied when the student is retrained on the teacher's original training inputs.<n>We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme.<n>For binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled.
• Score: 61.07540493350384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.

Related papers

• Fast Rates for Nonstationary Weighted Risk Minimization [6.015898117103068]
  This article studies its out-of-sample prediction error under nonstationarity.<n>We provide a general decomposition of the excess risk into a learning term and an error term associated with distribution drift.
  arXiv  Detail & Related papers  (2026-02-05T15:10:07Z)
• How Is Uncertainty Propagated in Knowledge Distillation? [7.132728938560163]
  We show how uncertainty propagates through knowledge distillation across three representative model classes.<n>We introduce two variance-aware strategies: averaging multiple teacher responses, and variance-weighting, which combines teacher and student estimates.<n>These results show that variance-aware distillation produces more stable students that better reflect teacher uncertainty.
  arXiv  Detail & Related papers  (2026-01-26T19:18:50Z)
• Cost-Sensitive Conformal Training with Provably Controllable Learning Bounds [21.86960662161151]
  Conformal prediction is a framework to quantify the predictive uncertainty of machine learning models.<n>To align the uncertainty measured by CP, conformal training methods minimize the size of the prediction sets.<n>We propose a simple cost-sensitive conformal training algorithm that does not rely on the indicator approximation mechanism.
  arXiv  Detail & Related papers  (2025-11-22T01:11:44Z)
• Active Data Sampling and Generation for Bias Remediation [0.0]
  A mixed active sampling and data generation strategy -- called samplation -- is proposed to compensate during fine-tuning of a pre-trained classifer the unfair classifications it produces.<n>Using as case study Deep Models for visual semantic role labeling, the proposed method has been able to fully cure a simulated gender bias starting from a 90/10 imbalance.
  arXiv  Detail & Related papers  (2025-03-26T10:42:15Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Twice Class Bias Correction for Imbalanced Semi-Supervised Learning [59.90429949214134]
  We introduce a novel approach called textbfTwice textbfClass textbfBias textbfCorrection (textbfTCBC) We estimate the class bias of the model parameters during the training process. We apply a secondary correction to the model's pseudo-labels for unlabeled samples.
  arXiv  Detail & Related papers  (2023-12-27T15:06:36Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Learning to Adapt Classifier for Imbalanced Semi-supervised Learning [38.434729550279116]
  Pseudo-labeling has proven to be a promising semi-supervised learning (SSL) paradigm. Existing pseudo-labeling methods commonly assume that the class distributions of training data are balanced. In this work, we investigate pseudo-labeling under imbalanced semi-supervised setups.
  arXiv  Detail & Related papers  (2022-07-28T02:15:47Z)
• Agree to Disagree: Diversity through Disagreement for Better Transferability [54.308327969778155]
  We propose D-BAT (Diversity-By-disAgreement Training), which enforces agreement among the models on the training data. We show how D-BAT naturally emerges from the notion of generalized discrepancy.
  arXiv  Detail & Related papers  (2022-02-09T12:03:02Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)
• Regularizing Class-wise Predictions via Self-knowledge Distillation [80.76254453115766]
  We propose a new regularization method that penalizes the predictive distribution between similar samples. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve the generalization ability.
  arXiv  Detail & Related papers  (2020-03-31T06:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.