Benign Overfitting in Linear Classifiers with a Bias Term

• URL: http://arxiv.org/abs/2511.12840v1
• Date: Sun, 16 Nov 2025 23:59:36 GMT
• Title: Benign Overfitting in Linear Classifiers with a Bias Term
• Authors: Yuta Kondo,
• Abstract summary: We show that machine learning models with a large number of parameters generalize well despite perfectly interpolating noisy data.<n>We find that the presence of the bias term introduces new constraints on the data's covariance structure required for generalization.<n>This work provides a more complete picture of benign overfitting, revealing the non-trivial impact of the bias term on the conditions required for good generalization.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern machine learning models with a large number of parameters often generalize well despite perfectly interpolating noisy training data - a phenomenon known as benign overfitting. A foundational explanation for this in linear classification was recently provided by Hashimoto et al. (2025). However, this analysis was limited to the setting of "homogeneous" models, which lack a bias (intercept) term - a standard component in practice. This work directly extends Hashimoto et al.'s results to the more realistic inhomogeneous case, which incorporates a bias term. Our analysis proves that benign overfitting persists in these more complex models. We find that the presence of the bias term introduces new constraints on the data's covariance structure required for generalization, an effect that is particularly pronounced when label noise is present. However, we show that in the isotropic case, these new constraints are dominated by the requirements inherited from the homogeneous model. This work provides a more complete picture of benign overfitting, revealing the non-trivial impact of the bias term on the conditions required for good generalization.

Related papers

• Generative Classifiers Avoid Shortcut Solutions [84.23247217037134]
  Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail under minor distribution shift.<n>We show that generative classifiers can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones.<n>We find that diffusion-based and autorerimigressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• A Classical View on Benign Overfitting: The Role of Sample Size [14.36840959836957]
  We focus on almost benign overfitting, where models simultaneously achieve both arbitrarily small training and test errors.<n>This behavior is characteristic of neural networks, which often achieve low (but non-zero) training error while still generalizing well.
  arXiv  Detail & Related papers  (2025-05-16T18:37:51Z)
• Universality of Benign Overfitting in Binary Linear Classification [1.6385815610837167]
  deep neural networks seem to generalize well in the over-parametrized regime.<n>It is now known that benign overfitting also occurs in various classical statistical models.
  arXiv  Detail & Related papers  (2025-01-17T20:22:06Z)
• Grokking at the Edge of Linear Separability [1.024113475677323]
  grokking is delayed generalization accompanied by non-monotonic test loss behavior.<n>We find that grokking arises naturally even when the parameters of the problem are close to a critical point.
  arXiv  Detail & Related papers  (2024-10-06T14:08:42Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models [51.66015254740692]
  We show that for an ensemble of deep learning based classification models, bias and variance are emphaligned at a sample level. We study this phenomenon from two theoretical perspectives: calibration and neural collapse.
  arXiv  Detail & Related papers  (2023-10-13T17:06:34Z)
• Instance-Dependent Generalization Bounds via Optimal Transport [51.71650746285469]
  Existing generalization bounds fail to explain crucial factors that drive the generalization of modern neural networks. We derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the learned prediction function in the data space. We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.
  arXiv  Detail & Related papers  (2022-11-02T16:39:42Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• On how to avoid exacerbating spurious correlations when models are overparameterized [33.315813572333745]
  We show that VS-loss learns a model that is fair towards minorities even when spurious features are strong. Compared to previous works, our bounds hold for more general models, they are non-asymptotic, and, they apply even at scenarios of extreme imbalance.
  arXiv  Detail & Related papers  (2022-06-25T21:53:44Z)
• A Generative Approach for Mitigating Structural Biases in Natural Language Inference [24.44419010439227]
  In this work, we reformulate the NLI task as a generative task, where a model is conditioned on the biased subset of the input and the label. We show that this approach is highly robust to large amounts of bias. We find that generative models are difficult to train and they generally perform worse than discriminative baselines.
  arXiv  Detail & Related papers  (2021-08-31T17:59:45Z)
• Interpolation can hurt robust generalization even when there is no noise [76.3492338989419]
  We show that avoiding generalization through ridge regularization can significantly improve generalization even in the absence of noise. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.
  arXiv  Detail & Related papers  (2021-08-05T23:04:15Z)
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition [34.235007566913396]
  We describe an interpretable, symmetric decomposition of the variance into terms associated with the labels. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior. We also analyze the strikingly rich phenomenology that arises.
  arXiv  Detail & Related papers  (2020-11-04T21:04:02Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.