The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression

• URL: http://arxiv.org/abs/2607.24041v1
• Date: Mon, 27 Jul 2026 06:30:49 GMT
• Title: The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression
• Abstract summary: We show that degeneracy of the covariance matrices and dependence can lead to multiple descent.<n>We show that maximum matching proofs and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.
• Score: 4.064516150252115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.

Related papers

• Identification of Causal Direction under an Arbitrary Number of Latent Confounders [54.76982125821112]
  In real-world scenarios, observed variables may be affected by multiple latent variables simultaneously.<n>We make use of the joint higher-order cumulant matrix of the observed variables constructed in a specific way.<n>We show that, surprisingly, causal asymmetry between two observed variables can be directly seen from the rank deficiency properties of such higher-order cumulant matrices.
  arXiv  Detail & Related papers  (2025-10-26T15:10:00Z)
• Causal vs. Anticausal merging of predictors [57.26526031579287]
  We study the differences arising from merging predictors in the causal and anticausal directions using the same data.<n>We use Causal Maximum Entropy (CMAXENT) as inductive bias to merge the predictors.
  arXiv  Detail & Related papers  (2025-01-14T20:38:15Z)
• Asymptotics of Linear Regression with Linearly Dependent Data [28.005935031887038]
  We study the computations of linear regression in settings with non-Gaussian covariates.<n>We show how dependencies influence estimation error and the choice of regularization parameters.
  arXiv  Detail & Related papers  (2024-12-04T20:31:47Z)
• Refined Risk Bounds for Unbounded Losses via Transductive Priors [67.12679195076387]
  We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression.<n>Our key tools are based on the exponential weights algorithm with carefully chosen transductive priors.
  arXiv  Detail & Related papers  (2024-10-29T00:01:04Z)
• Linear causal disentanglement via higher-order cumulants [0.0]
  We study the identifiability of linear causal disentanglement, assuming access to data under multiple contexts. We show that one perfect intervention on each latent variable is sufficient and in the worst case necessary to recover parameters under perfect interventions.
  arXiv  Detail & Related papers  (2024-07-05T15:53:16Z)
• High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile [0.0]
  We study the predictive risk of the ridge estimator for linear regression with a variance profile.<n>For certain class of variance profile, our work highlights the emergence of the well-known double descent phenomenon.<n>We also investigate the similarities and differences that exist with the standard setting of independent and identically distributed data.
  arXiv  Detail & Related papers  (2024-03-29T14:24:49Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• High-dimensional analysis of double descent for linear regression with random projections [0.0]
  We consider linear regression problems with a varying number of random projections, where we provably exhibit a double descent curve for a fixed prediction problem. We first consider the ridge regression estimator and re-interpret earlier results using classical notions from non-parametric statistics. We then compute equivalents of the generalization performance (in terms of bias and variance) of the minimum norm least-squares fit with random projections, providing simple expressions for the double descent phenomenon.
  arXiv  Detail & Related papers  (2023-03-02T15:58:09Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Kernel regression in high dimensions: Refined analysis beyond double descent [21.15702374555439]
  We show that while the bias is (almost) independent of d and monotonically decreases with n, the variance depends on n, d and can be unimodal or monotonically decreasing under different regularization schemes. Our refined analysis goes beyond the double descent theory by showing that, depending on the data eigen-profile and the level of regularization, the kernel regression risk curve can be a double-descent-like, bell-shaped, or monotonic function of n.
  arXiv  Detail & Related papers  (2020-10-06T12:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.