Dimension-free bounds in high-dimensional linear regression via error-in-operator approach

• URL: http://arxiv.org/abs/2502.15437v1
• Date: Fri, 21 Feb 2025 13:07:14 GMT
• Title: Dimension-free bounds in high-dimensional linear regression via error-in-operator approach
• Authors: Fedor Noskov, Nikita Puchkin, Vladimir Spokoiny,
• Abstract summary: We consider a problem of high-dimensional linear regression with random design.<n>We suggest a novel approach referred to as error-in-operator which does not estimate the design covariance $Sigma$ directly but incorporates it into empirical risk minimization.<n>We provide an expansion of the excess prediction risk and derive non-asymptotic dimension-free bounds on the leading term and the remainder.
• Score: 4.929399529593514
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider a problem of high-dimensional linear regression with random design. We suggest a novel approach referred to as error-in-operator which does not estimate the design covariance $\Sigma$ directly but incorporates it into empirical risk minimization. We provide an expansion of the excess prediction risk and derive non-asymptotic dimension-free bounds on the leading term and the remainder. This helps us to show that auxiliary variables do not increase the effective dimension of the problem, provided that parameters of the procedure are tuned properly. We also discuss computational aspects of our method and illustrate its performance with numerical experiments.

Related papers

• RieszBoost: Gradient Boosting for Riesz Regression [49.737777802061984]
  We propose a novel gradient boosting algorithm to directly estimate the Riesz representer without requiring its explicit analytical form. We show that our algorithm performs on par with or better than indirect estimation techniques across a range of functionals.
  arXiv  Detail & Related papers  (2025-01-08T23:04:32Z)
• Refined Risk Bounds for Unbounded Losses via Transductive Priors [58.967816314671296]
  We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression.<n>Our key tools are based on the exponential weights algorithm with carefully chosen transductive priors.
  arXiv  Detail & Related papers  (2024-10-29T00:01:04Z)
• Contrastive inverse regression for dimension reduction [0.0]
  We propose a supervised dimension reduction method called contrastive inverse regression (CIR) specifically designed for the contrastive setting. CIR introduces an optimization problem defined on the Stiefel manifold with a non-standard loss function. We prove the convergence of CIR to a local optimum using a gradient descent-based algorithm, and our numerical study empirically demonstrates the improved performance over competing methods for high-dimensional data.
  arXiv  Detail & Related papers  (2023-05-20T21:44:11Z)
• Riemannian Optimization for Variance Estimation in Linear Mixed Models [0.0]
  We take a completely novel view on parameter estimation in linear mixed models by exploiting the intrinsic geometry of the parameter space. Our approach yields a higher quality of the variance parameter estimates compared to existing approaches.
  arXiv  Detail & Related papers  (2022-12-18T13:08:45Z)
• Noise Estimation in Gaussian Process Regression [1.5002438468152661]
  The presented method can be used to estimate the variance of the correlated error, and the variance of the noise based on maximizing a marginal likelihood function. We demonstrate the computational advantages and robustness of the presented approach compared to traditional parameter optimization.
  arXiv  Detail & Related papers  (2022-06-20T19:36:03Z)
• Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds [54.51566432934556]
  We consider distributed optimization methods for problems where forming the Hessian is computationally challenging. We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems.
  arXiv  Detail & Related papers  (2022-03-18T05:49:13Z)
• Distributionally Robust Optimization with Markovian Data [8.126833795693699]
  We study a program where the probability distribution of the uncertain problem parameters is unknown. We propose a data-driven distributionally to estimate the problem's objective function and optimal solution.
  arXiv  Detail & Related papers  (2021-06-12T10:59:02Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization [56.05635751529922]
  We propose a new randomized algorithm for solving L2-regularized least-squares problems based on sketching. We consider two of the most popular random embeddings, namely, Gaussian embeddings and the Subsampled Randomized Hadamard Transform (SRHT)
  arXiv  Detail & Related papers  (2020-06-10T15:00:09Z)
• Support recovery and sup-norm convergence rates for sparse pivotal estimation [79.13844065776928]
  In high dimensional sparse regression, pivotal estimators are estimators for which the optimal regularization parameter is independent of the noise level. We show minimax sup-norm convergence rates for non smoothed and smoothed, single task and multitask square-root Lasso-type estimators.
  arXiv  Detail & Related papers  (2020-01-15T16:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.