Random-projection ensemble dimension reduction

• URL: http://arxiv.org/abs/2410.04922v1
• Date: Mon, 7 Oct 2024 11:09:50 GMT
• Title: Random-projection ensemble dimension reduction
• Abstract summary: We introduce a new framework for dimension reduction in the context of high-dimensional regression. We consider disjoint groups of independent random projections, apply a base regression method after each projection, and retain the projection in each group based on the empirical performance. We demonstrate the excellent empirical performance of our proposal in a large numerical study using simulated and real data.
• Score: 1.6114012813668932
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new framework for dimension reduction in the context of high-dimensional regression. Our proposal is to aggregate an ensemble of random projections, which have been carefully chosen based on the empirical regression performance after being applied to the covariates. More precisely, we consider disjoint groups of independent random projections, apply a base regression method after each projection, and retain the projection in each group based on the empirical performance. We aggregate the selected projections by taking the singular value decomposition of their empirical average and then output the leading order singular vectors. A particularly appealing aspect of our approach is that the singular values provide a measure of the relative importance of the corresponding projection directions, which can be used to select the final projection dimension. We investigate in detail (and provide default recommendations for) various aspects of our general framework, including the projection distribution and the base regression method, as well as the number of random projections used. Additionally, we investigate the possibility of further reducing the dimension by applying our algorithm twice in cases where projection dimension recommended in the initial application is too large. Our theoretical results show that the error of our algorithm stabilises as the number of groups of projections increases. We demonstrate the excellent empirical performance of our proposal in a large numerical study using simulated and real data.

Related papers

• Generalization Error Estimation for Primal--Dual Algorithms in Non-Smooth Regression [6.462588783417218]
  We study trajectory-wise estimation of generalization error for primal--dual algorithms in non-smooth regression.<n>We develop a general framework that includes the Chambolle--Pock algorithm and related primal--dual splitting methods.
  arXiv  Detail & Related papers  (2026-08-14T01:45:11Z)
• A Bayesian Approach for Task-Specific Next-Best-View Selection with Uncertain Geometry [50.782490644267334]
  We develop a framework for task-specific active next-best-view selection in 3D reconstruction from point clouds.<n>Our framework achieves superior task performance with fewer views compared to commonly used baselines and prior general uncertainty-reduction techniques.
  arXiv  Detail & Related papers  (2026-05-06T16:32:55Z)
• Proximal Projection for Doubly Sparse Regularized Models [1.843072866673446]
  The structure of a predictor graph can be exploited during regularization.<n>Regularization is then performed on the latent variables rather than on the coefficient vector directly.<n>Results suggest that our method exhibits stable performance relative to other singly or doubly sparse graphical regression models.
  arXiv  Detail & Related papers  (2026-05-06T16:31:20Z)
• Approximate Bayesian inference for cumulative probit regression models [0.0]
  Ordinal categorical data are routinely encountered in a wide range of practical applications.<n>We propose three scalable algorithms for approximating the posterior distribution of the regression coefficients in cumulative probit models.<n>We illustrate the utility of the proposed algorithms on a challenging case study to investigate the structure of a criminal network.
  arXiv  Detail & Related papers  (2025-11-10T11:15:42Z)
• Meta-Learning with Generalized Ridge Regression: High-dimensional Asymptotics, Optimality and Hyper-covariance Estimation [14.194212772887699]
  We consider meta-learning within the framework of high-dimensional random-effects linear models. We show the precise behavior of the predictive risk for a new test task when the data dimension grows proportionally to the number of samples per task. We propose and analyze an estimator inverse random regression coefficients based on data from the training tasks.
  arXiv  Detail & Related papers  (2024-03-27T21:18:43Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Diverse Projection Ensembles for Distributional Reinforcement Learning [6.144680854063937]
  distributional reinforcement learning algorithms aim to learn the distribution of returns rather than their expected value. We study the combination of several different projections and representations in a distributional ensemble. We derive an algorithm that uses ensemble disagreement, measured by the average 1-Wasserstein distance, as a bonus for deep exploration.
  arXiv  Detail & Related papers  (2023-06-12T13:59:48Z)
• Sharp-SSL: Selective high-dimensional axis-aligned random projections for semi-supervised learning [16.673022545571566]
  We propose a new method for high-dimensional semi-supervised learning problems. It is based on the careful aggregation of the results of a low-dimensional procedure applied to many axis-aligned random projections of the data.
  arXiv  Detail & Related papers  (2023-04-18T17:49:02Z)
• High-dimensional analysis of double descent for linear regression with random projections [0.0]
  We consider linear regression problems with a varying number of random projections, where we provably exhibit a double descent curve for a fixed prediction problem. We first consider the ridge regression estimator and re-interpret earlier results using classical notions from non-parametric statistics. We then compute equivalents of the generalization performance (in terms of bias and variance) of the minimum norm least-squares fit with random projections, providing simple expressions for the double descent phenomenon.
  arXiv  Detail & Related papers  (2023-03-02T15:58:09Z)
• Optimal Discriminant Analysis in High-Dimensional Latent Factor Models [1.4213973379473654]
  In high-dimensional classification problems, a commonly used approach is to first project the high-dimensional features into a lower dimensional space. We formulate a latent-variable model with a hidden low-dimensional structure to justify this two-step procedure. We propose a computationally efficient classifier that takes certain principal components (PCs) of the observed features as projections.
  arXiv  Detail & Related papers  (2022-10-23T21:45:53Z)
• Refined bounds for randomized experimental design [7.899055512130904]
  Experimental design is an approach for selecting samples among a given set so as to obtain the best estimator for a given criterion. We propose theoretical guarantees for randomized strategies on E and G-optimal design.
  arXiv  Detail & Related papers  (2020-12-22T20:37:57Z)
• Slice Sampling for General Completely Random Measures [74.24975039689893]
  We present a novel Markov chain Monte Carlo algorithm for posterior inference that adaptively sets the truncation level using auxiliary slice variables. The efficacy of the proposed algorithm is evaluated on several popular nonparametric models.
  arXiv  Detail & Related papers  (2020-06-24T17:53:53Z)
• Deep Dimension Reduction for Supervised Representation Learning [51.10448064423656]
  We propose a deep dimension reduction approach to learning representations with essential characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero.
  arXiv  Detail & Related papers  (2020-06-10T14:47:43Z)
• Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants [94.46276668068327]
  In [1], an ensemble of randomly projected linear discriminants is used to classify datasets. We develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator. We also demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.
  arXiv  Detail & Related papers  (2020-04-17T12:47:04Z)
• Distributed Averaging Methods for Randomized Second Order Optimization [54.51566432934556]
  We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a bottleneck. We develop unbiased parameter averaging methods for randomized second order optimization that employ sampling and sketching of the Hessian. We also extend the framework of second order averaging methods to introduce an unbiased distributed optimization framework for heterogeneous computing systems.
  arXiv  Detail & Related papers  (2020-02-16T09:01:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.