Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey

• URL: http://arxiv.org/abs/2110.09620v1
• Date: Mon, 18 Oct 2021 21:05:08 GMT
• Title: Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey
• Authors: Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark Crowley
• Abstract summary: This is a tutorial and survey paper on various methods for Sufficient Dimension Reduction (SDR) We cover these methods with both statistical high-dimensional regression perspective and machine learning approach for dimensionality reduction.
• Score: 5.967999555890417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This is a tutorial and survey paper on various methods for Sufficient Dimension Reduction (SDR). We cover these methods with both statistical high-dimensional regression perspective and machine learning approach for dimensionality reduction. We start with introducing inverse regression methods including Sliced Inverse Regression (SIR), Sliced Average Variance Estimation (SAVE), contour regression, directional regression, Principal Fitted Components (PFC), Likelihood Acquired Direction (LAD), and graphical regression. Then, we introduce forward regression methods including Principal Hessian Directions (pHd), Minimum Average Variance Estimation (MAVE), Conditional Variance Estimation (CVE), and deep SDR methods. Finally, we explain Kernel Dimension Reduction (KDR) both for supervised and unsupervised learning. We also show that supervised KDR and supervised PCA are equivalent.

Related papers

• Geometry-Aware Instrumental Variable Regression [56.16884466478886]
  We propose a transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings.
  arXiv  Detail & Related papers  (2024-05-19T17:49:33Z)
• Online Kernel Sliced Inverse Regression [4.561305216067566]
  Online dimension reduction is a common method for high-dimensional streaming data processing. In this article, an online kernel sliced inverse regression method is proposed. We transform the problem into an online generalized eigendecomposition problem, and use the optimization method to update the centered dimension reduction directions.
  arXiv  Detail & Related papers  (2023-01-23T16:05:51Z)
• Vector-Valued Least-Squares Regression under Output Regularity Assumptions [73.99064151691597]
  We propose and analyse a reduced-rank method for solving least-squares regression problems with infinite dimensional output. We derive learning bounds for our method, and study under which setting statistical performance is improved in comparison to full-rank method.
  arXiv  Detail & Related papers  (2022-11-16T15:07:00Z)
• A selective review of sufficient dimension reduction for multivariate response regression [2.492300648514129]
  A wide range of SDR methods are characterized as inverse regression SDR estimators or forward regression SDR estimators. Ordinary least squares, partial least squares, and semiparametric SDR estimators are discussed as estimators from the forward regression family.
  arXiv  Detail & Related papers  (2022-02-02T04:53:09Z)
• Memory-Efficient Backpropagation through Large Linear Layers [107.20037639738433]
  In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to perform backpropagation through linear layers.
  arXiv  Detail & Related papers  (2022-01-31T13:02:41Z)
• Dimension Reduction and Data Visualization for Fr\'echet Regression [8.713190936209156]
  Fr'echet regression model provides a promising framework for regression analysis with metric spacevalued responses. We introduce a flexible sufficient dimension reduction (SDR) method for Fr'echet regression to achieve two purposes.
  arXiv  Detail & Related papers  (2021-10-01T15:01:32Z)
• Human Pose Regression with Residual Log-likelihood Estimation [48.30425850653223]
  We propose a novel regression paradigm with Residual Log-likelihood Estimation (RLE) to capture the underlying output distribution. RLE learns the change of the distribution instead of the unreferenced underlying distribution to facilitate the training process. Compared to the conventional regression paradigm, regression with RLE bring 12.4 mAP improvement on MSCOCO without any test-time overhead.
  arXiv  Detail & Related papers  (2021-07-23T15:06:31Z)
• Robust Kernel-based Distribution Regression [13.426195476348955]
  We study distribution regression (DR) which involves two stages of sampling, and aims at regressing from probability measures to real-valued responses over a kernel reproducing Hilbert space (RKHS) By introducing a robust loss function $l_sigma$ for two-stage sampling problems, we present a novel robust distribution regression (RDR) scheme.
  arXiv  Detail & Related papers  (2021-04-21T17:03:46Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• A Hypergradient Approach to Robust Regression without Correspondence [85.49775273716503]
  We consider a variant of regression problem, where the correspondence between input and output data is not available. Most existing methods are only applicable when the sample size is small. We propose a new computational framework -- ROBOT -- for the shuffled regression problem.
  arXiv  Detail & Related papers  (2020-11-30T21:47:38Z)
• Stochastic Gradient Descent Meets Distribution Regression [0.0]
  gradient descent (SGD) provides a simple and efficient way to solve a broad range of machine learning problems. We focus on distribution regression (DR), involving two stages of sampling: Firstly, we regress from probability measures to real-valued responses.
  arXiv  Detail & Related papers  (2020-10-24T09:03:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.