Solution-Set Geometry and Regularization Path of a Nonconvexly Regularized Convex Sparse Model

• URL: http://arxiv.org/abs/2311.18438v2
• Date: Fri, 22 Mar 2024 13:26:52 GMT
• Title: Solution-Set Geometry and Regularization Path of a Nonconvexly Regularized Convex Sparse Model
• Authors: Yi Zhang, Isao Yamada,
• Abstract summary: The Osborne generalized minimax geometry penalty is a non uniqueness solution which can preserve the overall-squareity of the regularized path. We show that similar to LASSO, the minimum $ell$-norm regularization is continuous piecewise in $lambda$.
• Score: 8.586951231230596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generalized minimax concave (GMC) penalty is a nonconvex sparse regularizer which can preserve the overall-convexity of the regularized least-squares problem. In this paper, we focus on a significant instance of the GMC model termed scaled GMC (sGMC), and present various notable findings on its solution-set geometry and regularization path. Our investigation indicates that while the sGMC penalty is a nonconvex extension of the LASSO penalty (i.e., the $\ell_1$-norm), the sGMC model preserves many celebrated properties of the LASSO model, hence can serve as a less biased surrogate of LASSO without losing its advantages. Specifically, for a fixed regularization parameter $\lambda$, we show that the solution-set geometry, solution uniqueness and sparseness of the sGMC model can be characterized in a similar elegant way to the LASSO model (see, e.g., Osborne et al. 2000, R. J. Tibshirani 2013). For a varying $\lambda$, we prove that the sGMC solution set is a continuous polytope-valued mapping of $\lambda$. Most noticeably, our study indicates that similar to LASSO, the minimum $\ell_2$-norm regularization path of the sGMC model is continuous and piecewise linear in $\lambda$. Based on these theoretical results, an efficient regularization path algorithm is proposed for the sGMC model, extending the well-known least angle regression (LARS) algorithm for LASSO. We prove the correctness and finite termination of the proposed algorithm under a mild assumption, and confirm its correctness-in-general-situation, efficiency, and practical utility through numerical experiments. Many results in this study also contribute to the theoretical research of LASSO.

Related papers

• A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization [83.12938977698988]
  Generalized Category Discovery (GCD) aims to classify both base and novel images using labeled base data. Current approaches inadequately address the intrinsic optimization of the co-occurrence matrix $barA$ based on cosine similarity. We propose a Non-Negative Generalized Category Discovery (NN-GCD) framework to address these deficiencies.
  arXiv  Detail & Related papers  (2024-10-29T07:24:11Z)
• Efficient Frameworks for Generalized Low-Rank Matrix Bandit Problems [61.85150061213987]
  We study the generalized low-rank matrix bandit problem, proposed in citelu2021low under the Generalized Linear Model (GLM) framework. To overcome the computational infeasibility and theoretical restrain of existing algorithms, we first propose the G-ESTT framework. We show that G-ESTT can achieve the $tildeO(sqrt(d_1+d_2)3/2Mr3/2T)$ bound of regret while G-ESTS can achineve the $tildeO
  arXiv  Detail & Related papers  (2024-01-14T14:14:19Z)
• KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal [70.15267479220691]
  We consider and analyze the sample complexity of model reinforcement learning with a generative variance-free model. Our analysis shows that it is nearly minimax-optimal for finding an $varepsilon$-optimal policy when $varepsilon$ is sufficiently small.
  arXiv  Detail & Related papers  (2022-05-27T19:39:24Z)
• Spike-and-Slab Generalized Additive Models and Scalable Algorithms for High-Dimensional Data [0.0]
  We propose hierarchical generalized additive models (GAMs) to accommodate high-dimensional data. We consider the smoothing penalty for proper shrinkage of curve and separation of smoothing function linear and nonlinear spaces. Two and deterministic algorithms, EM-Coordinate Descent and EM-Iterative Weighted Least Squares, are developed for different utilities.
  arXiv  Detail & Related papers  (2021-10-27T14:11:13Z)
• Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning [52.76230802067506]
  A novel model-free algorithm is proposed to minimize regret in episodic reinforcement learning. The proposed algorithm employs an em early-settled reference update rule, with the aid of two Q-learning sequences. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings.
  arXiv  Detail & Related papers  (2021-10-09T21:13:48Z)
• Structured Stochastic Gradient MCMC [20.68905354115655]
  We propose a new non-parametric variational approximation that makes no assumptions about the approximate posterior's functional form. We obtain better predictive likelihoods and larger effective sample sizes than full SGMCMC.
  arXiv  Detail & Related papers  (2021-07-19T17:18:10Z)
• On the Adversarial Robustness of LASSO Based Feature Selection [72.54211869067979]
  In the considered model, there is a malicious adversary who can observe the whole dataset, and then will carefully modify the response values or the feature matrix. We formulate the modification strategy of the adversary as a bi-level optimization problem. Numerical examples with synthetic and real data illustrate that our method is efficient and effective.
  arXiv  Detail & Related papers  (2020-10-20T05:51:26Z)
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity [67.02490430380415]
  We show that model-based MARL achieves a sample complexity of $tilde O(|S||B|(gamma)-3epsilon-2)$ for finding the Nash equilibrium (NE) value up to some $epsilon$ error. We also show that such a sample bound is minimax-optimal (up to logarithmic factors) if the algorithm is reward-agnostic, where the algorithm queries state transition samples without reward knowledge.
  arXiv  Detail & Related papers  (2020-07-15T03:25:24Z)
• Effective Learning of a GMRF Mixture Model [8.336315962271396]
  We propose restricting the GMM to a Gaussian Markov Random Field Mixture Model (GMRF-MM) When the sparsity pattern of each matrix is known, we propose an efficient optimization method for the Maximum Likelihood Estimate (MLE) of that matrix. We show that our "debiasing" approach outperforms GLASSO in both the single-GMRF and the GMRF-MM cases.
  arXiv  Detail & Related papers  (2020-05-18T19:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.