LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso

• URL: http://arxiv.org/abs/2111.02790v1
• Date: Thu, 4 Nov 2021 12:05:09 GMT
• Title: LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso
• Authors: Kenan \v{S}ehi\'c, Alexandre Gramfort, Joseph Salmon and Luigi Nardi
• Abstract summary: LassoBench is a new benchmark suite tailored for an important open research topic in the Lasso community. We evaluate 5 state-of-the-art HPO methods and 3 baselines, and demonstrate that Bayesian optimization, in particular, can improve over the methods commonly used for sparse regression.
• Score: 84.6451154376526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Even though Weighted Lasso regression has appealing statistical guarantees, it is typically avoided due to its complex search space described with thousands of hyperparameters. On the other hand, the latest progress with high-dimensional HPO methods for black-box functions demonstrates that high-dimensional applications can indeed be efficiently optimized. Despite this initial success, the high-dimensional HPO approaches are typically applied to synthetic problems with a moderate number of dimensions which limits its impact in scientific and engineering applications. To address this limitation, we propose LassoBench, a new benchmark suite tailored for an important open research topic in the Lasso community that is Weighted Lasso regression. LassoBench consists of benchmarks on both well-controlled synthetic setups (number of samples, SNR, ambient and effective dimensionalities, and multiple fidelities) and real-world datasets, which enable the use of many flavors of HPO algorithms to be improved and extended to the high-dimensional setting. We evaluate 5 state-of-the-art HPO methods and 3 baselines, and demonstrate that Bayesian optimization, in particular, can improve over the methods commonly used for sparse regression while highlighting limitations of these frameworks in very high-dimensions. Remarkably, Bayesian optimization improve the Lasso baselines on 60, 100, 300, and 1000 dimensional problems by 45.7%, 19.2%, 19.7% and 15.5%, respectively.

Related papers

• Benchmarking Lie-Algebraic Pretraining and Non-Variational QWOA for the MaxCut Problem [4.103893081207555]
  This paper provides a comparative performance analysis of two strategies designed to improve trainability.<n>We benchmark both methods on the unweighted Maxcut problem using a circuit depth of $p = 256$ across 200 Erds-Rényi and 200 3-regular graphs.<n>Both approaches significantly improve upon the standard randomly QWOA. NV-QWOA attains a mean approximation ratio of 98.9% in just 60 iterations, while the Lie-algebraic pretrained QWOA improves to 77.71% after 500 iterations.
  arXiv  Detail & Related papers  (2025-12-28T09:42:02Z)
• Hyperellipsoid Density Sampling: Exploitative Sequences to Accelerate High-Dimensional Optimization [0.0]
  An adaptive sampling strategy is presented to accelerate optimization in the search space.<n>The method, referred to as Hyperellipsoid Density Sampling (HDS), generates its sequences by defining multiple hyperellipsoids.<n>The results show statistically significant improvements in solution geometric mean error, with average performance gains ranging from 3% in 30D to 37% in 10D.
  arXiv  Detail & Related papers  (2025-11-11T05:12:00Z)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data? [82.09573568241724]
  EssenceBench is a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA)<n>Our approach yields superior compression results with lower reconstruction error and markedly higher efficiency.<n>On the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples.
  arXiv  Detail & Related papers  (2025-10-12T05:38:10Z)
• Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization [4.1234601543641904]
  Black-box optimization problems with high evaluation costs entail solving black-box optimization problems with sample efficiency.<n>We propose a neural network-based surrogate to model the mapping from the high-dimensional input variables to the objective function.<n>We demonstrate the compelling performance of the proposed (ENS-)LoRA-VBLL approaches on various high-dimensional benchmarks and the real-world molecular optimization tasks.
  arXiv  Detail & Related papers  (2025-10-01T21:28:50Z)
• ESSA: Evolutionary Strategies for Scalable Alignment [8.418036456622158]
  We present ESSA, a gradient-free framework that aligns Large Language Models (LLMs) using only forward inference and black-box optimization.<n>ESSA improves the test accuracy of Qwen2.5-Math-7B by 12.6% on GSM8K and 14.8% on PRM800K, and raises the accuracy of LLaMA3.1-8B on IFEval by 22.5%.<n>In large-scale settings ESSA shows stronger scaling than gradient-based methods.
  arXiv  Detail & Related papers  (2025-07-06T16:23:07Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• Scalable Differentially Private Bayesian Optimization [17.28046301424826]
  We develop a method to privately estimate potentially high-dimensional parameter spaces using Gradient Informative Bayesian Optimization. Our theoretical results prove that under suitable conditions, our method converges exponentially fast to a ball around the optimal parameter configuration. Regardless of whether the assumptions are satisfied, we show that our algorithm maintains privacy and empirically demonstrates superior performance to existing methods.
  arXiv  Detail & Related papers  (2025-02-09T21:49:50Z)
• HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization [55.972018549438964]
  Federated fine-tuning of pre-trained Large Language Models (LLMs) enables task-specific adaptation across diverse datasets while preserving privacy.<n>We propose HAFLQ (Heterogeneous Adaptive Federated Low-Rank Adaptation Fine-tuned LLM with Quantization), a novel framework for efficient and scalable fine-tuning of LLMs in heterogeneous environments.<n> Experimental results on the text classification task demonstrate that HAFLQ reduces memory usage by 31%, lowers communication cost by 49%, improves accuracy by 50%, and achieves faster convergence compared to the baseline method.
  arXiv  Detail & Related papers  (2024-11-10T19:59:54Z)
• Scattered Forest Search: Smarter Code Space Exploration with LLMs [55.71665969800222]
  We propose SCATTERED FOREST SEARCH (SFS), a novel approach that improves solution diversity and better exploits feedback during evolutionary search. Our approach scales more efficiently than existing search techniques, including tree search, line search, and repeated sampling.
  arXiv  Detail & Related papers  (2024-10-22T01:58:29Z)
• ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models [14.310720048047136]
  ALPS is an optimization-based framework that tackles the pruning problem using the operator splitting technique and a preconditioned gradient conjugate-based post-processing step. Our approach incorporates novel techniques to accelerate and theoretically guarantee convergence while leveraging vectorization and GPU parallelism for efficiency. On the OPT-30B model with 70% sparsity, ALPS achieves a 13% reduction in test perplexity on the WikiText dataset and a 19% improvement in zero-shot benchmark performance compared to existing methods.
  arXiv  Detail & Related papers  (2024-06-12T02:57:41Z)
• Uncertainty-Aware Testing-Time Optimization for 3D Human Pose Estimation [68.75387874066647]
  We propose an Uncertainty-Aware testing-time optimization framework for 3D human pose estimation. Our approach outperforms the previous best result by a large margin of 4.5% on Human3.6M.
  arXiv  Detail & Related papers  (2024-02-04T04:28:02Z)
• HomOpt: A Homotopy-Based Hyperparameter Optimization Method [10.11271414863925]
  We propose HomOpt, a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. We show how HomOpt can boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous discrete, and categorical domain spaces.
  arXiv  Detail & Related papers  (2023-08-07T06:01:50Z)
• Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration [59.6021678234829]
  We propose a novel method to restore the intermediate features for two sparsely sampled and adjacent video frames. With the integration of our method, the efficiency of three commonly used baselines has been improved by over 50%, with a mere 0.5% reduction in recognition accuracy.
  arXiv  Detail & Related papers  (2023-07-27T13:52:42Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization [73.96101108943986]
  A Needle-in-a-Haystack problem arises when there is an extreme imbalance of optimum conditions relative to the size of the dataset. We present a Zooming Memory-Based Initialization algorithm that builds on conventional Bayesian optimization principles.
  arXiv  Detail & Related papers  (2022-08-26T23:57:41Z)
• Improving LSHADE by means of a pre-screening mechanism [0.0]
  The paper introduces an extension to the well-known LSHADE algorithm in the form of a pre-screening mechanism (psLSHADE) The proposed pre-screening relies on the three following components: a specific initial sampling procedure, an archive of samples, and a global linear meta-model of a fitness function. The performance of psLSHADE is evaluated using the CEC2021 benchmark in an expensive scenario with an optimization budget of 102-104 FFEs per dimension.
  arXiv  Detail & Related papers  (2022-04-08T14:52:46Z)
• Certifiable Outlier-Robust Geometric Perception: Exact Semidefinite Relaxations and Scalable Global Optimization [29.738513596063946]
  We propose the first general framework to design cert algorithms for robust geometric perception in the presence of outliers. Our experiments demonstrate that our SDP relaxation is exact with up to outliers across applications.
  arXiv  Detail & Related papers  (2021-09-07T21:42:16Z)
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization [4.191847852775072]
  We present an alternative objective that is equivalent to a Probably Approximately Correct-Bayes (PAC-Bayes) bound on the expected out-of-sample error. We then devise an efficient gradient-based algorithm to minimize this objective.
  arXiv  Detail & Related papers  (2020-08-14T15:54:51Z)
• Implicit differentiation of Lasso-type models for hyperparameter optimization [82.73138686390514]
  We introduce an efficient implicit differentiation algorithm, without matrix inversion, tailored for Lasso-type problems. Our approach scales to high-dimensional data by leveraging the sparsity of the solutions.
  arXiv  Detail & Related papers  (2020-02-20T18:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.