Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks

• URL: http://arxiv.org/abs/2604.03541v2
• Date: Tue, 07 Apr 2026 17:44:33 GMT
• Title: Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks
• Abstract summary: We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations.<n>For prediction accuracy when the sample-to-feature ratio is sufficient, Ridge, Lasso, and ElasticNet are nearly interchangeable.<n>We advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes.
• Score: 0.1862063702912854
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This study surveys the historical development of regularization, tracing its evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features, Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations spanning a 7-dimensional manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.

Related papers

• Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloads [0.0]
  Analytical models of peak VRAM consumption for LLM inference decompose memory into weight-storage, KV-cache, and activation terms.<n>We evaluate this decomposition empirically within a strictly scoped measurement study.<n>We find no justification for deploying complex predictive VRAM models in highly quantized, weight-dominated regimes.
  arXiv  Detail & Related papers  (2026-08-15T08:39:43Z)
• Adversarial LassoNet: Robust Feature Selection via Stability-Driven Sparse Learning [9.282673102334869]
  AdLNet is a stability-driven sparse feature selection framework that integrates input-space adversarial perturbations with the hierarchical sparsity mechanism of LassoNet.<n> Experiments show that AdLNet maintains competitive sparse-selection performance while improving out-of-distribution robustness by 4.4% and feature support by 6.3% under nearly matched support sparsity on ColoredMNIST.
  arXiv  Detail & Related papers  (2026-07-04T12:01:53Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• LLMs can Compress LLMs: Adaptive Pruning by Agents [0.0]
  Post-training pruning has emerged as a promising approach to reduce computational costs while preserving performance.<n>We introduce agent-guided pruning, where a foundation model acts as an adaptive pruning agent.<n>We evaluate our approach on Q3 models (4B and 8B parameters) at approximately 45% sparsity, demonstrating substantial improvements over structured pruning baselines.
  arXiv  Detail & Related papers  (2026-01-14T18:45:36Z)
• From Memorization to Creativity: LLM as a Designer of Novel Neural-Architectures [48.83701310501069]
  Large language models (LLMs) excel in program synthesis, yet their ability to autonomously navigate neural architecture design-balancing reliability, performance, and structural novelty--remains underexplored.<n>We address this by placing a code-oriented LLM within a closed-loop synthesis framework, analyzing its evolution over 22 supervised fine-tuning cycles.
  arXiv  Detail & Related papers  (2026-01-06T13:20:28Z)
• Mitigating Spurious Correlations in NLI via LLM-Synthesized Counterfactuals and Dynamic Balanced Sampling [0.0]
  Natural Language Inference (NLI) models frequently rely on spurious correlations rather than semantic reasoning.<n>Existing mitigation strategies often incur high annotation costs or trigger catastrophic forgetting during fine-tuning.<n>We propose an automated, scalable pipeline to address these limitations.
  arXiv  Detail & Related papers  (2025-12-20T18:30:54Z)
• Machine-Precision Prediction of Low-Dimensional Chaotic Systems [0.0]
  We show that learning from low-dimensional chaotic systems can be achieved up to machine precision.<n>We obtain valid prediction times of 32 to 105 Lyapunov times for the Lorenz-63 system.<n>Our findings suggest that learning low-temporal chaotic systems from noisefree data is a problem.
  arXiv  Detail & Related papers  (2025-07-13T14:36:47Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Landslide Susceptibility Prediction Modeling Based on Self-Screening Deep Learning Model [9.7723814375467]
  A self-screening graph convolutional network and long short-term memory network (SGCN-LSTM) is proposed in this paper. The landslide samples with large errors outside the set threshold interval are eliminated by self-screening network. The nonlinear relationship between environmental factors can be extracted from both spatial nodes and time series.
  arXiv  Detail & Related papers  (2023-04-12T10:31:03Z)
• Machine learning tools to improve nonlinear modeling parameters of RC columns [0.0]
  This paper addresses two of the most significant barriers to improving nonlinear modeling provisions in seismic evaluation standards. It implements data fitting techniques capable of recognizing interdependencies between input parameters and nonlinear relationships between input parameters and model outputs. It was found that machine learning regression models and machine learning black-boxes were more accurate than current provisions in the ACI 369.1/ASCE 41 Standards.
  arXiv  Detail & Related papers  (2023-03-09T01:00:38Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder [59.69303945834122]
  We present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of time series. Our model parameterizes mean and variance for each time-stamp with flexible neural networks. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.
  arXiv  Detail & Related papers  (2021-02-02T06:15:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.