Strong screening rules for group-based SLOPE models

• URL: http://arxiv.org/abs/2405.15357v1
• Date: Fri, 24 May 2024 08:48:06 GMT
• Title: Strong screening rules for group-based SLOPE models
• Authors: Fabio Feser, Marina Evangelou,
• Abstract summary: We develop screening rules for group-based Sorted L-One Penalized Estimation (SLOPE) models. The developed rules are applicable for the wider family of group-based models, including OSCAR.
• Score: 0.49109372384514843
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tuning the regularization parameter in penalized regression models is an expensive task, requiring multiple models to be fit along a path of parameters. Strong screening rules drastically reduce computational costs by lowering the dimensionality of the input prior to fitting. We develop strong screening rules for group-based Sorted L-One Penalized Estimation (SLOPE) models: Group SLOPE and Sparse-group SLOPE. The developed rules are applicable for the wider family of group-based OWL models, including OSCAR. Our experiments on both synthetic and real data show that the screening rules significantly accelerate the fitting process. The screening rules make it accessible for group SLOPE and sparse-group SLOPE to be applied to high-dimensional datasets, particularly those encountered in genetics.

Related papers

• Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification [76.14641982122696]
  We propose a constraint learning schema for fine-tuning Large Language Models (LLMs) with attribute control. We show that our approach leads to an LLM that produces fewer inappropriate responses while achieving competitive performance on benchmarks and a toxicity detection task.
  arXiv  Detail & Related papers  (2024-10-07T23:38:58Z)
• Group Robust Preference Optimization in Reward-free RLHF [23.622835830345725]
  We propose a novel Group Robust Preference Optimization (GRPO) method to align large language models to individual groups' preferences robustly. To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies.
  arXiv  Detail & Related papers  (2024-05-30T17:50:04Z)
• Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
  Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups. We reformulate the group DRO framework by proposing Q-Diversity. Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
  arXiv  Detail & Related papers  (2023-05-20T07:02:27Z)
• AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization [109.91265884632239]
  Group distributionally robust optimization (G-DRO) can minimize the worst-case loss over a set of pre-defined groups over training data. We propose AGRO -- Adversarial Group discovery for Distributionally Robust Optimization. AGRO results in 8% higher model performance on average on known worst-groups, compared to prior group discovery approaches.
  arXiv  Detail & Related papers  (2022-12-02T00:57:03Z)
• The non-overlapping statistical approximation to overlapping group lasso [4.197110761923662]
  We propose a separable penalty as an approximation of the overlapping group lasso penalty. Thanks to the separability, the computation of regularization based on our penalty is substantially faster than that of the overlapping group lasso. We show that the estimator based on the proposed separable penalty is statistically equivalent to the one based on the overlapping group lasso penalty.
  arXiv  Detail & Related papers  (2022-11-16T21:21:41Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Just Train Twice: Improving Group Robustness without Training Group Information [101.84574184298006]
  Standard training via empirical risk minimization can produce models that achieve high accuracy on average but low accuracy on certain groups. Prior approaches that achieve high worst-group accuracy, like group distributionally robust optimization (group DRO) require expensive group annotations for each training point. We propose a simple two-stage approach, JTT, that first trains a standard ERM model for several epochs, and then trains a second model that upweights the training examples that the first model misclassified.
  arXiv  Detail & Related papers  (2021-07-19T17:52:32Z)
• Cluster Regularization via a Hierarchical Feature Regression [0.0]
  This paper proposes a novel cluster-based regularization - the hierarchical feature regression (HFR) It mobilizes insights from the domains of machine learning and graph theory to estimate parameters along a supervised hierarchical representation of the predictor set. An application to the prediction of economic growth is used to illustrate the HFR's effectiveness in an empirical setting.
  arXiv  Detail & Related papers  (2021-07-10T13:03:01Z)
• Inference post Selection of Group-sparse Regression Models [2.1485350418225244]
  Conditional inference provides a rigorous approach to counter bias when data from automated model selections is reused for inference. We develop in this paper a statistically consistent Bayesian framework to assess uncertainties within linear models. Finding wide applications when genes, proteins, genetic variants, neuroimaging measurements are grouped respectively by their biological pathways, molecular functions, regulatory regions, cognitive roles, these models are selected through a useful class of group-sparse learning algorithms.
  arXiv  Detail & Related papers  (2020-12-31T15:43:26Z)
• Overcoming Data Sparsity in Group Recommendation [52.00998276970403]
  Group recommender systems should be able to accurately learn not only users' personal preferences but also preference aggregation strategy. In this paper, we take Bipartite Graphding Model (BGEM), the self-attention mechanism and Graph Convolutional Networks (GCNs) as basic building blocks to learn group and user representations in a unified way.
  arXiv  Detail & Related papers  (2020-10-02T07:11:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.