Regularization Trade-offs with Fake Features

• URL: http://arxiv.org/abs/2212.00433v2
• Date: Tue, 5 Dec 2023 08:12:53 GMT
• Title: Regularization Trade-offs with Fake Features
• Authors: Martin Hellkvist and Ay\c{c}a \"Oz\c{c}elikkale and Anders Ahl\'en
• Abstract summary: This paper considers a framework where the possibly overparametrized model includes fake features. We present a non-asymptotic high-probability bound on the generalization error of the ridge regression problem.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent successes of massively overparameterized models have inspired a new line of work investigating the underlying conditions that enable overparameterized models to generalize well. This paper considers a framework where the possibly overparametrized model includes fake features, i.e., features that are present in the model but not in the data. We present a non-asymptotic high-probability bound on the generalization error of the ridge regression problem under the model misspecification of having fake features. Our highprobability results provide insights into the interplay between the implicit regularization provided by the fake features and the explicit regularization provided by the ridge parameter. Numerical results illustrate the trade-off between the number of fake features and how the optimal ridge parameter may heavily depend on the number of fake features.

Related papers

• Aliasing and Label-Independent Decomposition of Risk: Beyond the bias-variance trade-off [0.0]
  A central problem in data science is to use potentially noisy samples to predict function values for unseen inputs. We introduce an alternative paradigm called the generalized aliasing decomposition (GAD) GAD can be explicitly calculated from the relationship between model class and samples without seeing any data labels.
  arXiv  Detail & Related papers  (2024-08-15T17:49:24Z)
• On the Parameter Identifiability of Partially Observed Linear Causal Models [23.08796869216895]
  We investigate whether the edge coefficients can be recovered given the causal structure and partially observed data. We identify three types of indeterminacy for the parameters in partially observed linear causal models. We propose a novel likelihood-based parameter estimation method that addresses the variance indeterminacy of latent variables in a specific way.
  arXiv  Detail & Related papers  (2024-07-24T03:43:55Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Should We Learn Most Likely Functions or Parameters? [51.133793272222874]
  We investigate the benefits and drawbacks of directly estimating the most likely function implied by the model and the data. We find that function-space MAP estimation can lead to flatter minima, better generalization, and improved to overfitting.
  arXiv  Detail & Related papers  (2023-11-27T16:39:55Z)
• Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers [66.36045164286854]
  We analyze a set of existing bias features and demonstrate there is no single model that works best for all the cases. By choosing an appropriate bias model, we can obtain a better robustness result than baselines with a more sophisticated model design.
  arXiv  Detail & Related papers  (2022-10-28T17:52:10Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Estimation and Model Misspecification: Fake and Missing Features [0.0]
  We consider estimation under model misspecification where there is a mismatch between the underlying system and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake and missing features.
  arXiv  Detail & Related papers  (2022-03-07T13:50:15Z)
• Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation [11.219641045667055]
  We propose an end-to-end approach for model learning which directly optimize the expected returns using implicit differentiation. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.
  arXiv  Detail & Related papers  (2021-06-06T23:15:49Z)
• Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately [83.68135652247496]
  A natural remedy is to remove spurious features from the model. We show that removal of spurious features can decrease accuracy due to inductive biases. We also show that robust self-training can remove spurious features without affecting the overall accuracy.
  arXiv  Detail & Related papers  (2020-12-07T23:08:59Z)
• Non-parametric Models for Non-negative Functions [48.7576911714538]
  We provide the first model for non-negative functions from the same good linear models. We prove that it admits a representer theorem and provide an efficient dual formulation for convex problems.
  arXiv  Detail & Related papers  (2020-07-08T07:17:28Z)
• Causal Inference with Deep Causal Graphs [0.0]
  Parametric causal modelling techniques rarely provide functionality for counterfactual estimation. Deep Causal Graphs is an abstract specification of the required functionality for a neural network to model causal distributions. We demonstrate its expressive power in modelling complex interactions and showcase applications to machine learning explainability and fairness.
  arXiv  Detail & Related papers  (2020-06-15T13:03:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.