The Variability of Model Specification

• URL: http://arxiv.org/abs/2110.02490v1
• Date: Wed, 6 Oct 2021 03:59:19 GMT
• Title: The Variability of Model Specification
• Authors: Joseph R. Barr, Peter Shaw, Marcus Sobel
• Abstract summary: It's regarded as an axiom that a good model is one that compromises between bias and variance. We investigate various regression model frameworks, including generalized linear models, Cox proportional hazard models, ARMA, and illustrate how misspecifying a model affects the variance.
• Score: 2.4939887831898457
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: It's regarded as an axiom that a good model is one that compromises between bias and variance. The bias is measured in training cost, while the variance of a (say, regression) model is measure by the cost associated with a validation set. If reducing bias is the goal, one will strive to fetch as complex a model as necessary, but complexity is invariably coupled with variance: greater complexity implies greater variance. In practice, driving training cost to near zero does not pose a fundamental problem; in fact, a sufficiently complex decision tree is perfectly capable of driving training cost to zero; however, the problem is often with controlling the model's variance. We investigate various regression model frameworks, including generalized linear models, Cox proportional hazard models, ARMA, and illustrate how misspecifying a model affects the variance.

Related papers

• Pre-Trained AI Model Assisted Online Decision-Making under Missing Covariates: A Theoretical Perspective [12.160708336715489]
  "Model elasticity" provides a unified way to characterize the regret incurred due to model imputation.<n>We show that under the missing at random (MAR) setting, it is possible to sequentially calibrate the pre-trained model.<n>Our analysis highlights the practical value of having an accurate pre-trained model in sequential decision-making tasks.
  arXiv  Detail & Related papers  (2025-07-10T15:33:27Z)
• Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning [6.278498348219108]
  We revisit model complexity from first principles, by first reinterpreting and then extending the classical statistical concept of (effective) degrees of freedom. We demonstrate the utility of our proposed complexity measures through a mix of conceptual arguments, theory, and experiments.
  arXiv  Detail & Related papers  (2024-10-02T06:09:57Z)
• Are Linear Regression Models White Box and Interpretable? [0.0]
  Explainable artificial intelligence (XAI) is a set of tools and algorithms that applied or embedded to machine learning models to understand and interpret the models. Simple models including linear regression are easy to implement, has less computational complexity and easy to visualize the output.
  arXiv  Detail & Related papers  (2024-07-16T21:05:51Z)
• Scaling Laws in Linear Regression: Compute, Parameters, and Data [86.48154162485712]
  We study the theory of scaling laws in an infinite dimensional linear regression setup. We show that the reducible part of the test error is $Theta(-(a-1) + N-(a-1)/a)$. Our theory is consistent with the empirical neural scaling laws and verified by numerical simulation.
  arXiv  Detail & Related papers  (2024-06-12T17:53:29Z)
• Tradeoff of generalization error in unsupervised learning [0.6554326244334868]
  unsupervised learning generally exhibits a two-component tradeoff of the generalization error (GE) Using a more complex model reduces the model error at the cost of the data error, with the data error playing a more significant role for a smaller training dataset. Our results also indicate that the optimal model tends to be more complex when the data to be learned are more complex.
  arXiv  Detail & Related papers  (2023-03-10T05:50:17Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition [34.235007566913396]
  We describe an interpretable, symmetric decomposition of the variance into terms associated with the labels. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior. We also analyze the strikingly rich phenomenology that arises.
  arXiv  Detail & Related papers  (2020-11-04T21:04:02Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models [0.0]
  The bias-variance trade-off is a central concept in supervised learning. Modern Deep Learning methods flout this dogma, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-26T22:31:04Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.