Estimation and Model Misspecification: Fake and Missing Features

• URL: http://arxiv.org/abs/2203.03398v1
• Date: Mon, 7 Mar 2022 13:50:15 GMT
• Title: Estimation and Model Misspecification: Fake and Missing Features
• Authors: Martin Hellkvist, Ay\c{c}a \"Oz\c{c}elikkale, Anders Ahl\'en
• Abstract summary: We consider estimation under model misspecification where there is a mismatch between the underlying system and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake and missing features.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider estimation under model misspecification where there is a model mismatch between the underlying system, which generates the data, and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake and missing features, as well as incorrect covariance assumptions on the unknowns and the noise. Here, features which are included in the model but are not present in the underlying system, and features which are not included in the model but are present in the underlying system, are referred to as fake and missing features, respectively. Under this framework, we characterize the estimation performance and reveal trade-offs between the missing and fake features and the possibly incorrect noise level assumption. In contrast to existing work focusing on incorrect covariance assumptions or missing features, fake features is a central component of our framework. Our results show that fake features can significantly improve the estimation performance, even though they are not correlated with the features in the underlying system. In particular, we show that the estimation error can be decreased by including more fake features in the model, even to the point where the model is overparametrized, i.e., the model contains more unknowns than observations.

Related papers

• Parameter uncertainties for imperfect surrogate models in the low-noise regime [0.3069335774032178]
  We analyze the generalization error of misspecified, near-deterministic surrogate models. We show posterior distributions must cover every training point to avoid a divergent generalization error. This is demonstrated on model problems before application to thousand dimensional datasets in atomistic machine learning.
  arXiv  Detail & Related papers  (2024-02-02T11:41:21Z)
• Regularization Trade-offs with Fake Features [0.0]
  This paper considers a framework where the possibly overparametrized model includes fake features. We present a non-asymptotic high-probability bound on the generalization error of the ridge regression problem.
  arXiv  Detail & Related papers  (2022-12-01T11:11:14Z)
• Rigorous Assessment of Model Inference Accuracy using Language Cardinality [5.584832154027001]
  We develop a systematic approach that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools.
  arXiv  Detail & Related papers  (2022-11-29T21:03:26Z)
• A view on model misspecification in uncertainty quantification [17.17262672213263]
  Estimating uncertainty of machine learning models is essential to assess the quality of the predictions that these models provide. Model misspecification always exists as models are mere simplifications or approximations to reality. This paper argues that model misspecification should receive more attention, by providing thought experiments and contextualizing these with relevant literature.
  arXiv  Detail & Related papers  (2022-10-30T20:26:43Z)
• Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers [66.36045164286854]
  We analyze a set of existing bias features and demonstrate there is no single model that works best for all the cases. By choosing an appropriate bias model, we can obtain a better robustness result than baselines with a more sophisticated model design.
  arXiv  Detail & Related papers  (2022-10-28T17:52:10Z)
• Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning [47.651130958272155]
  Underspecification refers to the existence of multiple models that are indistinguishable in their in-domain accuracy. We formalize the concept of underspecification and propose a method to identify and partially address it.
  arXiv  Detail & Related papers  (2022-07-06T11:20:40Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)
• Closed-form discovery of structural errors in models of chaotic systems by integrating Bayesian sparse regression and data assimilation [0.0]
  We introduce a framework named MEDIDA: Model Error Discovery with Interpretability and Data Assimilation. In MEDIDA, first the model error is estimated from differences between the observed states and model-predicted states. If observations are noisy, a data assimilation technique such as ensemble Kalman filter (EnKF) is first used to provide a noise-free analysis state of the system. Finally, an equation-discovery technique, such as the relevance vector machine (RVM), a sparsity-promoting Bayesian method, is used to identify an interpretable, parsimonious, closed
  arXiv  Detail & Related papers  (2021-10-01T17:19:28Z)
• Predicting with Confidence on Unseen Distributions [90.68414180153897]
  We connect domain adaptation and predictive uncertainty literature to predict model accuracy on challenging unseen distributions. We find that the difference of confidences (DoC) of a classifier's predictions successfully estimates the classifier's performance change over a variety of shifts. We specifically investigate the distinction between synthetic and natural distribution shifts and observe that despite its simplicity DoC consistently outperforms other quantifications of distributional difference.
  arXiv  Detail & Related papers  (2021-07-07T15:50:18Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately [83.68135652247496]
  A natural remedy is to remove spurious features from the model. We show that removal of spurious features can decrease accuracy due to inductive biases. We also show that robust self-training can remove spurious features without affecting the overall accuracy.
  arXiv  Detail & Related papers  (2020-12-07T23:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.