Parameter uncertainties for imperfect surrogate models in the low-noise regime

• URL: http://arxiv.org/abs/2402.01810v5
• Date: Tue, 05 Nov 2024 18:54:14 GMT
• Title: Parameter uncertainties for imperfect surrogate models in the low-noise regime
• Authors: Thomas D Swinburne, Danny Perez,
• Abstract summary: We analyze the generalization error of misspecified, near-deterministic surrogate models. We show posterior distributions must cover every training point to avoid a divergent generalization error. This is demonstrated on model problems before application to thousand dimensional datasets in atomistic machine learning.
• Score: 0.3069335774032178
• License:
• Abstract: Bayesian regression determines model parameters by minimizing the expected loss, an upper bound to the true generalization error. However, the loss ignores misspecification, where models are imperfect. Parameter uncertainties from Bayesian regression are thus significantly underestimated and vanish in the large data limit. This is particularly problematic when building models of low-noise, or near-deterministic, calculations, as the main source of uncertainty is neglected. We analyze the generalization error of misspecified, near-deterministic surrogate models, a regime of broad relevance in science and engineering. We show posterior distributions must cover every training point to avoid a divergent generalization error and design an ansatz that respects this constraint, which for linear models incurs minimal overhead. This is demonstrated on model problems before application to thousand dimensional datasets in atomistic machine learning. Our efficient misspecification-aware scheme gives accurate prediction and bounding of test errors where existing schemes fail, allowing this important source of uncertainty to be incorporated in computational workflows.

Related papers

• Uncertainty Quantification of Surrogate Models using Conformal Prediction [7.445864392018774]
  We formalise a conformal prediction framework that satisfies predictions in a model-agnostic manner, requiring near-zero computational costs. The paper looks at providing statistically valid error bars for deterministic models, as well as crafting guarantees to the error bars of probabilistic models.
  arXiv  Detail & Related papers  (2024-08-19T10:46:19Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• ALUM: Adversarial Data Uncertainty Modeling from Latent Model Uncertainty Compensation [25.67258563807856]
  We propose a novel method called ALUM to handle the model uncertainty and data uncertainty in a unified scheme. Our proposed ALUM is model-agnostic which can be easily implemented into any existing deep model with little extra overhead.
  arXiv  Detail & Related papers  (2023-03-29T17:24:12Z)
• Training Normalizing Flows with the Precision-Recall Divergence [73.92251251511199]
  We show that achieving a specified precision-recall trade-off corresponds to minimising -divergences from a family we call the em PR-divergences We propose a novel generative model that is able to train a normalizing flow to minimise any -divergence, and in particular, achieve a given precision-recall trade-off.
  arXiv  Detail & Related papers  (2023-02-01T17:46:47Z)
• Rigorous Assessment of Model Inference Accuracy using Language Cardinality [5.584832154027001]
  We develop a systematic approach that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools.
  arXiv  Detail & Related papers  (2022-11-29T21:03:26Z)
• Data Uncertainty without Prediction Models [0.8223798883838329]
  We propose an uncertainty estimation method named a Distance-weighted Class Impurity without explicit use of prediction models. We verified that the Distance-weighted Class Impurity works effectively regardless of prediction models.
  arXiv  Detail & Related papers  (2022-04-25T13:26:06Z)
• Improving Generalization via Uncertainty Driven Perturbations [107.45752065285821]
  We consider uncertainty-driven perturbations of the training data points. Unlike loss-driven perturbations, uncertainty-guided perturbations do not cross the decision boundary. We show that UDP is guaranteed to achieve the robustness margin decision on linear models.
  arXiv  Detail & Related papers  (2022-02-11T16:22:08Z)
• Theoretical characterization of uncertainty in high-dimensional linear classification [24.073221004661427]
  We show that uncertainty for learning from limited number of samples of high-dimensional input data and labels can be obtained by the approximate message passing algorithm. We discuss how over-confidence can be mitigated by appropriately regularising, and show that cross-validating with respect to the loss leads to better calibration than with the 0/1 error.
  arXiv  Detail & Related papers  (2022-02-07T15:32:07Z)
• Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
  This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets. Part of the challenge of learning robust models lies in the influence of unobserved confounders. We demonstrate the empirical performance of our approach on healthcare data from different modalities.
  arXiv  Detail & Related papers  (2020-07-21T08:18:06Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.