Robust Output Analysis with Monte-Carlo Methodology

• URL: http://arxiv.org/abs/2207.13612v3
• Date: Wed, 25 Oct 2023 17:37:35 GMT
• Title: Robust Output Analysis with Monte-Carlo Methodology
• Authors: Kimia Vahdat and Sara Shashaani
• Abstract summary: In predictive modeling with simulation or machine learning, it is critical to accurately assess the quality of estimated values. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In predictive modeling with simulation or machine learning, it is critical to accurately assess the quality of estimated values through output analysis. In recent decades output analysis has become enriched with methods that quantify the impact of input data uncertainty in the model outputs to increase robustness. However, most developments are applicable assuming that the input data adheres to a parametric family of distributions. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling. This framework provides nonparametric quantification of the variance and bias induced in the outputs with higher-order accuracy. Our new bias-corrected estimation from the model outputs leverages the extension of fast iterative bootstrap sampling and higher-order influence functions. For the scalability of the proposed estimation methods, we devise budget-optimal rules and leverage control variates for variance reduction. Our theoretical and numerical results demonstrate a clear advantage in building more robust confidence intervals from the model outputs with higher coverage probability.

Related papers

• Probabilistic Surrogate Model for Accelerating the Design of Electric Vehicle Battery Enclosures for Crash Performance [0.0]
  This paper presents a probabilistic surrogate model for the accelerated design of electric vehicle battery enclosures with a focus on crash performance. The model was trained using data generated from thermoforming and crash simulations over a range of material and process parameters. validation against new simulation data demonstrated the model's predictive accuracy with mean absolute percentage errors within 8.08% for all output variables.
  arXiv  Detail & Related papers  (2024-08-06T21:03:16Z)
• Predictability Analysis of Regression Problems via Conditional Entropy Estimations [1.8913544072080544]
  Conditional entropy estimators are developed to assess predictability in regression problems. Experiments on synthesized and real-world datasets demonstrate the robustness and utility of these estimators.
  arXiv  Detail & Related papers  (2024-06-06T07:59:19Z)
• Fusion of Gaussian Processes Predictions with Monte Carlo Sampling [61.31380086717422]
  In science and engineering, we often work with models designed for accurate prediction of variables of interest. Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes.
  arXiv  Detail & Related papers  (2024-03-03T04:21:21Z)
• Boosted Control Functions [10.503777692702952]
  This work aims to bridge the gap between causal effect estimation and prediction tasks. We establish a novel connection between the field of distribution from machine learning, and simultaneous equation models and control function from econometrics. Within this framework, we propose a strong notion of invariance for a predictive model and compare it with existing (weaker) versions.
  arXiv  Detail & Related papers  (2023-10-09T15:43:46Z)
• Efficient Propagation of Uncertainty via Reordering Monte Carlo Samples [0.7087237546722617]
  Uncertainty propagation is a technique to determine model output uncertainties based on the uncertainty in its input variables. In this work, we investigate the hypothesis that while all samples are useful on average, some samples must be more useful than others. We introduce a methodology to adaptively reorder MC samples and show how it results in reduction of computational expense of UP processes.
  arXiv  Detail & Related papers  (2023-02-09T21:28:15Z)
• Bayesian Sparse Regression for Mixed Multi-Responses with Application to Runtime Metrics Prediction in Fog Manufacturing [6.288767115532775]
  Fog manufacturing can greatly enhance traditional manufacturing systems through distributed computation Fog units. It is known that the predictive offloading methods highly depend on accurate prediction and uncertainty quantification of runtime performance metrics. We propose a Bayesian sparse regression for multivariate mixed responses to enhance the prediction of runtime performance metrics.
  arXiv  Detail & Related papers  (2022-10-10T16:14:08Z)
• Heavy-tailed Streaming Statistical Estimation [58.70341336199497]
  We consider the task of heavy-tailed statistical estimation given streaming $p$ samples. We design a clipped gradient descent and provide an improved analysis under a more nuanced condition on the noise of gradients.
  arXiv  Detail & Related papers  (2021-08-25T21:30:27Z)
• Latent Causal Invariant Model [128.7508609492542]
  Current supervised learning can learn spurious correlation during the data-fitting process. We propose a Latent Causal Invariance Model (LaCIM) which pursues causal prediction.
  arXiv  Detail & Related papers  (2020-11-04T10:00:27Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Balance-Subsampled Stable Prediction [55.13512328954456]
  We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design. A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift. Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
  arXiv  Detail & Related papers  (2020-06-08T07:01:38Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.