Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling

• URL: http://arxiv.org/abs/2411.17826v1
• Date: Tue, 26 Nov 2024 19:05:46 GMT
• Title: Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling
• Authors: Aman Sinha, Payam Nikdel, Supratik Paul, Shimon Whiteson,
• Abstract summary: This paper introduces adaptive multifidelity sampling (BAMS) to achieve efficient discovery while simultaneously estimating the rate of adverse events. We demonstrate that BAMS discovers 10 times as many issues as Monte Carlo (MC) and importance sampling (IS) baselines, while at the same time generating rate estimates with variances 15 and 6 times narrower than MC and IS baselines respectively.
• Score: 36.106248147331804
• License:
• Abstract: Ensuring the safety of autonomous vehicles (AVs) requires both accurate estimation of their performance and efficient discovery of potential failure cases. This paper introduces Bayesian adaptive multifidelity sampling (BAMS), which leverages the power of adaptive Bayesian sampling to achieve efficient discovery while simultaneously estimating the rate of adverse events. BAMS prioritizes exploration of regions with potentially low performance, leading to the identification of novel and critical scenarios that traditional methods might miss. Using real-world AV data we demonstrate that BAMS discovers 10 times as many issues as Monte Carlo (MC) and importance sampling (IS) baselines, while at the same time generating rate estimates with variances 15 and 6 times narrower than MC and IS baselines respectively.

Related papers

• Confidence Aware Learning for Reliable Face Anti-spoofing [52.23271636362843]
  We propose a Confidence Aware Face Anti-spoofing model, which is aware of its capability boundary. We estimate its confidence during the prediction of each sample. Experiments show that the proposed CA-FAS can effectively recognize samples with low prediction confidence.
  arXiv  Detail & Related papers  (2024-11-02T14:29:02Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• MedBN: Robust Test-Time Adaptation against Malicious Test Samples [11.397666167665484]
  Test-time adaptation (TTA) has emerged as a promising solution to address performance decay due to unforeseen distribution shifts between training and test data. Previous studies have uncovered security vulnerabilities within TTA even when a small proportion of the test batch is maliciously manipulated. We propose median batch normalization (MedBN), leveraging the robustness of the median for statistics estimation within the batch normalization layer during test-time inference.
  arXiv  Detail & Related papers  (2024-03-28T11:33:02Z)
• Neural Importance Sampling for Rapid and Reliable Gravitational-Wave Inference [59.040209568168436]
  We first generate a rapid proposal for the Bayesian posterior using neural networks, and then attach importance weights based on the underlying likelihood and prior. This provides (1) a corrected posterior free from network inaccuracies, (2) a performance diagnostic (the sample efficiency) for assessing the proposal and identifying failure cases, and (3) an unbiased estimate of the Bayesian evidence. We carry out a large study analyzing 42 binary black hole mergers observed by LIGO and Virgo with the SEOBNRv4PHM and IMRPhenomHMXP waveform models.
  arXiv  Detail & Related papers  (2022-10-11T18:00:02Z)
• Empirical Bayesian Approaches for Robust Constraint-based Causal Discovery under Insufficient Data [38.883810061897094]
  Causal discovery methods assume data sufficiency, which may not be the case in many real world datasets. We propose Bayesian-augmented frequentist independence tests to improve the performance of constraint-based causal discovery methods under insufficient data. Experiments show significant performance improvement in terms of both accuracy and efficiency over SOTA methods.
  arXiv  Detail & Related papers  (2022-06-16T21:08:49Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Accelerated Policy Evaluation: Learning Adversarial Environments with Adaptive Importance Sampling [19.81658135871748]
  A biased or inaccurate policy evaluation in a safety-critical system could potentially cause unexpected catastrophic failures. We propose the Accelerated Policy Evaluation (APE) method, which simultaneously uncovers rare events and estimates the rare event probability. APE is scalable to large discrete or continuous spaces by incorporating function approximators.
  arXiv  Detail & Related papers  (2021-06-19T20:03:26Z)
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting [67.77102283276409]
  We describe two types of forecasting problems: regular grid-based and graph-based. We analyze UQ methods from both the Bayesian and the frequentist point view, casting in a unified framework via statistical decision theory. Through extensive experiments on real-world road network traffic, epidemics, and air quality forecasting tasks, we reveal the statistical computational trade-offs for different UQ methods.
  arXiv  Detail & Related papers  (2021-05-25T14:35:46Z)
• Entropy-based adaptive design for contour finding and estimating reliability [0.24466725954625884]
  In reliability analysis, methods used to estimate failure probability are often limited by the costs associated with model evaluations. We introduce an entropy-based GP adaptive design that, when paired with MFIS, provides more accurate failure probability estimates. Illustrative examples are provided on benchmark data as well as an application to an impact damage simulator for National Aeronautics and Space Administration (NASA) spacesuits.
  arXiv  Detail & Related papers  (2021-05-24T15:41:15Z)
• Adaptive Sampling for Estimating Distributions: A Bayesian Upper Confidence Bound Approach [30.76846526324949]
  A Bayesian variant of the existing upper confidence bound (UCB) based approaches is proposed. The effectiveness of this strategy is discussed using data obtained from a seroprevalence survey in Los Angeles county.
  arXiv  Detail & Related papers  (2020-12-08T00:53:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.