Confidence Calibration for Systems with Cascaded Predictive Modules

• URL: http://arxiv.org/abs/2309.12510v1
• Date: Thu, 21 Sep 2023 22:12:24 GMT
• Title: Confidence Calibration for Systems with Cascaded Predictive Modules
• Authors: Yunye Gong, Yi Yao, Xiao Lin, Ajay Divakaran, Melinda Gervasio
• Abstract summary: We present novel solutions based on conformal prediction to provide prediction intervals calibrated for a predictive system consisting of cascaded modules. Our key idea is to leverage module-level validation data to characterize the system-level error distribution without direct access to end-to-end validation data. In comparison to prediction intervals calibrated for individual modules, our solutions generate improved intervals with more accurate performance guarantees for system predictions.
• Score: 9.393699753285997
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples. However, considering an autonomous system consisting of multiple modules, prediction intervals constructed for individual modules fall short of accommodating uncertainty propagation over different modules and thus cannot provide reliable predictions on system behavior. We address this limitation and present novel solutions based on conformal prediction to provide prediction intervals calibrated for a predictive system consisting of cascaded modules (e.g., an upstream feature extraction module and a downstream regression module). Our key idea is to leverage module-level validation data to characterize the system-level error distribution without direct access to end-to-end validation data. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of proposed solutions. In comparison to prediction intervals calibrated for individual modules, our solutions generate improved intervals with more accurate performance guarantees for system predictions, which are demonstrated on both synthetic systems and real-world systems performing overlap prediction for indoor navigation using the Matterport3D dataset.

Related papers

• Adaptive Uncertainty Quantification for Trajectory Prediction Under Distributional Shift [6.029850098632435]
  Trajectory prediction models can infer both finite future trajectories and their associated uncertainties in an online setting. We propose the Conformal Uncertainty Quantification under Distribution Shift framework, CUQDS, to quantify the uncertainty of the predicted trajectories.
  arXiv  Detail & Related papers  (2024-06-17T21:25:36Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Deep Non-Parametric Time Series Forecaster [19.800783133682955]
  The proposed approach does not assume any parametric form for the predictive distribution and instead generates predictions by sampling from the empirical distribution according to a tunable strategy. We develop a global version of the proposed method that automatically learns the sampling strategy by exploiting the information across multiple related time series.
  arXiv  Detail & Related papers  (2023-12-22T12:46:30Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? [93.10694819127608]
  We propose a unified evaluation pipeline for forecasting methods with real-world perception inputs. Our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data.
  arXiv  Detail & Related papers  (2023-06-15T17:03:14Z)
• Online machine-learning forecast uncertainty estimation for sequential data assimilation [0.0]
  Quantifying forecast uncertainty is a key aspect of state-of-the-art numerical weather prediction and data assimilation systems. In this work a machine learning method is presented based on convolutional neural networks that estimates the state-dependent forecast uncertainty. The hybrid data assimilation method shows similar performance to the ensemble Kalman filter outperforming it when the ensembles are relatively small.
  arXiv  Detail & Related papers  (2023-05-12T19:23:21Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks [12.607426130997336]
  We propose a Bayesian deep learning based method, namely BDLPerf, that can incorporate uncertainty into the prediction model. We develop a novel uncertainty calibration technique to ensure the reliability of the confidence intervals generated by a Bayesian prediction model. Our experimental results on 10 real-world systems show that BDLPerf achieves higher accuracy than existing approaches.
  arXiv  Detail & Related papers  (2022-12-27T04:39:26Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)
• Semi-Modular Inference: enhanced learning in multi-modular models by tempering the influence of components [0.0]
  We show existing Modular/Cut-model inference is coherent, and write down a new family of Semi-Modular Inference schemes. We give a meta-learning criterion and estimation procedure to choose the inference scheme. We illustrate our methods on two standard test cases from the literature and a motivating archaeological data set.
  arXiv  Detail & Related papers  (2020-03-15T11:55:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.