Multi-Modal Conformal Prediction Regions with Simple Structures by Optimizing Convex Shape Templates

• URL: http://arxiv.org/abs/2312.07434v2
• Date: Tue, 25 Jun 2024 16:10:41 GMT
• Title: Multi-Modal Conformal Prediction Regions with Simple Structures by Optimizing Convex Shape Templates
• Authors: Renukanandan Tumu, Matthew Cleaveland, Rahul Mangharam, George J. Pappas, Lars Lindemann,
• Abstract summary: Conformal prediction is a statistical tool for producing prediction regions for machine learning models that are valid with high probability. A key component of conformal prediction algorithms is a emphnon-conformity score function that quantifies how different a model's prediction is from the unknown ground truth value. We propose a method that optimize parameterized emphshape template functions over calibration data, which results in non-conformity score functions that produce prediction regions with minimum volume.
• Score: 19.504348671777006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal prediction is a statistical tool for producing prediction regions for machine learning models that are valid with high probability. A key component of conformal prediction algorithms is a \emph{non-conformity score function} that quantifies how different a model's prediction is from the unknown ground truth value. Essentially, these functions determine the shape and the size of the conformal prediction regions. While prior work has gone into creating score functions that produce multi-model prediction regions, such regions are generally too complex for use in downstream planning and control problems. We propose a method that optimizes parameterized \emph{shape template functions} over calibration data, which results in non-conformity score functions that produce prediction regions with minimum volume. Our approach results in prediction regions that are \emph{multi-modal}, so they can properly capture residuals of distributions that have multiple modes, and \emph{practical}, so each region is convex and can be easily incorporated into downstream tasks, such as a motion planner using conformal prediction regions. Our method applies to general supervised learning tasks, while we illustrate its use in time-series prediction. We provide a toolbox and present illustrative case studies of F16 fighter jets and autonomous vehicles, showing an up to $68\%$ reduction in prediction region area compared to a circular baseline region.

Related papers

• Conformal online model aggregation [29.43493007296859]
  This paper proposes a new approach towards conformal model aggregation in online settings. It is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
  arXiv  Detail & Related papers  (2024-03-22T15:40:06Z)
• Regression Trees for Fast and Adaptive Prediction Intervals [2.6763498831034043]
  We present a family of methods to calibrate prediction intervals for regression problems with local coverage guarantees. We create a partition by training regression trees and Random Forests on conformity scores. Our proposal is versatile, as it applies to various conformity scores and prediction settings.
  arXiv  Detail & Related papers  (2024-02-12T01:17:09Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Conformal Prediction Regions for Time Series using Linear Complementarity Programming [25.094249285804224]
  We propose an optimization-based method for reducing conservatism to enable long horizon planning and verification. We show that this problem can be cast as a mixed integer linear complementarity program (MILCP), which we then relax into a linear complementarity program (LCP)
  arXiv  Detail & Related papers  (2023-04-03T15:32:38Z)
• RbX: Region-based explanations of prediction models [69.3939291118954]
  Region-based explanations (RbX) is a model-agnostic method to generate local explanations of scalar outputs from a black-box prediction model. RbX is guaranteed to satisfy a "sparsity axiom," which requires that features which do not enter into the prediction model are assigned zero importance.
  arXiv  Detail & Related papers  (2022-10-17T03:38:06Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• MD-split+: Practical Local Conformal Inference in High Dimensions [0.5439020425819]
  MD-split+ is a practical local conformal approach that creates X partitions based on localized model performance. We discuss how our local partitions philosophically align with expected behavior from an unattainable conditional conformal inference approach.
  arXiv  Detail & Related papers  (2021-07-07T15:19:16Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.