Multi-model Ensemble Conformal Prediction in Dynamic Environments

• URL: http://arxiv.org/abs/2411.03678v1
• Date: Wed, 06 Nov 2024 05:57:28 GMT
• Title: Multi-model Ensemble Conformal Prediction in Dynamic Environments
• Authors: Erfan Hajihashemi, Yanning Shen,
• Abstract summary: We introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage.
• Score: 14.188004615463742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.

Related papers

• DANCE: Doubly Adaptive Neighborhood Conformal Estimation [12.643121779828526]
  We propose a doubly locally adaptive nearest-neighbor based conformal algorithm combining two novel nonconformity scores directly using the data's embedded representation.<n>We test against state-of-the-art local, task-adapted and zero-shot conformal baselines, demonstrating DANCE's superior blend of set size efficiency and robustness across various datasets.
  arXiv  Detail & Related papers  (2026-02-24T07:54:53Z)
• Enhanced Multi-model Online Conformal Prediction [25.495949162960624]
  Conformal prediction is a framework for uncertainty quantification that constructs prediction sets for previously unseen data.<n>The efficiency of these prediction sets, measured by their size, depends on the choice of the underlying learning model.<n>This work develops a novel multi-model online conformal prediction algorithm that reduces computational complexity and improves prediction efficiency.
  arXiv  Detail & Related papers  (2026-01-04T23:44:43Z)
• Distribution-informed Online Conformal Prediction [53.674678995825666]
  We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule.<n>COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate.<n>We prove that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.
  arXiv  Detail & Related papers  (2025-12-08T17:51:49Z)
• Adaptive Conformal Prediction Intervals Over Trajectory Ensembles [50.31074512684758]
  Future trajectories play an important role across domains such as autonomous driving, hurricane forecasting, and epidemic modeling.<n>We propose a unified framework based on conformal prediction that transforms sampled trajectories into calibrated prediction intervals with theoretical coverage guarantees.
  arXiv  Detail & Related papers  (2025-08-18T21:14:07Z)
• Graph-Structured Feedback Multimodel Ensemble Online Conformal Prediction [14.188004615463742]
  We propose a novel multi-model online conformal prediction algorithm.<n>It identifies a subset of effective models at each time step by collecting feedback from a bipartite graph.<n>A model is then selected from this subset to construct the prediction set, resulting in reduced computational complexity and smaller prediction sets.
  arXiv  Detail & Related papers  (2025-06-26T00:06:11Z)
• Minimum Volume Conformal Sets for Multivariate Regression [44.99833362998488]
  Conformal prediction provides a principled framework for constructing predictive sets with finite-sample validity. We propose an optimization-driven framework based on a novel loss function that directly learns minimum-conformity covering sets. Our approach optimize over prediction sets defined by arbitrary norm balls, including single and multi-norm formulations.
  arXiv  Detail & Related papers  (2025-03-24T18:54:22Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples. We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• Stochastic Online Conformal Prediction with Semi-Bandit Feedback [29.334511328067777]
  We consider the online learning setting, where examples arrive over time, and the goal is to construct prediction sets dynamically. We propose a novel conformal prediction algorithm targeted at this setting, and prove that it obtains sublinear regret compared to the optimal conformal predictor.
  arXiv  Detail & Related papers  (2024-05-22T00:42:49Z)
• Conformal online model aggregation [29.43493007296859]
  This paper proposes a new approach towards conformal model aggregation in online settings. It is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
  arXiv  Detail & Related papers  (2024-03-22T15:40:06Z)
• Regression Trees for Fast and Adaptive Prediction Intervals [2.6763498831034043]
  We present a family of methods to calibrate prediction intervals for regression problems with local coverage guarantees. We create a partition by training regression trees and Random Forests on conformity scores. Our proposal is versatile, as it applies to various conformity scores and prediction settings.
  arXiv  Detail & Related papers  (2024-02-12T01:17:09Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• CovarianceNet: Conditional Generative Model for Correct Covariance Prediction in Human Motion Prediction [71.31516599226606]
  We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories. Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
  arXiv  Detail & Related papers  (2021-09-07T09:38:24Z)
• Adaptive Conformal Inference Under Distribution Shift [0.0]
  We develop methods for forming prediction sets in an online setting where the data generating distribution is allowed to vary over time in an unknown fashion. Our framework builds on ideas from conformal inference to provide a general wrapper that can be combined with any black box method. We test our method, adaptive conformal inference, on two real world datasets and find that its predictions are robust to visible and significant distribution shifts.
  arXiv  Detail & Related papers  (2021-06-01T01:37:32Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation [74.06904875527556]
  We propose a generic and efficient segmentation framework to construct ensemble segmentation models. In the proposed method, ensemble models can be efficiently generated by using the layer selection method. We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
  arXiv  Detail & Related papers  (2020-05-21T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.