A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression

• URL: http://arxiv.org/abs/2501.10533v2
• Date: Mon, 03 Feb 2025 12:58:06 GMT
• Title: A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression
• Authors: Victor Dheur, Matteo Fontana, Yorick Estievenart, Naomi Desobry, Souhaib Ben Taieb,
• Abstract summary: We present a unified comparative study of nine conformal methods with different multi-output base models. We also introduce two novel classes of conformity scores for multi-output regression. One class is compatible with any generative model, while the other is computationally efficient, leveraging the properties of invertible generative models.
• Score: 1.747623282473278
• License:
• Abstract: Conformal prediction provides a powerful framework for constructing distribution-free prediction regions with finite-sample coverage guarantees. While extensively studied in univariate settings, its extension to multi-output problems presents additional challenges, including complex output dependencies and high computational costs, and remains relatively underexplored. In this work, we present a unified comparative study of nine conformal methods with different multivariate base models for constructing multivariate prediction regions within the same framework. This study highlights their key properties while also exploring the connections between them. Additionally, we introduce two novel classes of conformity scores for multi-output regression that generalize their univariate counterparts. These scores ensure asymptotic conditional coverage while maintaining exact finite-sample marginal coverage. One class is compatible with any generative model, offering broad applicability, while the other is computationally efficient, leveraging the properties of invertible generative models. Finally, we conduct a comprehensive empirical evaluation across 13 tabular datasets, comparing all the multi-output conformal methods explored in this work. To ensure a fair and consistent comparison, all methods are implemented within a unified code base.

Related papers

• Optimal Transport-based Conformal Prediction [8.302146576157497]
  Conformal Prediction (CP) is a principled framework for uncertainty in blackbox learning models. We introduce a novel CP procedure handling prediction score functions through a lens. We then adapt our method for quantifying multi-output regression and multiclass classification.
  arXiv  Detail & Related papers  (2025-01-31T09:48:28Z)
• Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing [49.60415088899208]
  We propose an efficient ensemble-selection approach that considers error diversity and avoids error accumulation. Results demonstrate that our approach outperforms each individual model as well as previous ensemble techniques.
  arXiv  Detail & Related papers  (2024-12-16T08:23:50Z)
• A Unified Combination Framework for Dependent Tests with Applications to Microbiome Association Studies [12.579558827555273]
  We introduce a novel meta-analysis framework to combine dependent tests under a general setting. We utilize it to synthesize various microbiome association tests that are calculated from the same dataset.
  arXiv  Detail & Related papers  (2024-04-14T20:33:39Z)
• Sample Complexity Characterization for Linear Contextual MDPs [67.79455646673762]
  Contextual decision processes (CMDPs) describe a class of reinforcement learning problems in which the transition kernels and reward functions can change over time with different MDPs indexed by a context variable. CMDPs serve as an important framework to model many real-world applications with time-varying environments. We study CMDPs under two linear function approximation models: Model I with context-varying representations and common linear weights for all contexts; and Model II with common representations for all contexts and context-varying linear weights.
  arXiv  Detail & Related papers  (2024-02-05T03:25:04Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• Statistical Comparisons of Classifiers by Generalized Stochastic Dominance [0.0]
  There is still no consensus on how to compare classifiers over multiple data sets with respect to several criteria. In this paper, we add a fresh view to the vivid debate by adopting recent developments in decision theory. We show that our framework ranks classifiers by a generalized concept of dominance, which powerfully circumvents the cumbersome, and often even self-contradictory, reliance on aggregates.
  arXiv  Detail & Related papers  (2022-09-05T09:28:15Z)
• Consensual Aggregation on Random Projected High-dimensional Features for Regression [0.0]
  We present a study of a kernel-based consensual aggregation on randomly projected high-dimensional features of predictions for regression. We numerically illustrate that the aggregation scheme upholds its performance on very large and highly correlated features. The efficiency of the proposed method is illustrated through several experiments evaluated on different types of synthetic and real datasets.
  arXiv  Detail & Related papers  (2022-04-06T06:35:47Z)
• A Unified Framework for Multi-distribution Density Ratio Estimation [101.67420298343512]
  Binary density ratio estimation (DRE) provides the foundation for many state-of-the-art machine learning algorithms. We develop a general framework from the perspective of Bregman minimization divergence. We show that our framework leads to methods that strictly generalize their counterparts in binary DRE.
  arXiv  Detail & Related papers  (2021-12-07T01:23:20Z)
• Optimal Ensemble Construction for Multi-Study Prediction with Applications to COVID-19 Excess Mortality Estimation [7.02598981483736]
  Multi-study ensembling uses a two-stage strategy which fits study-specific models and estimates ensemble weights separately. This approach ignores the ensemble properties at the model-fitting stage, potentially resulting in a loss of efficiency. We show that when little data is available for a country before the onset of the pandemic, leveraging data from other countries can substantially improve prediction accuracy.
  arXiv  Detail & Related papers  (2021-09-19T16:52:41Z)
• Trusted Multi-View Classification [76.73585034192894]
  We propose a novel multi-view classification method, termed trusted multi-view classification. It provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The proposed algorithm jointly utilizes multiple views to promote both classification reliability and robustness.
  arXiv  Detail & Related papers  (2021-02-03T13:30:26Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.