One-Shot Federated Conformal Prediction
- URL: http://arxiv.org/abs/2302.06322v2
- Date: Mon, 31 Jul 2023 14:45:28 GMT
- Title: One-Shot Federated Conformal Prediction
- Authors: Pierre Humbert (LMO, CELESTE), Batiste Le Bars (MAGNET, CRIStAL),
Aur\'elien Bellet (MAGNET, CRIStAL), Sylvain Arlot (LMO, CELESTE)
- Abstract summary: We introduce a conformal prediction method to construct prediction sets in a oneshot federated learning setting.
We prove that for any distribution, it is possible to output prediction sets with desired coverage in only one round of communication.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we introduce a conformal prediction method to construct
prediction sets in a oneshot federated learning setting. More specifically, we
define a quantile-of-quantiles estimator and prove that for any distribution,
it is possible to output prediction sets with desired coverage in only one
round of communication. To mitigate privacy issues, we also describe a locally
differentially private version of our estimator. Finally, over a wide range of
experiments, we show that our method returns prediction sets with coverage and
length very similar to those obtained in a centralized setting. Overall, these
results demonstrate that our method is particularly well-suited to perform
conformal predictions in a one-shot federated learning setting.
Related papers
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.
Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.
We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
arXiv Detail & Related papers (2024-10-13T15:37:11Z) - Conformalized Interval Arithmetic with Symmetric Calibration [9.559062601251464]
We develop conformal prediction intervals for single target to the prediction interval for sum of multiple targets.
We show that our method outperforms existing conformalized approaches as well as non-conformal approaches.
arXiv Detail & Related papers (2024-08-20T15:27:18Z) - Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution.
Our method consistently outperforms existing approaches in terms of conditional coverage.
arXiv Detail & Related papers (2024-07-01T20:44:48Z) - Robust Conformal Prediction Using Privileged Information [17.886554223172517]
We develop a method to generate prediction sets with a guaranteed coverage rate that is robust to corruptions in the training data.
Our approach builds on conformal prediction, a powerful framework to construct prediction sets that are valid under the i.i.d assumption.
arXiv Detail & Related papers (2024-06-08T08:56:47Z) - Marginal and training-conditional guarantees in one-shot federated conformal prediction [17.197488145781858]
We study conformal prediction in the one-shot federated learning setting.
The main goal is to compute marginally and training-conditionally valid prediction sets, at the server-level, in only one round of communication between the agents and the server.
arXiv Detail & Related papers (2024-05-21T08:08:00Z) - Conformal Language Modeling [61.94417935386489]
We propose a novel approach to conformal prediction for generative language models (LMs)
Standard conformal prediction produces prediction sets with rigorous, statistical guarantees.
We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
arXiv Detail & Related papers (2023-06-16T21:55:08Z) - Post-selection Inference for Conformal Prediction: Trading off Coverage
for Precision [0.0]
Traditionally, conformal prediction inference requires a data-independent specification of miscoverage level.
We develop simultaneous conformal inference to account for data-dependent miscoverage levels.
arXiv Detail & Related papers (2023-04-12T20:56:43Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Test-time Collective Prediction [73.74982509510961]
Multiple parties in machine learning want to jointly make predictions on future test points.
Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters.
We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
arXiv Detail & Related papers (2021-06-22T18:29:58Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.