Marginal and training-conditional guarantees in one-shot federated conformal prediction

• URL: http://arxiv.org/abs/2405.12567v1
• Date: Tue, 21 May 2024 08:08:00 GMT
• Title: Marginal and training-conditional guarantees in one-shot federated conformal prediction
• Authors: Pierre Humbert, Batiste Le Bars, Aurélien Bellet, Sylvain Arlot,
• Abstract summary: We study conformal prediction in the one-shot federated learning setting. The main goal is to compute marginally and training-conditionally valid prediction sets, at the server-level, in only one round of communication between the agents and the server.
• Score: 17.197488145781858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study conformal prediction in the one-shot federated learning setting. The main goal is to compute marginally and training-conditionally valid prediction sets, at the server-level, in only one round of communication between the agents and the server. Using the quantile-of-quantiles family of estimators and split conformal prediction, we introduce a collection of computationally-efficient and distribution-free algorithms that satisfy the aforementioned requirements. Our approaches come from theoretical results related to order statistics and the analysis of the Beta-Beta distribution. We also prove upper bounds on the coverage of all proposed algorithms when the nonconformity scores are almost surely distinct. For algorithms with training-conditional guarantees, these bounds are of the same order of magnitude as those of the centralized case. Remarkably, this implies that the one-shot federated learning setting entails no significant loss compared to the centralized case. Our experiments confirm that our algorithms return prediction sets with coverage and length similar to those obtained in a centralized setting.

Related papers

• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• PAC Prediction Sets Under Label Shift [52.30074177997787]
  Prediction sets capture uncertainty by predicting sets of labels rather than individual labels. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. We evaluate our approach on five datasets.
  arXiv  Detail & Related papers  (2023-10-19T17:57:57Z)
• Conformal Prediction for Federated Uncertainty Quantification Under Label Shift [57.54977668978613]
  Federated Learning (FL) is a machine learning framework where many clients collaboratively train models. We develop a new conformal prediction method based on quantile regression and take into account privacy constraints.
  arXiv  Detail & Related papers  (2023-06-08T11:54:58Z)
• Post-selection Inference for Conformal Prediction: Trading off Coverage for Precision [0.0]
  Traditionally, conformal prediction inference requires a data-independent specification of miscoverage level. We develop simultaneous conformal inference to account for data-dependent miscoverage levels.
  arXiv  Detail & Related papers  (2023-04-12T20:56:43Z)
• One-Shot Federated Conformal Prediction [0.0]
  We introduce a conformal prediction method to construct prediction sets in a oneshot federated learning setting. We prove that for any distribution, it is possible to output prediction sets with desired coverage in only one round of communication.
  arXiv  Detail & Related papers  (2023-02-13T12:46:39Z)
• Conformal prediction set for time-series [16.38369532102931]
  Uncertainty quantification is essential to studying complex machine learning methods. We develop Ensemble Regularized Adaptive Prediction Set (ERAPS) to construct prediction sets for time-series. We show valid marginal and conditional coverage by ERAPS, which also tends to yield smaller prediction sets than competing methods.
  arXiv  Detail & Related papers  (2022-06-15T23:48:53Z)
• Practical Adversarial Multivalid Conformal Prediction [27.179891682629183]
  We give a generic conformal prediction method for sequential prediction. It achieves target empirical coverage guarantees against adversarially chosen data. It is computationally lightweight -- comparable to split conformal prediction.
  arXiv  Detail & Related papers  (2022-06-02T14:33:00Z)
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning [24.3370326359959]
  We propose to predict with a Gaussian mixture model posterior that consists of a weighted sum of Laplace approximations of independently trained deep neural networks. We theoretically validate that our approach mitigates overconfidence "far away" from the training data and empirically compare against state-of-the-art baselines on standard uncertainty quantification benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T15:52:48Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms [67.67377846416106]
  We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
  arXiv  Detail & Related papers  (2020-03-27T05:13:29Z)
• A General Method for Robust Learning from Batches [56.59844655107251]
  We consider a general framework of robust learning from batches, and determine the limits of both classification and distribution estimation over arbitrary, including continuous, domains. We derive the first robust computationally-efficient learning algorithms for piecewise-interval classification, and for piecewise-polynomial, monotone, log-concave, and gaussian-mixture distribution estimation.
  arXiv  Detail & Related papers  (2020-02-25T18:53:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.