Predictive Multiplicity in Probabilistic Classification
- URL: http://arxiv.org/abs/2206.01131v3
- Date: Fri, 23 Jun 2023 19:12:05 GMT
- Title: Predictive Multiplicity in Probabilistic Classification
- Authors: Jamelle Watson-Daniels, David C. Parkes and Berk Ustun
- Abstract summary: We present a framework for measuring predictive multiplicity in probabilistic classification.
We demonstrate the incidence and prevalence of predictive multiplicity in real-world tasks.
Our results emphasize the need to report predictive multiplicity more widely.
- Score: 25.111463701666864
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Machine learning models are often used to inform real world risk assessment
tasks: predicting consumer default risk, predicting whether a person suffers
from a serious illness, or predicting a person's risk to appear in court. Given
multiple models that perform almost equally well for a prediction task, to what
extent do predictions vary across these models? If predictions are relatively
consistent for similar models, then the standard approach of choosing the model
that optimizes a penalized loss suffices. But what if predictions vary
significantly for similar models? In machine learning, this is referred to as
predictive multiplicity i.e. the prevalence of conflicting predictions assigned
by near-optimal competing models. In this paper, we present a framework for
measuring predictive multiplicity in probabilistic classification (predicting
the probability of a positive outcome). We introduce measures that capture the
variation in risk estimates over the set of competing models, and develop
optimization-based methods to compute these measures efficiently and reliably
for convex empirical risk minimization problems. We demonstrate the incidence
and prevalence of predictive multiplicity in real-world tasks. Further, we
provide insight into how predictive multiplicity arises by analyzing the
relationship between predictive multiplicity and data set characteristics
(outliers, separability, and majority-minority structure). Our results
emphasize the need to report predictive multiplicity more widely.
Related papers
- Reconciling Model Multiplicity for Downstream Decision Making [24.335927243672952]
We show that even when the two predictive models approximately agree on their individual predictions almost everywhere, it is still possible for their induced best-response actions to differ on a substantial portion of the population.
We propose a framework that calibrates the predictive models with regard to both the downstream decision-making problem and the individual probability prediction.
arXiv Detail & Related papers (2024-05-30T03:36:46Z) - An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification [0.0]
This paper examines the impact of balancing methods on predictive multiplicity using the Rashomon effect.
It is crucial because the blind model selection in data-centric AI is risky from a set of approximately equally accurate models.
arXiv Detail & Related papers (2024-03-22T13:08:22Z) - Predictive Churn with the Set of Good Models [64.05949860750235]
We study the effect of conflicting predictions over the set of near-optimal machine learning models.
We present theoretical results on the expected churn between models within the Rashomon set.
We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
arXiv Detail & Related papers (2024-02-12T16:15:25Z) - Structured Radial Basis Function Network: Modelling Diversity for
Multiple Hypotheses Prediction [51.82628081279621]
Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions.
A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems.
It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
arXiv Detail & Related papers (2023-09-02T01:27:53Z) - Post-Selection Confidence Bounds for Prediction Performance [2.28438857884398]
In machine learning, the selection of a promising model from a potentially large number of competing models and the assessment of its generalization performance are critical tasks.
We propose an algorithm how to compute valid lower confidence bounds for multiple models that have been selected based on their prediction performances in the evaluation set.
arXiv Detail & Related papers (2022-10-24T13:28:43Z) - Rashomon Capacity: A Metric for Predictive Multiplicity in Probabilistic
Classification [4.492630871726495]
Predictive multiplicity occurs when classification models assign conflicting predictions to individual samples.
We introduce a new measure of predictive multiplicity in probabilistic classification called Rashomon Capacity.
We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders.
arXiv Detail & Related papers (2022-06-02T20:44:19Z) - Mitigating multiple descents: A model-agnostic framework for risk
monotonization [84.6382406922369]
We develop a general framework for risk monotonization based on cross-validation.
We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
arXiv Detail & Related papers (2022-05-25T17:41:40Z) - Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic
Regression [51.770998056563094]
Probabilistic Gradient Boosting Machines (PGBM) is a method to create probabilistic predictions with a single ensemble of decision trees.
We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods.
arXiv Detail & Related papers (2021-06-03T08:32:13Z) - Characterizing Fairness Over the Set of Good Models Under Selective
Labels [69.64662540443162]
We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance.
We provide tractable algorithms to compute the range of attainable group-level predictive disparities.
We extend our framework to address the empirically relevant challenge of selectively labelled data.
arXiv Detail & Related papers (2021-01-02T02:11:37Z) - Ambiguity in Sequential Data: Predicting Uncertain Futures with
Recurrent Models [110.82452096672182]
We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data.
We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
arXiv Detail & Related papers (2020-03-10T09:15:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.