Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation

• URL: http://arxiv.org/abs/2402.00728v1
• Date: Thu, 1 Feb 2024 16:25:00 GMT
• Title: Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation
• Authors: Hsiang Hsu, Guihong Li, Shaohan Hu, Chun-Fu (Richard) Chen
• Abstract summary: Predictive multiplicity refers to the phenomenon in which classification tasks admit multiple competing models that achieve almost-equally-optimal performance. We propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation.
• Score: 15.556756363296543
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-equally-optimal performance, yet generate conflicting outputs for individual samples. This presents significant concerns, as it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness in practical applications. Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need to explore all such almost-equally-optimal models, known as the Rashomon set, in potentially huge hypothesis spaces. To address this challenge, we propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We provide rigorous theoretical derivations to connect the dropout parameters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation. Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation, with runtime speedup up to $20\times \sim 5000\times$. With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.

Related papers

• Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection [63.96018203905272]
  We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets.
  arXiv  Detail & Related papers  (2024-09-23T21:27:26Z)
• Batch Ensemble for Variance Dependent Regret in Stochastic Bandits [41.95653110232677]
  Efficiently trading off exploration and exploitation is one of the key challenges in online Reinforcement Learning (RL) Inspired by practical ensemble methods, in this work we propose a simple and novel batch ensemble scheme that achieves near-optimal regret for Multi-Armed Bandits (MAB) Our algorithm has just a single parameter namely the number of batches, and its value does not depend on distributional properties such as the scale and variance of the losses.
  arXiv  Detail & Related papers  (2024-09-13T06:40:56Z)
• Efficient Exploration of the Rashomon Set of Rule Set Models [18.187800166484507]
  An emerging paradigm in interpretable machine learning aims at exploring the Rashomon set of all models exhibiting near-optimal performance. Existing work on Rashomon-set exploration focuses on exhaustive search of the Rashomon set for particular classes of models. We propose, for the first time, efficient methods to explore the Rashomon set of rule set models with or without exhaustive search.
  arXiv  Detail & Related papers  (2024-06-05T08:37:41Z)
• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
  We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model. We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions. Our findings have important practical implications for expert selection.
  arXiv  Detail & Related papers  (2024-02-05T12:31:18Z)
• Maximum Likelihood Estimation of Flexible Survival Densities with Importance Sampling [3.4082496470541312]
  Survival analysis is a widely-used technique for analyzing time-to-event data in the presence of censoring. We propose a survival analysis approach which eliminates the need to tune hyper parameters. We show that the proposed approach matches or outperforms baselines on several real-world datasets.
  arXiv  Detail & Related papers  (2023-11-03T01:46:48Z)
• Arbitrariness Lies Beyond the Fairness-Accuracy Frontier [3.383670923637875]
  We show that state-of-the-art fairness interventions can mask high predictive multiplicity behind favorable group fairness and accuracy metrics. We propose an ensemble algorithm applicable to any fairness intervention that provably ensures more consistent predictions.
  arXiv  Detail & Related papers  (2023-06-15T18:15:46Z)
• Rashomon Capacity: A Metric for Predictive Multiplicity in Probabilistic Classification [4.492630871726495]
  Predictive multiplicity occurs when classification models assign conflicting predictions to individual samples. We introduce a new measure of predictive multiplicity in probabilistic classification called Rashomon Capacity. We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders.
  arXiv  Detail & Related papers  (2022-06-02T20:44:19Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.