Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

• URL: http://arxiv.org/abs/2609.01397v1
• Date: Tue, 01 Sep 2026 15:22:01 GMT
• Title: Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity
• Abstract summary: The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs.<n>We propose a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model.<n>Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked.
• Score: 4.924900955306918
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs (predictive multiplicity). Existing work primarily focuses on multiplicity within individual models, but in more complex decision systems, the impact of the Rashomon effect is less well understood. In this work, we study multiplicity from the perspective of auditing incorrect ensemble predictions, where the decision to divert an instance for human review is based on a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model. With mild assumptions about stability and smoothness, we show that the consistency scores of finite ensembles converge to the corresponding consistency score of the expected model from the Rashomon set as the ensemble size and the number of samples used to measure local prediction variability increase. To demonstrate the efficacy of the proposed criterion, we evaluate the framework with respect to transformer models applied to natural language understanding tasks and parameter-efficient fine-tuning of large language models used for tabular data classification tasks. Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked compared with auditing a single model, while incurring only a moderate increase in the number of diversions. Moreover, the auditing behavior of the full Rashomon set can be closely approximated by finite ensembles of relatively modest size, with the risk approaching zero for some datasets. We further demonstrate that the proposed measure exhibits stronger agreement with established predictive multiplicity metrics than existing consistency measures, providing a more reliable way to capture multiplicity in the Rashomon set.

Related papers

• Scalable Model-Assisted Multi-Target Estimation in Large Image Collections [21.336432095392954]
  We study the problem of multi-target estimation, where many quantities must be estimated simultaneously.<n>Our framework effectively combines biased model predictions and limited human labels into rigorous scientific measurements.
  arXiv  Detail & Related papers  (2026-07-20T05:54:36Z)
• Mitigating the Multiplicity Burden: The Role of Calibration in Reducing Predictive Multiplicity of Classifiers [0.0]
  This paper examines the interplay between classification calibration and predictive multiplicity.<n>Minority class observations bear a disproportionate multiplicity burden.<n>Applying post-hoc calibration methods is associated with lower obscurity across the Rashomon set.
  arXiv  Detail & Related papers  (2026-03-12T09:54:07Z)
• On Arbitrary Predictions from Equally Valid Models [49.56463611078044]
  Model multiplicity refers to multiple machine learning models that admit conflicting predictions for the same patient.<n>We show that even small ensembles can mitigate/eliminate predictive multiplicity in practice.
  arXiv  Detail & Related papers  (2025-07-25T16:15:59Z)
• Regularized Neural Ensemblers [55.15643209328513]
  In this study, we explore employing regularized neural networks as ensemble methods.<n>Motivated by the risk of learning low-diversity ensembles, we propose regularizing the ensembling model by randomly dropping base model predictions.<n>We demonstrate this approach provides lower bounds for the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs [10.494477811252034]
  Fine-tuning multiplicity can arise in Tabular LLMs on classification tasks.<n>Our work formalizes this unique challenge of fine-tuning multiplicity in Tabular LLMs.<n>We propose a novel measure to quantify consistency of individual predictions without expensive model retraining.
  arXiv  Detail & Related papers  (2024-07-04T22:22:09Z)
• Multi-View Conformal Learning for Heterogeneous Sensor Fusion [0.12086712057375555]
  We build and test multi-view and single-view conformal models for heterogeneous sensor fusion. Our models provide theoretical marginal confidence guarantees since they are based on the conformal prediction framework. Our results also showed that multi-view models generate prediction sets with less uncertainty compared to single-view models.
  arXiv  Detail & Related papers  (2024-02-19T17:30:09Z)
• Predictive Churn with the Set of Good Models [61.00058053669447]
  This paper explores connections between two seemingly unrelated concepts of predictive inconsistency.<n>The first, known as predictive multiplicity, occurs when models that perform similarly produce conflicting predictions for individual samples.<n>The second concept, predictive churn, examines the differences in individual predictions before and after model updates.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation [15.556756363296543]
  Predictive multiplicity refers to the phenomenon in which classification tasks admit multiple competing models that achieve almost-equally-optimal performance. We propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation.
  arXiv  Detail & Related papers  (2024-02-01T16:25:00Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Trusted Multi-View Classification [76.73585034192894]
  We propose a novel multi-view classification method, termed trusted multi-view classification. It provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The proposed algorithm jointly utilizes multiple views to promote both classification reliability and robustness.
  arXiv  Detail & Related papers  (2021-02-03T13:30:26Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
  This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets. Part of the challenge of learning robust models lies in the influence of unobserved confounders. We demonstrate the empirical performance of our approach on healthcare data from different modalities.
  arXiv  Detail & Related papers  (2020-07-21T08:18:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.