Conformal Set-based Human-AI Complementarity with Multiple Experts

• URL: http://arxiv.org/abs/2508.06997v1
• Date: Sat, 09 Aug 2025 14:17:51 GMT
• Title: Conformal Set-based Human-AI Complementarity with Multiple Experts
• Authors: Helbert Paat, Guohao Shen,
• Abstract summary: This study focuses on the selection of instance-specific experts from a pool of multiple human experts.<n>We introduce a greedy algorithm that utilizes conformal sets to identify the subset of expert predictions that will be used in classifying an instance.
• Score: 1.1510009152620668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decision support systems are designed to assist human experts in classification tasks by providing conformal prediction sets derived from a pre-trained model. This human-AI collaboration has demonstrated enhanced classification performance compared to using either the model or the expert independently. In this study, we focus on the selection of instance-specific experts from a pool of multiple human experts, contrasting it with existing research that typically focuses on single-expert scenarios. We characterize the conditions under which multiple experts can benefit from the conformal sets. With the insight that only certain experts may be relevant for each instance, we explore the problem of subset selection and introduce a greedy algorithm that utilizes conformal sets to identify the subset of expert predictions that will be used in classifying an instance. This approach is shown to yield better performance compared to naive methods for human subset selection. Based on real expert predictions from the CIFAR-10H and ImageNet-16H datasets, our simulation study indicates that our proposed greedy algorithm achieves near-optimal subsets, resulting in improved classification performance among multiple experts.

Related papers

• Imitation Learning for Combinatorial Optimisation under Uncertainty [1.0781866671930855]
  This paper introduces a systematic taxonomy of experts for IL optimisation under uncertainty.<n>Experts are classified along three dimensions: (i) their treatment of uncertainty, including myopic, deterministic, full-information, two-stage, and multi-stage formulations; (ii) their level of optimality, distinguishing task-optimal and approximate experts; and (iii) their interaction mode with the learner, ranging from one-shot supervision to iterative, interactive schemes.
  arXiv  Detail & Related papers  (2026-01-08T21:16:25Z)
• Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making [5.4873584968147275]
  We explore the use of expert-guided bandit learning, which we refer to as online mixture-of-experts (OMoE)<n>The first algorithm combines aggregate voting with UCB-driven successive elimination, efficiently pruning suboptimal exploration actions.<n>The second algorithm employs an online weighted-majority-voting mechanism, leveraging the respective voting power of each expert proportional to their predictive power.
  arXiv  Detail & Related papers  (2025-10-19T11:00:10Z)
• Bayesian Inference for Correlated Human Experts and Classifiers [15.98474253538141]
  We investigate the problem of querying experts for class label predictions using as few human queries as possible.<n>We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation.<n>We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H.
  arXiv  Detail & Related papers  (2025-06-05T23:39:41Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection [63.96018203905272]
  We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets.
  arXiv  Detail & Related papers  (2024-09-23T21:27:26Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• Bayesian Online Learning for Consensus Prediction [16.890828000688174]
  We propose a family of methods that dynamically estimate expert consensus from partial feedback. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H.
  arXiv  Detail & Related papers  (2023-12-12T19:18:04Z)
• Diversified Dynamic Routing for Vision Tasks [36.199659460868496]
  We propose a novel architecture where each layer is composed of a set of experts. In our method, the model is explicitly trained to solve the challenge of finding relevant partitioning of the data. We conduct several experiments on semantic segmentation on Cityscapes and object detection and instance segmentation on MS-COCO.
  arXiv  Detail & Related papers  (2022-09-26T23:27:51Z)
• Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts [0.0]
  We propose an approach that trains a classification model to complement the capabilities of multiple human experts. We evaluate our proposed approach in experiments on public datasets with "synthetic" experts and a real-world medical dataset annotated by multiple radiologists.
  arXiv  Detail & Related papers  (2022-06-16T06:42:10Z)
• Doubting AI Predictions: Influence-Driven Second Opinion Recommendation [92.30805227803688]
  We propose a way to augment human-AI collaboration by building on a common organizational practice: identifying experts who are likely to provide complementary opinions. The proposed approach aims to leverage productive disagreement by identifying whether some experts are likely to disagree with an algorithmic assessment.
  arXiv  Detail & Related papers  (2022-04-29T20:35:07Z)
• On the Representation Collapse of Sparse Mixture of Experts [102.83396489230375]
  Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse.
  arXiv  Detail & Related papers  (2022-04-20T01:40:19Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.