Expert-Agnostic Learning to Defer

• URL: http://arxiv.org/abs/2502.10533v1
• Date: Fri, 14 Feb 2025 19:59:25 GMT
• Title: Expert-Agnostic Learning to Defer
• Authors: Joshua Strong, Pramit Saha, Yasin Ibrahim, Cheng Ouyang, Alison Noble,
• Abstract summary: We introduce EA-L2D: Expert-Agnostic Learning to Defer, a novel L2D framework that leverages a Bayesian approach to model expert behaviour. We observe performance gains over the next state-of-the-art of 1-16% for seen experts and 4-28% for unseen experts in settings with high expert diversity.
• Score: 4.171294900540735
• License:
• Abstract: Learning to Defer (L2D) learns autonomous systems to independently manage straightforward cases, while deferring uncertain cases to human experts. Recent advancements in this field have introduced features enabling flexibility to unseen experts at test-time, but we find these approaches have significant limitations. To address these, we introduce EA-L2D: Expert-Agnostic Learning to Defer, a novel L2D framework that leverages a Bayesian approach to model expert behaviour in an expert-agnostic manner, facilitating optimal deferral decisions. EA-L2D offers several critical improvements over prior methods, including the ability to incorporate prior knowledge about experts, a reduced reliance on expert-annotated data, and robust performance when deferring to experts with expertise not seen during training. Evaluating on CIFAR-10, HAM10000, German Traffic Lights, Breast Ultrasound, Axial Organ Slices, and Blood Cell MNIST, we observe performance gains over the next state-of-the-art of 1-16\% for seen experts and 4-28\% for unseen experts in settings with high expert diversity.

Related papers

• Learning to Defer for Causal Discovery with Imperfect Experts [59.071731337922664]
  We propose L2D-CD, a method for gauging the correctness of expert recommendations and optimally combining them with data-driven causal discovery results. We evaluate L2D-CD on the canonical T"ubingen pairs dataset and demonstrate its superior performance compared to both the causal discovery method and the expert used in isolation.
  arXiv  Detail & Related papers  (2025-02-18T18:55:53Z)
• Utilising Deep Learning to Elicit Expert Uncertainty [2.9686400658670578]
  We show how analysts can adopt a deep learning approach to utilize the method proposed in [14 ] with the actual information experts use. We provide an overview of deep learning models that can effectively model expert decision-making to elicit distributions that capture expert uncertainty.
  arXiv  Detail & Related papers  (2025-01-21T01:36:12Z)
• Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs [64.9693406713216]
  Internal mechanisms that contribute to the effectiveness of RAG systems remain underexplored. Our experiments reveal that several core groups of experts are primarily responsible for RAG-related behaviors. We propose several strategies to enhance RAG's efficiency and effectiveness through expert activation.
  arXiv  Detail & Related papers  (2024-10-20T16:08:54Z)
• Learning to Defer to a Population: A Meta-Learning Approach [4.64209268422757]
  We formulate an L2D system that can cope with never-before-seen experts at test-time. We employ an attention mechanism that is able to look for points in the context set that are similar to a given test point. In the experiments, we validate our methods on image recognition, traffic sign detection, and skin lesion diagnosis benchmarks.
  arXiv  Detail & Related papers  (2024-03-05T06:10:28Z)
• Defining Expertise: Applications to Treatment Effect Estimation [58.7977683502207]
  We argue that expertise - particularly the type of expertise the decision-makers of a domain are likely to have - can be informative in designing and selecting methods for treatment effect estimation. We define two types of expertise, predictive and prognostic, and demonstrate empirically that: (i) the prominent type of expertise in a domain significantly influences the performance of different methods in treatment effect estimation, and (ii) it is possible to predict the type of expertise present in a dataset.
  arXiv  Detail & Related papers  (2024-03-01T17:30:49Z)
• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• Learning to Defer with Limited Expert Predictions [0.0]
  We propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions.
  arXiv  Detail & Related papers  (2023-04-14T09:22:34Z)
• (Re)Defining Expertise in Machine Learning Development [3.096615629099617]
  We conduct a systematic literature review of machine learning research to understand 1) the bases on which expertise is defined and recognized and 2) the roles experts play in ML development. Our goal is to produce a high-level taxonomy to highlight limits and opportunities in how experts are identified and engaged in ML research.
  arXiv  Detail & Related papers  (2023-02-08T21:10:20Z)
• Experts in the Loop: Conditional Variable Selection for Accelerating Post-Silicon Analysis Based on Deep Learning [6.6357750579293935]
  Post-silicon validation is one of the most critical processes in semiconductor manufacturing. This work aims to design a novel conditional variable selection approach while keeping experts in the loop.
  arXiv  Detail & Related papers  (2022-09-30T06:12:12Z)
• Decision Rule Elicitation for Domain Adaptation [93.02675868486932]
  Human-in-the-loop machine learning is widely used in artificial intelligence (AI) to elicit labels from experts. In this work, we allow experts to additionally produce decision rules describing their decision-making. We show that decision rule elicitation improves domain adaptation of the algorithm and helps to propagate expert's knowledge to the AI model.
  arXiv  Detail & Related papers  (2021-02-23T08:07:22Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.