Consistent Estimators for Learning to Defer to an Expert

• URL: http://arxiv.org/abs/2006.01862v3
• Date: Mon, 25 Jan 2021 01:43:28 GMT
• Title: Consistent Estimators for Learning to Defer to an Expert
• Authors: Hussein Mozannar, David Sontag
• Abstract summary: We show how to learn predictors that can either predict or choose to defer the decision to a downstream expert. We show the effectiveness of our approach on a variety of experimental tasks.
• Score: 5.076419064097734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.

Related papers

• Neural Active Learning Beyond Bandits [69.99592173038903]
  We study both stream-based and pool-based active learning with neural network approximations. We propose two algorithms based on the newly designed exploitation and exploration neural networks for stream-based and pool-based active learning.
  arXiv  Detail & Related papers  (2024-04-18T21:52:14Z)
• Principled Approaches for Learning to Defer with Multiple Experts [30.389055604165222]
  We introduce a new family of surrogate losses specifically tailored for the multiple-expert setting. We prove that these surrogate losses benefit from strong $H$-consistency bounds.
  arXiv  Detail & Related papers  (2023-10-23T10:19:09Z)
• Causal Imitation Learning with Unobserved Confounders [82.22545916247269]
  We study imitation learning when sensory inputs of the learner and the expert differ. We show that imitation could still be feasible by exploiting quantitative knowledge of the expert trajectories.
  arXiv  Detail & Related papers  (2022-08-12T13:29:53Z)
• Sample Efficient Learning of Predictors that Complement Humans [5.830619388189559]
  We provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. We design active learning schemes that require minimal amount of data of human expert predictions.
  arXiv  Detail & Related papers  (2022-07-19T23:19:25Z)
• Knowledge-driven Active Learning [70.37119719069499]
  Active learning strategies aim at minimizing the amount of labelled data required to train a Deep Learning model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. Here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples.
  arXiv  Detail & Related papers  (2021-10-15T06:11:53Z)
• Improving Human Sequential Decision-Making with Reinforcement Learning [29.334511328067777]
  We design a novel machine learning algorithm that is capable of extracting "best practices" from trace data. Our algorithm selects the tip that best bridges the gap between the actions taken by human workers and those taken by the optimal policy. Experiments show that the tips generated by our algorithm can significantly improve human performance.
  arXiv  Detail & Related papers  (2021-08-19T02:57:58Z)
• Online Learning with Uncertain Feedback Graphs [12.805267089186533]
  The relationship among experts can be captured by a feedback graph, which can be used to assist the learner's decision making. In practice, the nominal feedback graph often entails uncertainties, which renders it impossible to reveal the actual relationship among experts. The present work studies various cases of potential uncertainties, and develops novel online learning algorithms to deal with them.
  arXiv  Detail & Related papers  (2021-06-15T21:21:30Z)
• Decision Rule Elicitation for Domain Adaptation [93.02675868486932]
  Human-in-the-loop machine learning is widely used in artificial intelligence (AI) to elicit labels from experts. In this work, we allow experts to additionally produce decision rules describing their decision-making. We show that decision rule elicitation improves domain adaptation of the algorithm and helps to propagate expert's knowledge to the AI model.
  arXiv  Detail & Related papers  (2021-02-23T08:07:22Z)
• Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms [91.3755431537592]
  We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression. We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
  arXiv  Detail & Related papers  (2021-01-26T17:11:40Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)
• Automatic Discovery of Interpretable Planning Strategies [9.410583483182657]
  We introduce AI-Interpret, a method for transforming idiosyncratic policies into simple and interpretable descriptions. We show that prividing the decision rules generated by AI-Interpret as flowcharts significantly improved people's planning strategies and decisions.
  arXiv  Detail & Related papers  (2020-05-24T12:24:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.