Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts

• URL: http://arxiv.org/abs/2206.07948v1
• Date: Thu, 16 Jun 2022 06:42:10 GMT
• Title: Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts
• Authors: Patrick Hemmer and Sebastian Schellhammer and Michael V\"ossing and Johannes Jakubik and Gerhard Satzger
• Abstract summary: We propose an approach that trains a classification model to complement the capabilities of multiple human experts. We evaluate our proposed approach in experiments on public datasets with "synthetic" experts and a real-world medical dataset annotated by multiple radiologists.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Machine learning (ML) models are increasingly being used in application domains that often involve working together with human experts. In this context, it can be advantageous to defer certain instances to a single human expert when they are difficult to predict for the ML model. While previous work has focused on scenarios with one distinct human expert, in many real-world situations several human experts with varying capabilities may be available. In this work, we propose an approach that trains a classification model to complement the capabilities of multiple human experts. By jointly training the classifier together with an allocation system, the classifier learns to accurately predict those instances that are difficult for the human experts, while the allocation system learns to pass each instance to the most suitable team member -- either the classifier or one of the human experts. We evaluate our proposed approach in multiple experiments on public datasets with "synthetic" experts and a real-world medical dataset annotated by multiple radiologists. Our approach outperforms prior work and is more accurate than the best human expert or a classifier. Furthermore, it is flexibly adaptable to teams of varying sizes and different levels of expert diversity.

Related papers

• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts [0.7349727826230862]
  We propose a hybrid system that creates artificial experts that learn to classify data instances from unknown classes. Our approach outperforms traditional HITL systems for several benchmarks on image classification.
  arXiv  Detail & Related papers  (2023-07-06T14:06:23Z)
• Domain Generalization for Mammographic Image Analysis with Contrastive Learning [62.25104935889111]
  The training of an efficacious deep learning model requires large data with diverse styles and qualities. A novel contrastive learning is developed to equip the deep learning models with better style generalization capability. The proposed method has been evaluated extensively and rigorously with mammograms from various vendor style domains and several public datasets.
  arXiv  Detail & Related papers  (2023-04-20T11:40:21Z)
• Learning to Defer with Limited Expert Predictions [0.0]
  We propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions.
  arXiv  Detail & Related papers  (2023-04-14T09:22:34Z)
• BO-Muse: A human expert and AI teaming framework for accelerated experimental design [58.61002520273518]
  Our algorithm lets the human expert take the lead in the experimental process. We show that our algorithm converges sub-linearly, at a rate faster than the AI or human alone.
  arXiv  Detail & Related papers  (2023-03-03T02:56:05Z)
• JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions [75.42526766746515]
  We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs. Our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models.
  arXiv  Detail & Related papers  (2022-10-18T19:20:53Z)
• Diversified Dynamic Routing for Vision Tasks [36.199659460868496]
  We propose a novel architecture where each layer is composed of a set of experts. In our method, the model is explicitly trained to solve the challenge of finding relevant partitioning of the data. We conduct several experiments on semantic segmentation on Cityscapes and object detection and instance segmentation on MS-COCO.
  arXiv  Detail & Related papers  (2022-09-26T23:27:51Z)
• SuperCone: Modeling Heterogeneous Experts with Concept Meta-learning for Unified Predictive Segments System [8.917697023052257]
  We present SuperCone, our unified predicative segments system. It builds on top of a flat concept representation that summarizes each user's heterogeneous digital footprints. It can outperform state-of-the-art recommendation and ranking algorithms on a wide range of predicative segment tasks.
  arXiv  Detail & Related papers  (2022-03-09T04:11:39Z)
• Towards Unbiased and Accurate Deferral to Multiple Experts [19.24068936057053]
  We propose a framework that simultaneously learns a classifier and a deferral system, with the deferral system choosing to defer to one or more human experts. We test our framework on a synthetic dataset and a content moderation dataset with biased synthetic experts, and show that it significantly improves the accuracy and fairness of the final predictions.
  arXiv  Detail & Related papers  (2021-02-25T17:08:39Z)
• Multiple Expert Brainstorming for Domain Adaptive Person Re-identification [140.3998019639158]
  We propose a multiple expert brainstorming network (MEB-Net) for domain adaptive person re-ID. MEB-Net adopts a mutual learning strategy, where multiple networks with different architectures are pre-trained within a source domain. Experiments on large-scale datasets demonstrate the superior performance of MEB-Net over the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-07-03T08:16:19Z)
• Learning to Complement Humans [67.38348247794949]
  A rising vision for AI in the open world centers on the development of systems that can complement humans for perceptual, diagnostic, and reasoning tasks. We demonstrate how an end-to-end learning strategy can be harnessed to optimize the combined performance of human-machine teams.
  arXiv  Detail & Related papers  (2020-05-01T20:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.