Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints

• URL: http://arxiv.org/abs/2403.06906v3
• Date: Mon, 19 Aug 2024 18:18:30 GMT
• Title: Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints
• Authors: Jean V. Alves, Diogo Leitão, Sérgio Jesus, Marco O. P. Sampaio, Javier Liébana, Pedro Saleiro, Mário A. T. Figueiredo, Pedro Bizarro,
• Abstract summary: Learning to defer aims to improve human-AI collaboration systems by learning how to defer decisions to humans when they are more likely to be correct than an ML classifier. Existing research in L2D overlooks key real-world aspects that impede its practical adoption. DeCCaF is a novel L2D approach, employing supervised learning to model the probability of human error under less restrictive data requirements.
• Score: 10.917274244918985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to defer (L2D) aims to improve human-AI collaboration systems by learning how to defer decisions to humans when they are more likely to be correct than an ML classifier. Existing research in L2D overlooks key real-world aspects that impede its practical adoption, namely: i) neglecting cost-sensitive scenarios, where type I and type II errors have different costs; ii) requiring concurrent human predictions for every instance of the training dataset; and iii) not dealing with human work-capacity constraints. To address these issues, we propose the \textit{deferral under cost and capacity constraints framework} (DeCCaF). DeCCaF is a novel L2D approach, employing supervised learning to model the probability of human error under less restrictive data requirements (only one expert prediction per instance) and using constraint programming to globally minimize the error cost, subject to workload limitations. We test DeCCaF in a series of cost-sensitive fraud detection scenarios with different teams of 9 synthetic fraud analysts, with individual work-capacity constraints. The results demonstrate that our approach performs significantly better than the baselines in a wide array of scenarios, achieving an average $8.4\%$ reduction in the misclassification cost. The code used for the experiments is available at https://github.com/feedzai/deccaf

Related papers

• FiFAR: A Fraud Detection Dataset for Learning to Defer [9.187694794359498]
  We introduce the Financial Fraud Alert Review dataset (FiFAR), a synthetic bank account fraud detection dataset. FiFAR contains the predictions of a team of 50 highly complex and varied synthetic fraud analysts, with varied bias and feature dependence. We use our dataset to develop a capacity-aware L2D method and rejection learning approach under realistic data availability conditions.
  arXiv  Detail & Related papers  (2023-12-20T17:36:36Z)
• Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking [3.9412826185755017]
  Few-shot and large language models (LLMs) can perform effectively with just 1-5 examples per class. Our work addresses the performance-cost trade-offs of these methods over the Banking77 financial intent detection dataset. To inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis.
  arXiv  Detail & Related papers  (2023-11-10T15:10:36Z)
• Cost-Effective Retraining of Machine Learning Models [2.9461360639852914]
  It is important to retrain a machine learning (ML) model in order to maintain its performance as the data changes over time. This creates a trade-off between retraining too frequently, which leads to unnecessary computing costs, and not retraining often enough. We propose ML systems that make automated and cost-effective decisions about when to retrain an ML model.
  arXiv  Detail & Related papers  (2023-10-06T13:02:29Z)
• Toward Theoretical Guidance for Two Common Questions in Practical Cross-Validation based Hyperparameter Selection [72.76113104079678]
  We show the first theoretical treatments of two common questions in cross-validation based hyperparameter selection. We show that these generalizations can, respectively, always perform at least as well as always performing retraining or never performing retraining.
  arXiv  Detail & Related papers  (2023-01-12T16:37:12Z)
• Optimizing Data Collection for Machine Learning [87.37252958806856]
  Modern deep learning systems require huge data sets to achieve impressive performance. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay. We propose a new paradigm for modeling the data collection as a formal optimal data collection problem.
  arXiv  Detail & Related papers  (2022-10-03T21:19:05Z)
• Quantization for decentralized learning under subspace constraints [61.59416703323886]
  We consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints. We propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates. The analysis shows that, under some general conditions on the quantization noise, the strategy is stable both in terms of mean-square error and average bit rate.
  arXiv  Detail & Related papers  (2022-09-16T09:38:38Z)
• Rethinking Cost-sensitive Classification in Deep Learning via Adversarial Data Augmentation [4.479834103607382]
  Cost-sensitive classification is critical in applications where misclassification errors widely vary in cost. This paper proposes a cost-sensitive adversarial data augmentation framework to make over- parameterized models cost-sensitive. Our method can effectively minimize the overall cost and reduce critical errors, while achieving comparable performance in terms of overall accuracy.
  arXiv  Detail & Related papers  (2022-08-24T19:00:30Z)
• Linear Speedup in Personalized Collaborative Learning [69.45124829480106]
  Personalization in federated learning can improve the accuracy of a model for a user by trading off the model's bias. We formalize the personalized collaborative learning problem as optimization of a user's objective. We explore conditions under which we can optimally trade-off their bias for a reduction in variance.
  arXiv  Detail & Related papers  (2021-11-10T22:12:52Z)
• Online Selective Classification with Limited Feedback [82.68009460301585]
  We study selective classification in the online learning model, wherein a predictor may abstain from classifying an instance. Two salient aspects of the setting we consider are that the data may be non-realisable, due to which abstention may be a valid long-term action. We construct simple versioning-based schemes for any $mu in (0,1],$ that make most $Tmu$ mistakes while incurring smash$tildeO(T1-mu)$ excess abstention against adaptive adversaries.
  arXiv  Detail & Related papers  (2021-10-27T08:00:53Z)
• Training Over-parameterized Models with Non-decomposable Objectives [46.62273918807789]
  We propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model.
  arXiv  Detail & Related papers  (2021-07-09T19:29:33Z)
• Online Apprenticeship Learning [58.45089581278177]
  In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms.
  arXiv  Detail & Related papers  (2021-02-13T12:57:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.