Budgeted Multiple-Expert Deferral

• URL: http://arxiv.org/abs/2510.26706v1
• Date: Thu, 30 Oct 2025 17:08:52 GMT
• Title: Budgeted Multiple-Expert Deferral
• Authors: Giulia DeSalvo, Clara Mohri, Mehryar Mohri, Yutao Zhong,
• Abstract summary: Training procedures for deferral algorithms typically require querying all experts for every training instance.<n>We introduce the budgeted deferral framework, which aims to train effective deferral algorithms while minimizing expert query costs during training.<n>We propose new algorithms for both two-stage and single-stage multiple-expert deferral settings that selectively query only a subset of experts per training example.
• Score: 38.13580998392063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to defer uncertain predictions to costly experts offers a powerful strategy for improving the accuracy and efficiency of machine learning systems. However, standard training procedures for deferral algorithms typically require querying all experts for every training instance, an approach that becomes prohibitively expensive when expert queries incur significant computational or resource costs. This undermines the core goal of deferral: to limit unnecessary expert usage. To overcome this challenge, we introduce the budgeted deferral framework, which aims to train effective deferral algorithms while minimizing expert query costs during training. We propose new algorithms for both two-stage and single-stage multiple-expert deferral settings that selectively query only a subset of experts per training example. While inspired by active learning, our setting is fundamentally different: labels are already known, and the core challenge is to decide which experts to query in order to balance cost and predictive performance. We establish theoretical guarantees for both of our algorithms, including generalization bounds and label complexity analyses. Empirical results across several domains show that our algorithms substantially reduce training costs without sacrificing prediction accuracy, demonstrating the practical value of our budget-aware deferral algorithms.

Related papers

• Theory and Algorithms for Learning with Multi-Class Abstention and Multi-Expert Deferral [20.76255397215973]
  Large language models (LLMs) have achieved remarkable performance but face critical challenges: hallucinations and high inference costs.<n>Leveraging multiple experts offers a solution: deferring uncertain inputs to more capable experts improves reliability.<n>This thesis presents a comprehensive study of this problem and the related problem of learning with abstention, supported by strong consistency guarantees.
  arXiv  Detail & Related papers  (2025-12-28T11:33:39Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction [3.746889836344766]
  We propose a training-free, model- and expert-agnostic framework for expert deferral based on conformal prediction.<n>Our method consistently outperforms both the standalone model and the strongest expert.
  arXiv  Detail & Related papers  (2025-09-16T02:01:21Z)
• Stepsize anything: A unified learning rate schedule for budgeted-iteration training [43.52874155421866]
  Budgeted-iteration training aims to achieve optimal learning within predetermined budgets.<n>While learning rate schedules govern the performance of different networks and tasks, their design remains largely lacking theoretical foundations.<n>We propose the Unified Budget-Aware (UBA) schedule, a theoretically grounded learning rate schedule that consistently outperforms commonly-used schedules.
  arXiv  Detail & Related papers  (2025-05-30T10:38:03Z)
• Why Ask One When You Can Ask $k$? Learning-to-Defer to the Top-$k$ Experts [6.792743621449621]
  We introduce the first framework for Top-$k$ Learning-to-Defer.<n>It allocates queries to the $k$ most cost-effective entities.<n>We also propose Top-$k(x)$ Learning-to-Defer, an adaptive variant that learns the optimal number of experts per query.
  arXiv  Detail & Related papers  (2025-04-17T14:50:40Z)
• Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guarantees [3.4289478404209826]
  Large Language Models excel in generative tasks but exhibit inefficiencies in structured text selection.<n>We propose a Learning-to-Defer framework that allocates queries to specialized experts, ensuring high-confidence predictions.
  arXiv  Detail & Related papers  (2024-10-21T08:21:00Z)
• Contractual Reinforcement Learning: Pulling Arms with Invisible Hands [68.77645200579181]
  We propose a theoretical framework for aligning economic interests of different stakeholders in the online learning problems through contract design. For the planning problem, we design an efficient dynamic programming algorithm to determine the optimal contracts against the far-sighted agent. For the learning problem, we introduce a generic design of no-regret learning algorithms to untangle the challenges from robust design of contracts to the balance of exploration and exploitation.
  arXiv  Detail & Related papers  (2024-07-01T16:53:00Z)
• An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models [55.01592097059969]
  Supervised finetuning on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool. We propose using experimental design to circumvent the computational bottlenecks of active learning.
  arXiv  Detail & Related papers  (2024-01-12T16:56:54Z)
• Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
  In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements. We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design. We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
  arXiv  Detail & Related papers  (2022-02-21T13:18:11Z)
• In Defense of the Unitary Scalarization for Deep Multi-Task Learning [121.76421174107463]
  We present a theoretical analysis suggesting that many specialized multi-tasks can be interpreted as forms of regularization. We show that, when coupled with standard regularization and stabilization techniques, unitary scalarization matches or improves upon the performance of complex multitasks.
  arXiv  Detail & Related papers  (2022-01-11T18:44:17Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)
• Consistent Estimators for Learning to Defer to an Expert [5.076419064097734]
  We show how to learn predictors that can either predict or choose to defer the decision to a downstream expert. We show the effectiveness of our approach on a variety of experimental tasks.
  arXiv  Detail & Related papers  (2020-06-02T18:21:38Z)
• Active Imitation Learning with Noisy Guidance [6.832341432995627]
  Imitation learning algorithms provide state-of-the-art results on many structured prediction tasks. Such algorithms assume training-time access to an expert that can provide the optimal action at any queried state. We consider an active learning setting in which the learning algorithm has additional access to a much cheaper noisy that provides noisy guidance.
  arXiv  Detail & Related papers  (2020-05-26T15:35:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.