Principled Bayesian Optimisation in Collaboration with Human Experts

• URL: http://arxiv.org/abs/2410.10452v1
• Date: Mon, 14 Oct 2024 12:46:02 GMT
• Title: Principled Bayesian Optimisation in Collaboration with Human Experts
• Authors: Wenjie Xu, Masaki Adachi, Colin N. Jones, Michael A. Osborne,
• Abstract summary: We consider a setup where experts provide advice through binary accept/reject recommendations (labels) Experts' labels are often costly, requiring efficient use of their efforts, and can at the same time be unreliable. We introduce the first principled approach that provides two key guarantees.
• Score: 23.988732776208053
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Bayesian optimisation for real-world problems is often performed interactively with human experts, and integrating their domain knowledge is key to accelerate the optimisation process. We consider a setup where experts provide advice on the next query point through binary accept/reject recommendations (labels). Experts' labels are often costly, requiring efficient use of their efforts, and can at the same time be unreliable, requiring careful adjustment of the degree to which any expert is trusted. We introduce the first principled approach that provides two key guarantees. (1) Handover guarantee: similar to a no-regret property, we establish a sublinear bound on the cumulative number of experts' binary labels. Initially, multiple labels per query are needed, but the number of expert labels required asymptotically converges to zero, saving both expert effort and computation time. (2) No-harm guarantee with data-driven trust level adjustment: our adaptive trust level ensures that the convergence rate will not be worse than the one without using advice, even if the advice from experts is adversarial. Unlike existing methods that employ a user-defined function that hand-tunes the trust level adjustment, our approach enables data-driven adjustments. Real-world applications empirically demonstrate that our method not only outperforms existing baselines, but also maintains robustness despite varying labelling accuracy, in tasks of battery design with human experts.

Related papers

• Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback [2.4352490146713364]
  We introduce TriTrust-PBRL, a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback.<n>TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically.
  arXiv  Detail & Related papers  (2026-01-26T18:21:48Z)
• Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems [54.916243942641444]
  Large language models (LLMs) are emerging as key enablers of automation in domains such as telecommunications.<n>We study an edge-cloud-expert cascaded LLM-based knowledge system that supports decision-making through a question-and-answer pipeline.
  arXiv  Detail & Related papers  (2025-12-23T03:10:09Z)
• Open-World Deepfake Attribution via Confidence-Aware Asymmetric Learning [78.92934995292113]
  We propose a Confidence-Aware Asymmetric Learning (CAL) framework, which balances confidence across known and novel forgery types.<n>CAL consistently outperforms previous methods, achieving new state-of-the-art performance on both known and novel forgery attribution.
  arXiv  Detail & Related papers  (2025-12-14T12:31:28Z)
• From Guess2Graph: When and How Can Unreliable Experts Safely Boost Causal Discovery in Finite Samples? [20.68174733590345]
  We propose the Guess2Graph framework, which uses expert guesses to guide the sequence of statistical tests rather than replacing them.<n>We develop two instantiations of G2G: PC-Guess, which augments the PC algorithm, and gPC-Guess, a learning-augmented variant designed to better leverage high-quality expert input.
  arXiv  Detail & Related papers  (2025-10-16T09:31:44Z)
• Optimizing Resources for On-the-Fly Label Estimation with Multiple Unknown Medical Experts [2.904892426557913]
  We propose an adaptive approach for real-time annotation that supports on-the-fly labeling of incoming data.<n>We evaluate our approach on three multi-annotator classification datasets across different modalities.
  arXiv  Detail & Related papers  (2025-10-04T21:41:26Z)
• No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction [3.746889836344766]
  We propose a training-free, model- and expert-agnostic framework for expert deferral based on conformal prediction.<n>Our method consistently outperforms both the standalone model and the strongest expert.
  arXiv  Detail & Related papers  (2025-09-16T02:01:21Z)
• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU) We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level. Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Active Ranking of Experts Based on their Performances in Many Tasks [72.96112117037465]
  We consider the problem of ranking n experts based on their performances on d tasks. We make a monotonicity assumption stating that for each pair of experts, one outperforms the other on all tasks.
  arXiv  Detail & Related papers  (2023-06-05T06:55:39Z)
• Experts in the Loop: Conditional Variable Selection for Accelerating Post-Silicon Analysis Based on Deep Learning [6.6357750579293935]
  Post-silicon validation is one of the most critical processes in semiconductor manufacturing. This work aims to design a novel conditional variable selection approach while keeping experts in the loop.
  arXiv  Detail & Related papers  (2022-09-30T06:12:12Z)
• Recommendation Systems with Distribution-Free Reliability Guarantees [83.80644194980042]
  We show how to return a set of items rigorously guaranteed to contain mostly good items. Our procedure endows any ranking model with rigorous finite-sample control of the false discovery rate. We evaluate our methods on the Yahoo! Learning to Rank and MSMarco datasets.
  arXiv  Detail & Related papers  (2022-07-04T17:49:25Z)
• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning [15.123493340717303]
  We propose a novel semi-supervised learning paradigm which allows partial-labeled or unlabeled clients to seek labeling assistance from data-related clients (helper agents) Experiments show that our proposed method can obtain superior performance and more stable convergence than other related works with partial labeled data.
  arXiv  Detail & Related papers  (2022-05-05T04:41:27Z)
• Confident in the Crowd: Bayesian Inference to Improve Data Labelling in Crowdsourcing [0.30458514384586394]
  We present new techniques to improve the quality of the labels while attempting to reduce the cost. This paper investigates the use of more sophisticated methods, such as Bayesian inference, to measure the performance of the labellers. Our methods outperform the standard voting methods in both cost and accuracy while maintaining higher reliability when there is disagreement within the crowd.
  arXiv  Detail & Related papers  (2021-05-28T17:09:45Z)
• RATT: Leveraging Unlabeled Data to Guarantee Generalization [96.08979093738024]
  We introduce a method that leverages unlabeled data to produce generalization bounds. We prove that our bound is valid for 0-1 empirical risk minimization. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable.
  arXiv  Detail & Related papers  (2021-05-01T17:05:29Z)
• Pointwise Binary Classification with Pairwise Confidence Comparisons [97.79518780631457]
  We propose pairwise comparison (Pcomp) classification, where we have only pairs of unlabeled data that we know one is more likely to be positive than the other. We link Pcomp classification to noisy-label learning to develop a progressive URE and improve it by imposing consistency regularization.
  arXiv  Detail & Related papers  (2020-10-05T09:23:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.