Bayesian Online Learning for Consensus Prediction
- URL: http://arxiv.org/abs/2312.07679v1
- Date: Tue, 12 Dec 2023 19:18:04 GMT
- Title: Bayesian Online Learning for Consensus Prediction
- Authors: Sam Showalter, Alex Boyd, Padhraic Smyth, Mark Steyvers
- Abstract summary: We propose a family of methods that dynamically estimate expert consensus from partial feedback.
We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H.
- Score: 16.890828000688174
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Given a pre-trained classifier and multiple human experts, we investigate the
task of online classification where model predictions are provided for free but
querying humans incurs a cost. In this practical but under-explored setting,
oracle ground truth is not available. Instead, the prediction target is defined
as the consensus vote of all experts. Given that querying full consensus can be
costly, we propose a general framework for online Bayesian consensus
estimation, leveraging properties of the multivariate hypergeometric
distribution. Based on this framework, we propose a family of methods that
dynamically estimate expert consensus from partial feedback by producing a
posterior over expert and model beliefs. Analyzing this posterior induces an
interpretable trade-off between querying cost and classification performance.
We demonstrate the efficacy of our framework against a variety of baselines on
CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
Related papers
- PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines [86.36060279469304]
We introduce PredBench, a benchmark tailored for the holistic evaluation of prediction-temporal networks.
This benchmark integrates 12 widely adopted methods with diverse datasets across multiple application domains.
Its multi-dimensional evaluation framework broadens the analysis with a comprehensive set of metrics.
arXiv Detail & Related papers (2024-07-11T11:51:36Z) - Rejection via Learning Density Ratios [50.91522897152437]
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.
We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
Our framework is tested empirically over clean and noisy datasets.
arXiv Detail & Related papers (2024-05-29T01:32:17Z) - Causal Unsupervised Semantic Segmentation [60.178274138753174]
Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations.
We propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference.
arXiv Detail & Related papers (2023-10-11T10:54:44Z) - Prototype-based Aleatoric Uncertainty Quantification for Cross-modal
Retrieval [139.21955930418815]
Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space.
However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts.
We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
arXiv Detail & Related papers (2023-09-29T09:41:19Z) - A Bayesian Approach to Robust Inverse Reinforcement Learning [54.24816623644148]
We consider a Bayesian approach to offline model-based inverse reinforcement learning (IRL)
The proposed framework differs from existing offline model-based IRL approaches by performing simultaneous estimation of the expert's reward function and subjective model of environment dynamics.
Our analysis reveals a novel insight that the estimated policy exhibits robust performance when the expert is believed to have a highly accurate model of the environment.
arXiv Detail & Related papers (2023-09-15T17:37:09Z) - Designing Decision Support Systems Using Counterfactual Prediction Sets [15.121082690769525]
Decision support systems for classification tasks are predominantly designed to predict the value of the ground truth labels.
This paper revisits the design of this type of systems from the perspective of online learning.
We develop a methodology that does not require, nor assumes, an expert model.
arXiv Detail & Related papers (2023-06-06T18:00:09Z) - Incorporating Experts' Judgment into Machine Learning Models [2.5363839239628843]
In some cases, domain experts might have a judgment about the expected outcome that might conflict with the prediction of machine learning models.
We present a novel framework that aims at leveraging experts' judgment to mitigate the conflict.
arXiv Detail & Related papers (2023-04-24T07:32:49Z) - Test-time Collective Prediction [73.74982509510961]
Multiple parties in machine learning want to jointly make predictions on future test points.
Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters.
We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
arXiv Detail & Related papers (2021-06-22T18:29:58Z) - Towards Unbiased and Accurate Deferral to Multiple Experts [19.24068936057053]
We propose a framework that simultaneously learns a classifier and a deferral system, with the deferral system choosing to defer to one or more human experts.
We test our framework on a synthetic dataset and a content moderation dataset with biased synthetic experts, and show that it significantly improves the accuracy and fairness of the final predictions.
arXiv Detail & Related papers (2021-02-25T17:08:39Z) - Financial Data Analysis Using Expert Bayesian Framework For Bankruptcy
Prediction [0.0]
We propose another route of generative modeling using Expert Bayesian framework.
The biggest advantage of the proposed framework is an explicit inclusion of expert judgment in the modeling process.
The proposed approach is well suited for highly regulated or safety critical applications such as in finance or in medical diagnosis.
arXiv Detail & Related papers (2020-10-19T19:09:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.