Dealing with Expert Bias in Collective Decision-Making
- URL: http://arxiv.org/abs/2106.13539v1
- Date: Fri, 25 Jun 2021 10:17:37 GMT
- Title: Dealing with Expert Bias in Collective Decision-Making
- Authors: Axel Abels, Tom Lenaerts, Vito Trianni, Ann Now\'e
- Abstract summary: We propose a new algorithmic approach based on contextual multi-armed bandit problems (CMAB) to identify and counteract biased expertises.
Our novel CMAB-inspired approach achieves a higher final performance and does so while converging more rapidly than previous adaptive algorithms.
- Score: 4.588028371034406
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Quite some real-world problems can be formulated as decision-making problems
wherein one must repeatedly make an appropriate choice from a set of
alternatives. Expert judgements, whether human or artificial, can help in
taking correct decisions, especially when exploration of alternative solutions
is costly. As expert opinions might deviate, the problem of finding the right
alternative can be approached as a collective decision making problem (CDM).
Current state-of-the-art approaches to solve CDM are limited by the quality of
the best expert in the group, and perform poorly if experts are not qualified
or if they are overly biased, thus potentially derailing the decision-making
process. In this paper, we propose a new algorithmic approach based on
contextual multi-armed bandit problems (CMAB) to identify and counteract such
biased expertises. We explore homogeneous, heterogeneous and polarised expert
groups and show that this approach is able to effectively exploit the
collective expertise, irrespective of whether the provided advice is directly
conducive to good performance, outperforming state-of-the-art methods,
especially when the quality of the provided expertise degrades. Our novel
CMAB-inspired approach achieves a higher final performance and does so while
converging more rapidly than previous adaptive algorithms, especially when
heterogeneous expertise is readily available.
Related papers
- Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
We study the theoretical properties of the class of reward functions that are compatible with a given set of experts.
Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards.
We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
arXiv Detail & Related papers (2024-01-08T12:39:25Z) - Expert-guided Bayesian Optimisation for Human-in-the-loop Experimental
Design of Known Systems [0.0]
We apply high- throughput (batch) Bayesian optimisation alongside anthropological decision theory to enable domain experts to influence the selection of optimal experiments.
Our methodology exploits the hypothesis that humans are better at making discrete choices than continuous ones and enables experts to influence critical early decisions.
arXiv Detail & Related papers (2023-12-05T16:09:31Z) - Online Decision Mediation [72.80902932543474]
Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior.
In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
arXiv Detail & Related papers (2023-10-28T05:59:43Z) - Expertise Trees Resolve Knowledge Limitations in Collective
Decision-Making [2.924868086534434]
We model such changes in depth and breadth of knowledge as a partitioning of the problem space into regions of differing expertise.
We provide here new algorithms that explicitly consider and adapt to the relationship between problem instances and experts' knowledge.
arXiv Detail & Related papers (2023-05-02T16:59:57Z) - On solving decision and risk management problems subject to uncertainty [91.3755431537592]
Uncertainty is a pervasive challenge in decision and risk management.
This paper develops a systematic understanding of such strategies, determine their range of application, and develop a framework to better employ them.
arXiv Detail & Related papers (2023-01-18T19:16:23Z) - A Machine Learning Framework Towards Transparency in Experts' Decision
Quality [0.0]
In many important settings, transparency in experts' decision quality is rarely possible because ground truth data for evaluating the experts' decisions is costly and available only for a limited set of decisions.
We first formulate the problem of estimating experts' decision accuracy in this setting and then develop a machine-learning-based framework to address it.
Our method effectively leverages both abundant historical data on workers' past decisions, and scarce decision instances with ground truth information.
arXiv Detail & Related papers (2021-10-21T18:50:40Z) - Decision Rule Elicitation for Domain Adaptation [93.02675868486932]
Human-in-the-loop machine learning is widely used in artificial intelligence (AI) to elicit labels from experts.
In this work, we allow experts to additionally produce decision rules describing their decision-making.
We show that decision rule elicitation improves domain adaptation of the algorithm and helps to propagate expert's knowledge to the AI model.
arXiv Detail & Related papers (2021-02-23T08:07:22Z) - Confidence-Budget Matching for Sequential Budgeted Learning [69.77435313099366]
We formalize decision-making problems with querying budget.
We consider multi-armed bandits, linear bandits, and reinforcement learning problems.
We show that CBM based algorithms perform well in the presence of adversity.
arXiv Detail & Related papers (2021-02-05T19:56:31Z) - Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap.
We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert.
Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
arXiv Detail & Related papers (2021-01-24T05:40:29Z) - Expert Selection in High-Dimensional Markov Decision Processes [5.52481973699219]
Our method takes a set of candidate expert policies and switches between them to rapidly identify the best performing expert.
This is useful in applications where several expert policies may be available, and one needs to be selected at run-time for the underlying environment.
arXiv Detail & Related papers (2020-10-26T03:57:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.