Learning the Preferences of Uncertain Humans with Inverse Decision Theory

• URL: http://arxiv.org/abs/2106.10394v1
• Date: Sat, 19 Jun 2021 00:11:13 GMT
• Title: Learning the Preferences of Uncertain Humans with Inverse Decision Theory
• Authors: Cassidy Laidlaw and Stuart Russell
• Abstract summary: We study the setting of inverse decision theory (IDT), a framework where a human is observed making non-sequential binary decisions under uncertainty. In IDT, the human's preferences are conveyed through their loss function, which expresses a tradeoff between different types of mistakes. We show that it is actually easier to identify preferences when the decision problem is more uncertain.
• Score: 10.926992035470372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing observational approaches for learning human preferences, such as inverse reinforcement learning, usually make strong assumptions about the observability of the human's environment. However, in reality, people make many important decisions under uncertainty. To better understand preference learning in these cases, we study the setting of inverse decision theory (IDT), a previously proposed framework where a human is observed making non-sequential binary decisions under uncertainty. In IDT, the human's preferences are conveyed through their loss function, which expresses a tradeoff between different types of mistakes. We give the first statistical analysis of IDT, providing conditions necessary to identify these preferences and characterizing the sample complexity -- the number of decisions that must be observed to learn the tradeoff the human is making to a desired precision. Interestingly, we show that it is actually easier to identify preferences when the decision problem is more uncertain. Furthermore, uncertain decision problems allow us to relax the unrealistic assumption that the human is an optimal decision maker but still identify their exact preferences; we give sample complexities in this suboptimal case as well. Our analysis contradicts the intuition that partial observability should make preference learning more difficult. It also provides a first step towards understanding and improving preference learning methods for uncertain and suboptimal humans.

Related papers

• Decision Theoretic Foundations for Experiments Evaluating Human Decisions [18.27590643693167]
  We argue that to attribute loss in human performance to forms of bias, an experiment must provide participants with the information that a rational agent would need to identify the utility-maximizing decision. As a demonstration, we evaluate the extent to which recent evaluations of decision-making from the literature on AI-assisted decisions achieve these criteria.
  arXiv  Detail & Related papers  (2024-01-25T16:21:37Z)
• Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making [0.0]
  Explanations help to facilitate communication between the algorithm and the human decision-maker. Feature-based explanations' semantics of causal models induce leakage from the decision-maker's prior beliefs. Such differences can lead to sub-optimal and biased decision outcomes.
  arXiv  Detail & Related papers  (2022-06-17T11:43:42Z)
• Models of human preference for learning reward functions [80.39289349661364]
  We learn the reward function from human-generated preferences between pairs of trajectory segments. We find this assumption to be flawed and propose modeling human preferences as informed by each segment's regret. Our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned.
  arXiv  Detail & Related papers  (2022-06-05T17:58:02Z)
• The Statistical Complexity of Interactive Decision Making [126.04974881555094]
  We provide a complexity measure, the Decision-Estimation Coefficient, that is proven to be both necessary and sufficient for sample-efficient interactive learning. A unified algorithm design principle, Estimation-to-Decisions (E2D), transforms any algorithm for supervised estimation into an online algorithm for decision making.
  arXiv  Detail & Related papers  (2021-12-27T02:53:44Z)
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning [52.74071439183113]
  We study the predict-then-optimize framework in the context of sequential decision problems (formulated as MDPs) solved via reinforcement learning. Two significant computational challenges arise in applying decision-focused learning to MDPs.
  arXiv  Detail & Related papers  (2021-06-06T23:53:31Z)
• Uncertainty quantification and exploration-exploitation trade-off in humans [0.0]
  The main objective of this paper is to outline a theoretical framework to analyse how humans' decision-making strategies under uncertainty manage the trade-off between information gathering (exploration) and reward seeking (exploitation)
  arXiv  Detail & Related papers  (2021-02-05T16:03:04Z)
• Indecision Modeling [50.00689136829134]
  It is important that AI systems act in ways which align with human values. People are often indecisive, and especially so when their decision has moral implications.
  arXiv  Detail & Related papers  (2020-12-15T18:32:37Z)
• When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making [68.19284302320146]
  We carry out user studies to assess how people with differing levels of expertise respond to different types of predictive uncertainty. We found that showing posterior predictive distributions led to smaller disagreements with the ML model's predictions. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.
  arXiv  Detail & Related papers  (2020-11-12T02:23:53Z)
• Implications of Human Irrationality for Reinforcement Learning [26.76732313120685]
  We argue that human decision making may be a better source of ideas for constraining how machine learning problems are defined than would otherwise be the case. One promising idea concerns human decision making that is dependent on apparently irrelevant aspects of the choice context. We propose a novel POMDP model for contextual choice tasks and show that, despite the apparent irrationalities, a reinforcement learner can take advantage of the way that humans make decisions.
  arXiv  Detail & Related papers  (2020-06-07T07:44:53Z)
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores [85.12096045419686]
  We study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We first show that humans do alter their behavior when the tool is deployed. We show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk.
  arXiv  Detail & Related papers  (2020-02-19T07:27:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.