Risk-Aware Preference Learning for Stochastic Outcomes

• URL: http://arxiv.org/abs/2607.15483v1
• Date: Thu, 16 Jul 2026 21:59:38 GMT
• Title: Risk-Aware Preference Learning for Stochastic Outcomes
• Abstract summary: We study the consequences of a mismatch in social robot navigation, where safety-critical outcomes are rare but highly consequential.<n>Our results highlight the importance of modeling human risk sensitivity when learning rewards from preferences over robot outcomes.
• Score: 42.156317447346815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning reward functions from human preferences is a widely used approach for aligning robot behavior with user expectations in human-robot interaction. Most existing approaches assume that humans evaluate uncertain outcomes using expected utility (EU), aggregating outcome utilities linearly with their probabilities. However, behavioral evidence shows that humans are systematically risk-sensitive, overweighting rare negative events and exhibiting loss aversion. We study the consequences of this mismatch in social robot navigation, where safety-critical outcomes (e.g., collisions) are rare but highly consequential. We compare EU with Cumulative Prospect Theory (CPT), a nonlinear model of human decision-making, within a Bradley-Terry preference learning framework. Our preliminary experiments show that when preferences are generated by risk-sensitive users, CPT-based learners recover reward functions with substantially lower regret compared to EU-based learners. Our results highlight the importance of modeling human risk sensitivity when learning rewards from preferences over stochastic robot outcomes.

Related papers

• Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration [2.8952292379640636]
  In Human-Robot Interaction, the standard approach to learn a reward model that represents human preferences for robot behavior consists of three steps.<n>First, the robot collects limited direct evidence from human feedback (e.g., positive or negative binary feedback)<n>Second, the robot utilizes the direct evidence to infer accepted or rejected labels feasible but unchosen actions using fixed implication rules.<n>Third, the robot updates the reward model with both the direct and derived evidence.
  arXiv  Detail & Related papers  (2026-09-12T14:54:54Z)
• Few-Shot Inference of Human Perceptions of Robot Performance in Social Navigation Scenarios [1.5415050466360671]
  We propose leveraging the few-shot learning capabilities of Large Language Models to improve how well a robot can predict a user's perception of its performance.<n>This work paves the path to improving robot behavior in a scalable manner through user-centered feedback.
  arXiv  Detail & Related papers  (2025-12-17T23:06:36Z)
• Prospect Theory in Physical Human-Robot Interaction: A Pilot Study of Probability Perception [2.675153459461615]
  Understanding how humans respond to uncertainty is critical for designing safe and effective physical human-robot interaction (pHRI)<n>This study implements a physically coupled target-reaching task in which the robot delivered assistance or disturbances with systematically varied probabilities.<n>Analysis of participants' force inputs and decision-making strategies revealed two distinct behavioral clusters.<n>These findings provide empirical evidence that human decision-making in pHRI is highly individualized and that the perception of probability can differ to its true value.
  arXiv  Detail & Related papers  (2025-12-09T10:55:58Z)
• Can Risk-taking AI-Assistants suitably represent entities [0.0]
  This study investigates the manipulability of risk aversion in language models (LMs)<n>It focuses on gender-specific attitudes, uncertainty, role-based decision-making, and the manipulability of risk aversion.<n>The results suggest directions for refining AI design to better align human and AI risk preferences.
  arXiv  Detail & Related papers  (2025-10-09T11:55:31Z)
• Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation [59.500347564280204]
  We propose a new Aleatoric Uncertainty-aware Recommendation (AUR) framework. AUR consists of a new uncertainty estimator along with a normal recommender model. As the chance of mislabeling reflects the potential of a pair, AUR makes recommendations according to the uncertainty.
  arXiv  Detail & Related papers  (2022-09-22T04:32:51Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)
• Two steps to risk sensitivity [4.974890682815778]
  conditional value-at-risk (CVaR) is a risk measure for modeling human and animal planning. We adopt a conventional distributional approach to CVaR in a sequential setting and reanalyze the choices of human decision-makers. We then consider a further critical property of risk sensitivity, namely time consistency, showing alternatives to this form of CVaR.
  arXiv  Detail & Related papers  (2021-11-12T16:27:47Z)
• Probabilistic Human Motion Prediction via A Bayesian Neural Network [71.16277790708529]
  We propose a probabilistic model for human motion prediction in this paper. Our model could generate several future motions when given an observed motion sequence. We extensively validate our approach on a large scale benchmark dataset Human3.6m.
  arXiv  Detail & Related papers  (2021-07-14T09:05:33Z)
• Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction [55.569050872780224]
  We present an online framework for safe crowd-robot interaction based on risk-sensitive optimal control, wherein the risk is modeled by the entropic risk measure. Our modular approach decouples the crowd-robot interaction into learning-based prediction and model-based control. A simulation study and a real-world experiment show that the proposed framework can accomplish safe and efficient navigation while avoiding collisions with more than 50 humans in the scene.
  arXiv  Detail & Related papers  (2020-09-12T02:02:52Z)
• Bounded Risk-Sensitive Markov Games: Forward Policy Design and Inverse Reward Learning with Iterative Reasoning and Cumulative Prospect Theory [33.57592649823294]
  We investigate the problem of bounded risk-sensitive Markov Game (BRSMG) and its inverse reward learning problem. We show that humans have bounded intelligence and maximize risk-sensitive utilities in BRSMGs. The results show that the behaviors of agents demonstrate both risk-averse and risk-seeking characteristics.
  arXiv  Detail & Related papers  (2020-09-03T07:32:32Z)
• Learning Bounds for Risk-sensitive Learning [86.50262971918276]
  In risk-sensitive learning, one aims to find a hypothesis that minimizes a risk-averse (or risk-seeking) measure of loss. We study the generalization properties of risk-sensitive learning schemes whose optimand is described via optimized certainty equivalents.
  arXiv  Detail & Related papers  (2020-06-15T05:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.