Related papers: Improving Human Performance with Value-Aware Interventions: A Case Study in Chess

Improving Human Performance with Value-Aware Interventions: A Case Study in Chess

URL: http://arxiv.org/abs/2604.14465v1
Date: Wed, 15 Apr 2026 22:55:48 GMT
Title: Improving Human Performance with Value-Aware Interventions: A Case Study in Chess
Abstract summary: A potential baseline is to recommend the optimal action according to a strong model.<n>Such actions assume optimal follow-up actions, which human decision makers may fail to execute.<n>We propose and study value-aware interventions, motivated by a basic principle in reinforcement learning.
Score: 12.313877328092824
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: AI systems are increasingly used to assist humans in sequential decision-making tasks, yet determining when and how an AI assistant should intervene remains a fundamental challenge. A potential baseline is to recommend the optimal action according to a strong model. However, such actions assume optimal follow-up actions, which human decision makers may fail to execute, potentially reducing overall performance. In this work, we propose and study value-aware interventions, motivated by a basic principle in reinforcement learning: under the Bellman equation, the optimal policy selects actions that maximize the immediate reward plus the value function. When a decision maker follows a suboptimal policy, this policy-value consistency no longer holds, creating discrepancies between the actions taken by the policy and those that maximize the immediate reward plus the value of the next state. We show that these policy-value inconsistencies naturally identify opportunities for intervention. We formalize this problem in a Markov decision process where an AI assistant may override human actions under an intervention budget. In the single-intervention regime, we show that the optimal strategy is to recommend the action that maximizes the human value function. For settings with multiple interventions, we propose a tractable approximation that prioritizes interventions based on the magnitude of the policy-value discrepancy. We evaluate these ideas in the domain of chess by learning models of humans from large-scale gameplay data. In simulation, our approach consistently outperforms interventions based on the strongest chess engine (Stockfish) in a wide range of settings. A within-subject human study with 20 players and 600 games further shows that our interventions significantly improve performance for low- and mid-skill players while matching expert-engine interventions for high-skill players.

Related papers

Anticipating Gaming to Incentivize Improvement: Guiding Agents in (Fair) Strategic Classification [6.660458629649826]
We explore individuals' choice between genuinely improving their qualifications ('improvement'') vs. attempting to deceive the algorithm.<n>We formulate these interactions as a Stackelberg game, where a firm deploys a (fair) classifier, and individuals strategically respond.
arXiv Detail & Related papers (2025-05-08T18:47:23Z)
MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention [76.83428371942735]
We introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning), designed for sample-efficient alignment from human intervention.<n>MereQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions.<n>It then employs Residual Q-Learning (RQL) to align the policy with human preferences using this residual reward function.
arXiv Detail & Related papers (2024-06-24T01:51:09Z)
Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games [34.34801907296059]
In human-AI interaction, a prominent goal is to attain humans desirable outcome with the assistance of AI agents. We employ a theoretical framework called structural causal game (SCG) to formalize the human-AI interactive process. We introduce a strategy referred to as pre-policy intervention on the SCG to steer AI agents towards attaining the humans desirable outcome.
arXiv Detail & Related papers (2024-05-26T14:42:49Z)
Algorithmic Decision-Making under Agents with Persistent Improvement [9.296248945826084]
We study algorithmic decision-making under human's strategic behavior. We first develop a dynamic model to characterize persistent improvements. We then study how the decision-maker can design an optimal policy to incentivize the largest improvements inside the agent population.
arXiv Detail & Related papers (2024-05-03T01:36:35Z)
Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
arXiv Detail & Related papers (2024-04-29T08:16:30Z)
A Minimaximalist Approach to Reinforcement Learning from Human Feedback [49.45285664482369]
We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches.
arXiv Detail & Related papers (2024-01-08T17:55:02Z)
Learning Complementary Policies for Human-AI Teams [13.371050441794651]
This paper tackles the challenge of human-AI complementarity in decision-making.<n>We develop a robust solution for human-AI collaboration when outcomes are only observed under assigned actions.<n>We show that substantial performance improvements are achievable by routing only a small fraction of instances to human decision-makers.
arXiv Detail & Related papers (2023-02-06T17:22:18Z)
Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
arXiv Detail & Related papers (2021-06-11T16:49:15Z)
Stateful Strategic Regression [20.7177095411398]
We describe the Stackelberg equilibrium of the resulting game and provide novel algorithms for computing it. Our analysis reveals several intriguing insights about the role of multiple interactions in shaping the game's outcome. Most importantly, we show that with multiple rounds of interaction at her disposal, the principal is more effective at incentivizing the agent to accumulate effort in her desired direction.
arXiv Detail & Related papers (2021-06-07T17:46:29Z)
End-to-End Learning and Intervention in Games [60.41921763076017]
We provide a unified framework for learning and intervention in games. We propose two approaches, respectively based on explicit and implicit differentiation. The analytical results are validated using several real-world problems.
arXiv Detail & Related papers (2020-10-26T18:39:32Z)
Cooperative Inverse Reinforcement Learning [64.60722062217417]
We propose a formal definition of the value alignment problem as cooperative reinforcement learning (CIRL) A CIRL problem is a cooperative, partial-information game with two agents human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions.
arXiv Detail & Related papers (2016-06-09T22:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.