The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

• URL: http://arxiv.org/abs/2603.20994v1
• Date: Sun, 22 Mar 2026 00:50:32 GMT
• Title: The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes
• Abstract summary: In shared autonomy, a tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm.<n>This paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games.
• Score: 6.084651238441389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.

Related papers

• When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making [68.12864562049957]
  Embodied robotic systems increasingly rely on large language model (LLM)-based agents to support high-level reasoning.<n>This raises a fundamental question for embodied agents: when should the agent reason, and when should it act?<n>We propose RARRL (Resource-Aware Reasoning via Reinforcement Learning), a hierarchical framework for resource-aware orchestration of embodied agents.
  arXiv  Detail & Related papers  (2026-03-17T15:38:50Z)
• Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspective [31.55000083809067]
  We show how game-theoretic deterrence can make AI oversight proactive, risk-aware, and resilient to manipulation.<n>We illustrate how this framework can inform (1) training-time auditing against data/feedback poisoning, (2) pre-deployment evaluation under constrained reviewer resources, and (3) robust multi-model deployment in adversarial environments.
  arXiv  Detail & Related papers  (2026-02-06T23:20:26Z)
• The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy [9.553819152637493]
  We study a minimal control interface where an agent chooses whether to act autonomously (play) or defer (ask)<n>If the agent defers, the human's choice determines the outcome, potentially leading to a corrective action or a system shutdown.<n>Our analysis focuses on cases where this game qualifies as a Markov Potential Game (MPG), a class of games where we can provide an alignment guarantee.
  arXiv  Detail & Related papers  (2025-10-30T17:46:49Z)
• The Ultimate Test of Superintelligent AI Agents: Can an AI Balance Care and Control in Asymmetric Relationships? [11.983461426678522]
  The Shepherd Test is a new conceptual test for assessing the moral and relational dimensions of superintelligent artificial agents.<n>We argue that AI crosses an important, and potentially dangerous, threshold of intelligence when it exhibits the ability to manipulate, nurture, and instrumentally use less intelligent agents.<n>This includes the ability to weigh moral trade-offs between self-interest and the well-being of subordinate agents.
  arXiv  Detail & Related papers  (2025-06-02T15:53:56Z)
• Toward a Theory of Agents as Tool-Use Decision-Makers [89.26889709510242]
  We argue that true autonomy requires agents to be grounded in a coherent epistemic framework that governs what they know, what they need to know, and how to acquire that knowledge efficiently.<n>We propose a unified theory that treats internal reasoning and external actions as equivalent epistemic tools, enabling agents to systematically coordinate introspection and interaction.<n>This perspective shifts the design of agents from mere action executors to knowledge-driven intelligence systems, offering a principled path toward building foundation agents capable of adaptive, efficient, and goal-directed behavior.
  arXiv  Detail & Related papers  (2025-06-01T07:52:16Z)
• Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents [61.132523071109354]
  This paper investigates the interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios.<n>Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" stances than pure game-theoretic agents.
  arXiv  Detail & Related papers  (2025-04-11T15:41:21Z)
• Strategy Masking: A Method for Guardrails in Value-based Reinforcement Learning Agents [0.27309692684728604]
  We study methods for constructing guardrails for AI agents that use reward functions to learn decision making.<n>We introduce a novel approach, which we call strategy masking, to explicitly learn and then suppress undesirable AI agent behavior.
  arXiv  Detail & Related papers  (2025-01-09T18:43:05Z)
• Combining AI Control Systems and Human Decision Support via Robustness and Criticality [53.10194953873209]
  We extend a methodology for adversarial explanations (AE) to state-of-the-art reinforcement learning frameworks. We show that the learned AI control system demonstrates robustness against adversarial tampering. In a training / learning framework, this technology can improve both the AI's decisions and explanations through human interaction.
  arXiv  Detail & Related papers  (2024-07-03T15:38:57Z)
• Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
  Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL) This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
  arXiv  Detail & Related papers  (2023-12-22T17:57:57Z)
• The Feasibility and Inevitability of Stealth Attacks [63.14766152741211]
  We study new adversarial perturbations that enable an attacker to gain control over decisions in generic Artificial Intelligence systems. In contrast to adversarial data modification, the attack mechanism we consider here involves alterations to the AI system itself.
  arXiv  Detail & Related papers  (2021-06-26T10:50:07Z)
• Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data [18.750834997334664]
  We argue that humans are bounded rational and have different intelligence levels when reasoning about others' decision-making process. We propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans' latent intelligence levels during learning.
  arXiv  Detail & Related papers  (2021-03-07T07:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.