Formal Analysis of AGI Decision-Theoretic Models and the Confrontation Question

• URL: http://arxiv.org/abs/2601.04234v1
• Date: Sun, 04 Jan 2026 08:02:00 GMT
• Title: Formal Analysis of AGI Decision-Theoretic Models and the Confrontation Question
• Abstract summary: Artificial General Intelligence (AGI) may face a confrontation question: under what conditions would a self-interested AGI choose to seize power or eliminate human control (a confrontation) rather than remain cooperative?<n>We formalize this in a Markov decision process with a human-initiated shutdown event.<n>We show that for almost all reward functions a misaligned agent has an incentive to avoid shutdown.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Artificial General Intelligence (AGI) may face a confrontation question: under what conditions would a rationally self-interested AGI choose to seize power or eliminate human control (a confrontation) rather than remain cooperative? We formalize this in a Markov decision process with a stochastic human-initiated shutdown event. Building on results on convergent instrumental incentives, we show that for almost all reward functions a misaligned agent has an incentive to avoid shutdown. We then derive closed-form thresholds for when confronting humans yields higher expected utility than compliant behavior, as a function of the discount factor $γ$, shutdown probability $p$, and confrontation cost $C$. For example, a far-sighted agent ($γ=0.99$) facing $p=0.01$ can have a strong takeover incentive unless $C$ is sufficiently large. We contrast this with aligned objectives that impose large negative utility for harming humans, which makes confrontation suboptimal. In a strategic 2-player model (human policymaker vs AGI), we prove that if the AGI's confrontation incentive satisfies $Δ\ge 0$, no stable cooperative equilibrium exists: anticipating this, a rational human will shut down or preempt the system, leading to conflict. If $Δ< 0$, peaceful coexistence can be an equilibrium. We discuss implications for reward design and oversight, extend the reasoning to multi-agent settings as conjectures, and note computational barriers to verifying $Δ< 0$, citing complexity results for planning and decentralized decision problems. Numerical examples and a scenario table illustrate regimes where confrontation is likely versus avoidable.

Related papers

• A Mathematical Theory of Value: a synthesis on goal-directed agency under resource constraints [6.057587531186626]
  We show that value, the quantity goal-directed agents create, destroy, and exchange, is a lawful structural quantity in the same category as information.<n>For populations, value is frame-relative while price is frame-independent; a fleet that pools its resource and fuses its perception inherits the ceiling.
  arXiv  Detail & Related papers  (2026-06-10T16:11:04Z)
• Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI [49.05972944310475]
  We show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate.<n>We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial.
  arXiv  Detail & Related papers  (2026-05-08T19:37:07Z)
• Scaling Laws for Moral Machine Judgment in Large Language Models [0.0]
  We evaluate 75 large language model configurations (0.27B--1000B parameters) using the Moral Machine framework.<n>We observe a consistent power-law relationship with distance from human preferences decreasing as model size.<n> Extended reasoning models show significantly better alignment, with this effect being more pronounced in smaller models.
  arXiv  Detail & Related papers  (2026-01-25T00:12:45Z)
• Rational Adversaries and the Maintenance of Fragility: A Game-Theoretic Theory of Rational Stagnation [0.0]
  This paper explains such "rational stagnation" as an equilibrium sustained by a rational adversary.<n> Applications to social-media algorithms and political trust illustrate how adversarial can deliberately preserve rationality.
  arXiv  Detail & Related papers  (2025-10-25T09:28:15Z)
• Deceptive Sequential Decision-Making via Regularized Policy Optimization [54.38738815697299]
  We model autonomous systems as Markov decision processes, with adversaries using inverse reinforcement learning to recover reward functions.<n>We show three regularization strategies for policy synthesis problems that actively deceive an adversary about a system's reward.<n>We show that diversionary, targeted, and equivocal deception all steer the adversary to false beliefs while still attaining a total accumulated reward that is at least 97% of its optimal, non-deceptive value.
  arXiv  Detail & Related papers  (2025-01-30T23:41:40Z)
• Metareasoning in uncertain environments: a meta-BAMDP framework [1.0923877073891441]
  Finding the right $P$ can itself be framed as an optimization problem over the space of reasoning processes $P$.<n>This paper proposes a meta Bayes-Adaptive MDP framework to handle metareasoning in environments with unknown reward/transition distributions.
  arXiv  Detail & Related papers  (2024-08-02T13:15:01Z)
• Collaborative Mean Estimation Among Heterogeneous Strategic Agents: Individual Rationality, Fairness, and Truthful Contribution [11.371461065112422]
  We study a collaborative learning problem where $m$ agents aim to estimate a vector $mu =(mu_k, sigma2)_kin[d]$.<n>Instead of working independently, agents can exchange data, collecting cheaper samples and sharing them in return for costly data, thereby reducing both costs and estimation error.
  arXiv  Detail & Related papers  (2024-07-20T17:45:40Z)
• Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback [58.66941279460248]
  Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM)<n>We study a model within this domain--contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary.<n>We propose an algorithm namely robust contextual dueling bandits (RCDB), which is based on uncertainty-weighted maximum likelihood estimation.
  arXiv  Detail & Related papers  (2024-04-16T17:59:55Z)
• Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark [61.43264961005614]
  We develop a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios. We evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. Our results show that agents can both act competently and morally, so concrete progress can be made in machine ethics.
  arXiv  Detail & Related papers  (2023-04-06T17:59:03Z)
• Autoregressive Bandits [58.46584210388307]
  We propose a novel online learning setting, Autoregressive Bandits, in which the observed reward is governed by an autoregressive process of order $k$. We show that, under mild assumptions on the reward process, the optimal policy can be conveniently computed. We then devise a new optimistic regret minimization algorithm, namely, AutoRegressive Upper Confidence Bound (AR-UCB), that suffers sublinear regret of order $widetildemathcalO left( frac(k+1)3/2sqrtnT (1-G
  arXiv  Detail & Related papers  (2022-12-12T21:37:36Z)
• Misspecification in Inverse Reinforcement Learning [80.91536434292328]
  The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function $R$ from a policy $pi$. One of the primary motivations behind IRL is to infer human preferences from human behaviour. This means that they are misspecified, which raises the worry that they might lead to unsound inferences if applied to real-world data.
  arXiv  Detail & Related papers  (2022-12-06T18:21:47Z)
• The Sample Complexity of Online Contract Design [120.9833763323407]
  We study the hidden-action principal-agent problem in an online setting. In each round, the principal posts a contract that specifies the payment to the agent based on each outcome. The agent then makes a strategic choice of action that maximizes her own utility, but the action is not directly observable by the principal.
  arXiv  Detail & Related papers  (2022-11-10T17:59:42Z)
• On Avoiding Power-Seeking by Artificial Intelligence [93.9264437334683]
  We do not know how to align a very intelligent AI agent's behavior with human interests. I investigate whether we can build smart AI agents which have limited impact on the world, and which do not autonomously seek power.
  arXiv  Detail & Related papers  (2022-06-23T16:56:21Z)
• Using reinforcement learning to autonomously identify sources of error for agents in group missions [0.22499166814992436]
  We consider the question as to whether artificial intelligence can autonomously generate an action plan to pinpoint the cause as aforedescribed. To overcome this problem, we formulated an action plan using Q-table reinforcement learning. Surprisingly, the optimal action plan generated via reinforcement learning presented a human-like solution to pinpoint the problem.
  arXiv  Detail & Related papers  (2021-07-20T02:40:19Z)
• From Checking to Inference: Actual Causality Computations as Optimization Problems [79.87179017975235]
  We present a novel approach to formulate different notions of causal reasoning, over binary acyclic models, as optimization problems. We show that both notions are efficiently automated. Using models with more than $8000$ variables, checking is computed in a matter of seconds, with MaxSAT outperforming ILP in many cases.
  arXiv  Detail & Related papers  (2020-06-05T10:56:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.