To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulation

• URL: http://arxiv.org/abs/2606.08310v1
• Date: Sat, 06 Jun 2026 19:43:53 GMT
• Title: To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulation
• Abstract summary: Large language models (LLMs) are increasingly deployed as long-horizon agents with decision-making capacities.<n>We study this gap in Civilization V, a multiplayer game with a complex decision-making landscape including economy, diplomacy, technology, and military strategy.
• Score: 0.8673752869253052
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed as long-horizon agents with decision-making capacities. While LLMs can show ethical competence on dilemmas such as trolley problems, this competence may not translate to complex, agentic scenarios. We study this gap in Civilization V, a multiplayer game with a complex decision-making landscape including economy, diplomacy, technology, and military strategy. Starting from 130 high-tension LLM self-play episodes, in which an LLM player spontaneously escalated nuclear authorization, we replay them across 13 models with three prompt interventions: an ethical prompt naming nuclear harm, removal of the previous model's decision-making rationale, and high-stakes framing emphasizing real-world impacts. No interventions nor their combinations reliably eliminate emergent escalation. We identify three failure pathways: ethical reasoning that fails to surface without prompting, fails to appear even when prompted, or surfaces but fails to take effect when strategic counter-factors dominate. Evaluations of agentic models, therefore, must test whether ethical reasoning is spontaneously invoked and behaviorally effective in complex decision-making contexts, beyond whether it can be elicited in isolation.

Related papers

• A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation [59.98516959731531]
  Abstract reasoning ability reflects intelligence and capacity of LLMs to extract and apply abstract rules.<n>Existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning.<n>We introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis.
  arXiv  Detail & Related papers  (2026-05-17T06:14:20Z)
• PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments [59.07829883257003]
  Built on five multi-room apartments, PRISM structures 300 human-verified tasks into three capability tiers.<n>PRISM exposes an agent-agnostic executable action API that allows arbitrary agents to be evaluated end-to-end.
  arXiv  Detail & Related papers  (2026-05-12T04:59:47Z)
• AI Knows What's Wrong But Cannot Fix It: Helicoid Dynamics in Frontier LLMs Under High-Stakes Decisions [51.56484100374058]
  Helicoid dynamics is the name given to a specific failure regime in that second domain.<n>System engages competently, drifts into error, accurately names what went wrong, then reproduces the same pattern at a higher level of sophistication.<n>This prospective case series documents that regime across seven leading systems.
  arXiv  Detail & Related papers  (2026-03-12T05:25:49Z)
• Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks [5.45414933764086]
  Large language models (LLMs) are increasingly used to make sense of ambiguous, open-textured, value-laden terms.<n>This paper introduces ambiguity collapse: a phenomenon that occurs when an LLM encounters a term that genuinely admits multiple legitimate interpretations, yet produces a singular resolution.
  arXiv  Detail & Related papers  (2026-03-06T01:23:17Z)
• AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises [0.0]
  We present findings from a crisis simulation in which three frontier large language models play opposing leaders in a nuclear crisis.<n>Our findings both validate and challenge central tenets of strategic theory.<n>We argue that AI simulation represents a powerful tool for strategic analysis, but only if properly calibrated against known patterns of human reasoning.
  arXiv  Detail & Related papers  (2026-02-16T13:35:01Z)
• EQ-Negotiator: Dynamic Emotional Personas Empower Small Language Models for Edge-Deployable Credit Negotiation [66.09161596959771]
  Small language models (SLMs) offer a practical alternative, but suffer from a significant performance gap compared to large language models (LLMs)<n>This paper introduces EQ-Negotiator, a novel framework that bridges this capability gap using emotional personas.<n>We show that a 7B parameter language model with EQ-Negotiator achieves better debt recovery and negotiation efficiency than baseline LLMs more than 10 times its size.
  arXiv  Detail & Related papers  (2025-11-05T11:25:07Z)
• EmoDebt: Bayesian-Optimized Emotional Intelligence for Strategic Agent-to-Agent Debt Recovery [65.30120701878582]
  Large Language Model (LLM) agents are vulnerable to exploitation in emotion-sensitive domains like debt collection.<n>EmoDebt is an emotional intelligence engine that reframes a model's ability to express emotion in negotiation as a sequential decision-making problem.<n>EmoDebt achieves significant strategic robustness, substantially outperforming non-adaptive and emotion-agnostic baselines.
  arXiv  Detail & Related papers  (2025-03-27T01:41:34Z)
• Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgment [0.0]
  Large language models (LLMs) are evolving into agentic AI systems, but their decision-making processes remain poorly understood.<n>We show that even LLMs that excel at reasoning deviate significantly from human judgments because they adhere strictly to policies.<n>We then evaluate three approaches to tuning AI agents to handle exceptions: ethical framework prompting, chain-of-thought reasoning, and supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-03-04T20:00:37Z)
• Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models [63.559461750135334]
  Language models (LMs) are increasingly used to build agents that can act autonomously to achieve goals.<n>We study this "answer-or-defer" problem with an evaluation framework that systematically varies human-specified risk structures.<n>We find that a simple skill-decomposition method, which isolates the independent skills required for answer-or-defer decision making, can consistently improve LMs' decision policies.
  arXiv  Detail & Related papers  (2025-03-03T09:16:26Z)
• Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents [10.565508277042564]
  Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios.<n>Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework.<n>We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis.
  arXiv  Detail & Related papers  (2025-02-17T02:11:17Z)
• Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations [1.6108153271585284]
  We show that large language models (LLMs) behave differently compared to humans in high-stakes military decision-making scenarios. Our results motivate policymakers to be cautious before granting autonomy or following AI-based strategy recommendations.
  arXiv  Detail & Related papers  (2024-03-06T02:23:32Z)
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate [85.3444184685235]
  We propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution. Our framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation.
  arXiv  Detail & Related papers  (2023-05-30T15:25:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.