CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

• URL: http://arxiv.org/abs/2604.09746v1
• Date: Fri, 10 Apr 2026 06:33:57 GMT
• Title: CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation
• Abstract summary: We introduce a large-scale multi-agent simulation in a simplified model of New York City.<n>Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes.<n>Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive.
• Score: 15.334072037636881
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.

Related papers

• Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks [56.45762972211923]
  We present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks.<n>It employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies.<n>We present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks.
  arXiv  Detail & Related papers  (2026-07-31T14:04:49Z)
• Learning Agent-Compatible Context Management for Long-Horizon Tasks [79.09330817637205]
  We introduce Adaptive Context Management (AdaCoM), which trains an external LLM to manage the context of a frozen agent.<n>AdaCoM substantially improves performance by preserving task constraints and progress while pruning stale content.<n>The learned strategies reveal a Fidelity-Reliability Trade-off: agents with higher vanilla ReAct performance benefit from higher-fidelity context preservation, whereas lower-performing agents require more aggressive compression to stay within a reliable reasoning regime.
  arXiv  Detail & Related papers  (2026-05-29T03:21:08Z)
• Defending LLM-based Multi-Agent Systems Against Cooperative Attacks with Sentence-Level Rectification [42.88763759237844]
  Malicious agents in Large Language Model-based Multi-Agent Systems (MAS) may inject misinformation to mislead other agents and disrupt system performance.<n>We propose an adaptive cooperative attack framework, where malicious agents autonomously coordinate and dynamically adjust their attack strategies.<n>We introduce Sentence-Level Trustworthiness Analysis and Rectification (STAR), a defense framework that identifies and rectifies misleading information at the sentence level within agent communications.
  arXiv  Detail & Related papers  (2026-05-27T07:56:33Z)
• Policy Disruption in Reinforcement Learning:Adversarial Attack with Large Language Models and Critical State Identification [8.292056374554162]
  Reinforcement learning (RL) has achieved remarkable success in fields like robotics and autonomous driving.<n>Existing approaches often rely on modifying the environment or policy, limiting their practicality.<n>This paper proposes an adversarial attack method in which existing agents in the environment guide the target policy to output suboptimal actions without altering the environment.
  arXiv  Detail & Related papers  (2025-07-24T05:52:06Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)
• EmoDebt: Bayesian-Optimized Emotional Intelligence for Strategic Agent-to-Agent Debt Recovery [65.30120701878582]
  Large Language Model (LLM) agents are vulnerable to exploitation in emotion-sensitive domains like debt collection.<n>EmoDebt is an emotional intelligence engine that reframes a model's ability to express emotion in negotiation as a sequential decision-making problem.<n>EmoDebt achieves significant strategic robustness, substantially outperforming non-adaptive and emotion-agnostic baselines.
  arXiv  Detail & Related papers  (2025-03-27T01:41:34Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Evil Geniuses: Delving into the Safety of LLM-based Agents [35.49857256840015]
  Large language models (LLMs) have revitalized in large language models (LLMs) This paper delves into the safety of LLM-based agents from three perspectives: agent quantity, role definition, and attack level.
  arXiv  Detail & Related papers  (2023-11-20T15:50:09Z)
• Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policy [32.1138935956272]
  Reinforcement learning agents are susceptible to evasion attacks during deployment. In this paper, we propose Intrinsically Motivated Adrial Policy (IMAP) for efficient black-box adversarial policy learning.
  arXiv  Detail & Related papers  (2023-05-04T07:24:12Z)
• Influencing Towards Stable Multi-Agent Interactions [12.477674452685756]
  Learning in multi-agent environments is difficult due to the non-stationarity introduced by an opponent's or partner's changing behaviors. We propose an algorithm to proactively influence the other agent's strategy to stabilize. We demonstrate the effectiveness of stabilizing in improving efficiency of maximizing the task reward in a variety of simulated environments.
  arXiv  Detail & Related papers  (2021-10-05T16:46:04Z)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary [86.0846119254031]
  We study the robustness of reinforcement learning with adversarially perturbed state observations. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
  arXiv  Detail & Related papers  (2021-01-21T05:38:52Z)
• Towards Learning Multi-agent Negotiations via Self-Play [2.28438857884398]
  We show how an iterative procedure of self-play can create progressively more diverse environments. This leads to the learning of sophisticated and robust multi-agent policies. We show a dramatic improvement in the success rate of merging maneuvers from 63% to over 98%.
  arXiv  Detail & Related papers  (2020-01-28T08:37:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.