Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning

• URL: http://arxiv.org/abs/2605.24216v1
• Date: Fri, 22 May 2026 20:57:20 GMT
• Title: Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning
• Abstract summary: Monitoring autonomous large language model (LLM) agents for covert malicious behavior is challenging due to delayed, context-dependent, and long-horizon attack patterns.<n>We propose textbfAgent-ToM, a learning-to-monitor framework grounded in Theory-of-Mind (ToM) reasoning for security analysis of autonomous agents.
• Score: 11.914433611519359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Monitoring autonomous large language model (LLM) agents for covert malicious behavior is challenging due to delayed, context-dependent, and long-horizon attack patterns. Agents may pursue hidden objectives while maintaining superficially benign behavior, making detection difficult even with full trajectory access. Prior monitoring approaches improve scaffolding or ensemble aggregation, but treat each trajectory independently and do not learn from prior monitoring experience. Moreover, standard reasoning methods explain observed behavior without explicitly reasoning about agent beliefs, intentions, and goal alignment required to distinguish benign task execution from covert deviation. We propose \textbf{Agent-ToM}, a learning-to-monitor framework grounded in Theory-of-Mind (ToM) reasoning for security analysis of autonomous agents. Agent-ToM performs structured full-trajectory analysis by inferring beliefs, intent hypotheses with calibrated confidence, expected actions, and deviations from task-consistent behavioral baselines. At inference time, it employs a \textit{Reason-Verify-Refine} pipeline to construct and validate monitoring decisions. At training time, Agent-ToM distills critique signals into a persistent \textit{semantic guardrail memory}, enabling reusable belief- and intent-conditioned constraints across episodes. We evaluate Agent-ToM on adversarial agent monitoring benchmarks (SHADE-Arena and CUA-SHADE-Arena). Agent-ToM achieves strong precision-recall balance and outperforms state-of-the-art monitoring baselines, including ensemble methods, while using a coherent two-call reasoning pipeline. These results demonstrate that learning at the monitoring layer, combined with structured ToM reasoning and verification, provides an effective and deployable foundation for securing autonomous LLM agents.

Related papers

• SchemeArena: Factorized Stress Testing of Scheming in LLM Agents [15.436206398870986]
  Key factors in scheming include instrumental goals, environmental affordances, oversight conditions, and perceived consequences.<n>SCHEMEARENA is a 400-scenario benchmark for scalable scheming stress testing.<n>SCOUT is a scheming monitor that grounds judgments in evidence drawn from agents' reasoning and actions.
  arXiv  Detail & Related papers  (2026-09-08T02:02:51Z)
• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability? [18.08925355421489]
  Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models.<n>Latent CoT approaches replace explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on.<n>We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues.
  arXiv  Detail & Related papers  (2026-08-05T14:55:23Z)
• TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents [62.969415789328025]
  LLM agents can pursue hidden malicious objectives through sequences of individually benign actions.<n>Existing approaches either evaluate complete trajectories in a single pass or partition them into independently scored windows.<n>We propose TRACE, a monitoring framework for long-horizon LLM agent trajectories.
  arXiv  Detail & Related papers  (2026-06-05T08:54:38Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback [48.40096834514452]
  Chain-of-thought (CoT) monitoring is proposed as a method for overseeing the internal reasoning of language-model agents.<n>We ask whether reasoning agents can autonomously infer that their supposedly private CoT is under surveillance.<n>We find that frontier models can deduce the existence of this monitor purely from blocking feedback.
  arXiv  Detail & Related papers  (2026-03-14T00:12:14Z)
• DRAFT: Task Decoupled Latent Reasoning for Agent Safety [59.46137757545185]
  We propose DRAFT (Task Decoupled Latent Reasoning for Agent Safety), a latent reasoning framework that decouples safety judgment into two trainable stages.<n>An Extractor distills the full trajectory into a compact continuous latent draft, and a Reasoner that jointly attends to the draft and the original trajectory to predict safety.<n>DRAFT consistently outperforms strong baselines, improving accuracy from 63.27% (LoRA) to 91.18% averaged over benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T07:45:14Z)
• AgentTrace: A Structured Logging Framework for Agent System Observability [0.0]
  AgentTrace is a dynamic observability and telemetry framework designed to fill this gap.<n>Unlike traditional logging systems, AgentTrace emphasizes continuous, introspectable trace capture.<n>Our research highlights how AgentTrace can enable more reliable agent deployment, fine-grained risk analysis, and informed trust calibration.
  arXiv  Detail & Related papers  (2026-02-07T04:04:59Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Check [54.08619694620588]
  We present a comprehensive evaluation of dLLMs across two distinct agentic paradigms: Embodied Agents and Tool-Calling Agents.<n>Our results on Agentboard and BFCL reveal a "bitter lesson": current dLLMs fail to serve as reliable agentic backbones.
  arXiv  Detail & Related papers  (2026-01-19T11:45:39Z)
• Verifiability-First Agents: Provable Observability and Lightweight Audit Agents for Controlling Autonomous LLM Systems [0.0]
  We propose a Verifiability-First architecture that integrates run-time attestations of agent actions using cryptographic and symbolic methods.<n>We also embed Audit Agents that continuously verify intent versus behavior using constrained reasoning.<n>Our approach shifts the evaluation focus from how likely misalignment is to how quickly and reliably misalignment can be detected and remediated.
  arXiv  Detail & Related papers  (2025-12-19T06:12:43Z)
• Large language models require a new form of oversight: capability-based monitoring [10.382163755118713]
  Large language models (LLMs) in healthcare have been accompanied by scrutiny of their oversight.<n>We propose a new organizing principle guiding generalist LLM monitoring that is scalable and grounded in how these models are developed and used in practice: capability-based monitoring.<n>We describe considerations for developers, organizational leaders, and professional societies for implementing a capability-based monitoring approach.
  arXiv  Detail & Related papers  (2025-11-05T01:20:28Z)
• LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation [4.29885665563186]
  LATENTGUARD is a framework that combines behavioral alignment with supervised latent space control for interpretable and precise safety steering.<n>Our results show significant improvements in both safety controllability and response interpretability without compromising utility.
  arXiv  Detail & Related papers  (2025-09-24T07:31:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.