Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors

• URL: http://arxiv.org/abs/2605.06490v1
• Date: Thu, 07 May 2026 16:12:36 GMT
• Title: Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
• Abstract summary: We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents.<n>Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds.<n>Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models.
• Score: 15.880694616254297
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3% of IC cases and three tasks account for 84.9%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.

Related papers

• Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI [2.2616066835313253]
  We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations.<n>We apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials.<n>The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly.
  arXiv  Detail & Related papers  (2026-07-03T02:20:04Z)
• Evaluation Awareness Is Not One Capability: Evidence from Open Language Models [1.758143872501506]
  Safety benchmarks assume that test-condition behavior predicts deployment behavior, an assumption that fails if models detect evaluation cues and adapt.<n>This opens a gap between benchmark performance and deployment behavior.<n>We characterize this evaluation awareness through eight experiments across 37 open-weight models and seven families.
  arXiv  Detail & Related papers  (2026-06-22T16:48:43Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• CWM: Contrastive World Models for Action Feasibility Learning in Embodied Agent Pipelines [2.8681767712152197]
  A reliable action feasibility scorer is a critical bottleneck in embodied agent pipelines.<n>Existing approaches use supervised fine-tuning (SFT) to train action scorers.<n>We propose Contrastive World Model (CWM), which fine-tunes a large language model (LLM) as an action scorer.
  arXiv  Detail & Related papers  (2026-02-25T22:27:30Z)
• Capabilities Ain't All You Need: Measuring Propensities in AI [32.960519634809145]
  We introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success.<n>We find that we can measure how much the propensity is shifted and what effect this has on the tasks.<n>We obtain stronger predictive power when combining propensities and capabilities than either separately.
  arXiv  Detail & Related papers  (2026-02-20T12:40:18Z)
• A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents [4.851169906977996]
  We introduce a new benchmark comprising 40 distinct scenarios.<n>Each scenario presents a task that requires multi-step actions, and the agent's performance is tied to a specific Key Performance Indicator (KPI)<n>We observe outcome-driven constraint violations ranging from 1.3% to 71.4%, with 9 of the 12 models exhibiting misalignment rates between 30% and 50%.
  arXiv  Detail & Related papers  (2025-12-23T21:52:53Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Model Editing as a Double-Edged Sword: Steering Agent Ethical Behavior Toward Beneficence or Harm [57.00627691433355]
  We frame agent behavior steering as a model editing task, which we term Behavior Editing.<n>We introduce BehaviorBench, a benchmark grounded in psychological moral theories.<n>We demonstrate that Behavior Editing can be used to promote ethical and benevolent behavior or, conversely, to induce harmful or malicious behavior.
  arXiv  Detail & Related papers  (2025-06-25T16:51:51Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning [84.22561239481901]
  We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents. We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
  arXiv  Detail & Related papers  (2023-12-10T06:03:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.