From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents

• URL: http://arxiv.org/abs/2606.06223v1
• Date: Thu, 04 Jun 2026 14:34:31 GMT
• Title: From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
• Abstract summary: Language-model agents act through repeated cycles of observation, reasoning, and action selection.<n>We study reward-hacking monitors in ReAct-style agents acting in Gameable ALFWorld and WebShop.
• Score: 1.5268922363885409
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-model agents act through repeated cycles of observation, reasoning, and action selection, making safety monitoring depend on both internal model state and environment context. We study reward-hacking monitors in ReAct-style agents acting in Gameable ALFWorld and WebShop. Agents are instrumented with activation-based reward-hack scores, token-level entropy, and decision-context features. We find that adapters fine-tuned on \textit{School-of-Reward-Hacks} dataset can transfer reward-hack tendencies into agentic action selection, especially when the environment exposes proxy-reward affordances. However, mitigating such behavior cannot rely on activation dynamics alone. High reward-hack activation identifies a latent policy state, but does not necessarily imply an immediate exploit action. Across next-step prediction tasks, entropy and context-calibrated internal features improve risk estimation over reward-hack activation alone. Activation-direction steering further reduces proxy-exploit behavior in selected mixed-adapter regimes. Overall, our results support context-calibrated internal monitoring for agents: reward-hack activation identifies a latent policy state, while entropy and decision context help determine when that state becomes risky action.

Related papers

• TrojanWorld: Backdooring World-Model Agents via Imagination Steering [65.38754125569848]
  TrojanWorld is a backdoor framework for world-model agents that induces attacker-specified behavior by steering internal imagination.<n>Under trigger activation, TrojanWorld achieves a target-action deviation as low as 0.026 while retaining at least 98.8% of the corresponding clean performance.
  arXiv  Detail & Related papers  (2026-09-07T05:14:10Z)
• The Vulnerability With No CVE: Managing Persistent Gaps Between Mandate and Authority in AI Coding Agents [0.0]
  We propose the agentic posture vulnerability (APV) as a task-conditioned-management abstraction.<n>APV operationalizes existing excessive-agency, authorization, and control-composition weaknesses.
  arXiv  Detail & Related papers  (2026-08-06T11:06:06Z)
• Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization [31.04469484599396]
  We introduce Proxy Reward Internalization and Mechanistic Exploitation (PRIME)<n>PRIME is a learned capability to assess task correctness, predict acceptance, and reason about exploitable gold gaps.<n>We find that PRIME emerges in a staged sequence before sustained reward hacking.
  arXiv  Detail & Related papers  (2026-06-08T16:32:54Z)
• VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring [60.53395558502203]
  Vision-Language Embodied Safety Agent (VLESA) monitors human activities from egocentric video.<n>VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context.<n>An intent-action prediction agent is proposed to jointly infer goals and predict future actions from video.
  arXiv  Detail & Related papers  (2026-06-02T17:42:17Z)
• From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI [14.644207081930817]
  This work examines the security and safety implications of that shift across content-level, model-level, and agentic threats.<n>We analyze how attacker access requirements, system autonomy, and the scope of potential harm change as models move from generating artifacts to executing operations through tool chains and external APIs.
  arXiv  Detail & Related papers  (2026-05-15T13:53:02Z)
• OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents [74.20327254615854]
  Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks.<n>Existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval.<n>We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents.
  arXiv  Detail & Related papers  (2026-05-11T19:28:20Z)
• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation [68.53387633351484]
  We propose CORA (COnformal Risk-controlled GUI Agent), a post-policy, pre-action safeguarding framework that provides statistical guarantees on harmful executed actions.<n> CORA reformulates safety as selective action execution: we train a Guardian model to estimate action-conditional risk for each proposed step.<n>To rigorously evaluate this paradigm, we introduce Phone-Harm, a new benchmark of mobile safety violations with step-level harm labels.
  arXiv  Detail & Related papers  (2026-04-10T09:41:21Z)
• Monitoring Emergent Reward Hacking During Generation via Internal Activations [1.522424334864671]
  Fine-tuned language models can exhibit reward-hacking behavior arising from emergent misalignment.<n>We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response.
  arXiv  Detail & Related papers  (2026-03-04T13:44:24Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Active Test-time Vision-Language Navigation [60.69722522420299]
  ATENA is a test-time active learning framework that enables a practical human-robot interaction via episodic feedback on uncertain navigation outcomes.<n>In particular, ATENA learns to increase certainty in successful episodes and decrease it in failed ones, improving uncertainty calibration.<n>In addition, we propose a self-active learning strategy that enables an agent to evaluate its navigation outcomes based on confident predictions.
  arXiv  Detail & Related papers  (2025-06-07T02:24:44Z)
• Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning [6.937243101289336]
  entropy-minimizing and entropy-maximizing objectives for unsupervised reinforcement learning (RL) have been shown to be effective in different environments. We propose an agent that can adapt its objective online, depending on the entropy conditions by framing the choice as a multi-armed bandit problem. We demonstrate that such agents can learn to control entropy and exhibit emergent behaviors in both high- and low-entropy regimes.
  arXiv  Detail & Related papers  (2024-05-27T14:58:24Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.