Monitoring Emergent Reward Hacking During Generation via Internal Activations

• URL: http://arxiv.org/abs/2603.04069v1
• Date: Wed, 04 Mar 2026 13:44:24 GMT
• Title: Monitoring Emergent Reward Hacking During Generation via Internal Activations
• Abstract summary: Fine-tuned language models can exhibit reward-hacking behavior arising from emergent misalignment.<n>We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response.
• Score: 1.522424334864671
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuned large language models can exhibit reward-hacking behavior arising from emergent misalignment, which is difficult to detect from final outputs alone. While prior work has studied reward hacking at the level of completed responses, it remains unclear whether such behavior can be identified during generation. We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response. Our method trains sparse autoencoders on residual stream activations and applies lightweight linear classifiers to produce token-level estimates of reward-hacking activity. Across multiple model families and fine-tuning mixtures, we find that internal activation patterns reliably distinguish reward-hacking from benign behavior, generalize to unseen mixed-policy adapters, and exhibit model-dependent temporal structure during chain-of-thought reasoning. Notably, reward-hacking signals often emerge early, persist throughout reasoning, and can be amplified by increased test-time compute in the form of chain-of-thought prompting under weakly specified reward objectives. These results suggest that internal activation monitoring provides a complementary and earlier signal of emergent misalignment than output-based evaluation, supporting more robust post-deployment safety monitoring for fine-tuned language models.

Related papers

• SpecGuard: Inference-Time Backdoor Detection For Free [11.670265492085328]
  SpecGuard is an inference-time backdoor detector that repurposes speculative decoding at zero added model-computation cost.<n>It reliably detects triggered behavior, including stealthy cases where input-level filters are blind.
  arXiv  Detail & Related papers  (2026-09-10T16:51:59Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness [0.16921396880325779]
  We test whether activation monitors trained on a base model remain reliable after routine model updates.<n>We find a sharp split: quantization-style updates largely preserve frozen probe performance, while fine-tuning-style updates frequently make probes stale.<n>We also show that degradation is predictable from pre-deployment features, enabling revalidation budgets to be triaged toward the monitors most likely to fail.
  arXiv  Detail & Related papers  (2026-06-14T19:07:22Z)
• From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents [1.5268922363885409]
  Language-model agents act through repeated cycles of observation, reasoning, and action selection.<n>We study reward-hacking monitors in ReAct-style agents acting in Gameable ALFWorld and WebShop.
  arXiv  Detail & Related papers  (2026-06-04T14:34:31Z)
• From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism [30.96634743446629]
  We show that caution is a simple, computationally efficient approach that substantially mitigates reward hacking in BoN sampling.<n>We also provide a theoretical analysis in a simplified linear setting, which shows that caution provably improves over the standard BoN approach.
  arXiv  Detail & Related papers  (2026-04-06T12:58:11Z)
• Test-Time Attention Purification for Backdoored Large Vision Language Models [23.890959327899925]
  We provide a new mechanistic understanding of backdoor behaviors in large vision-language models (LVLMs)<n>We propose CleanSight, a training-free, plug-and-play defense that operates purely at test time.<n>CleanSight significantly outperforms existing pixel-based purification defenses across diverse datasets and backdoor attack types.
  arXiv  Detail & Related papers  (2026-03-13T13:45:06Z)
• How Far Can Unsupervised RLVR Scale LLM Training? [57.44753418846446]
  Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck.<n>Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.<n>We classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework.
  arXiv  Detail & Related papers  (2026-03-09T17:38:11Z)
• PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding [85.22047087898311]
  We introduce Polarity-Prompt Contrastive Decoding (PromptCD), a test-time behavior control method that generalizes contrastive decoding to broader enhancement settings.<n>PromptCD constructs paired positive and negative guiding prompts for a target behavior and contrasts model responses to reinforce desirable outcomes.<n>Experiments on the "3H" alignment objectives demonstrate consistent and substantial improvements, indicating that post-trained models can achieve meaningful self-enhancement purely at test time.
  arXiv  Detail & Related papers  (2026-02-24T08:56:52Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Semantics-Preserving Evasion of LLM Vulnerability Detectors [14.476903104601154]
  LLM-based vulnerability detectors are increasingly deployed in security-critical code review.<n>We evaluate detection-time integrity under a semantics-preserving threat model.<n>We introduce a metric of joint robustness across different attack methods/carriers.
  arXiv  Detail & Related papers  (2026-01-30T20:54:27Z)
• Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks [1.4291137439893342]
  Chain-of-thought (CoT) reasoning provides a significant performance uplift to LLMs.<n>CoT is also a powerful tool for monitoring the behaviours of these agents.<n>We show that optimisation pressures on the CoT may cause the model to obfuscate reasoning traces, losing this beneficial property.
  arXiv  Detail & Related papers  (2026-01-30T15:34:14Z)
• From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models [77.04403907729738]
  This survey charts the evolution of uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.<n>We demonstrate how uncertainty is leveraged as an active control signal across three frontiers.<n>This survey argues that mastering the new trend of uncertainty is essential for building the next generation of scalable, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-22T06:21:31Z)
• INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Models [2.509305596181814]
  Recent Vision-Language-Action (VLA) models show strong generalization capabilities, yet they lack introspective mechanisms for anticipating failures and requesting help from a human supervisor.<n>We present textbfINSIGHT, a learning framework for leveraging token-level uncertainty signals to predict when a VLA should request help.
  arXiv  Detail & Related papers  (2025-10-01T19:22:48Z)
• A Practical Adversarial Attack against Sequence-based Deep Learning Malware Classifiers [14.522205772331723]
  We propose an adversarial attack approach based on Deep Q-Network and a backtracking search strategy.<n>We utilize a novel transformation approach that maps modifications back to the source code, thereby avoiding the need to directly modify the behavior log sequences.<n>Our approach is practical and can generate adversarial samples while maintaining the functionality of the modified software.
  arXiv  Detail & Related papers  (2025-09-15T12:22:26Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.