Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

• URL: http://arxiv.org/abs/2605.02964v1
• Date: Sun, 03 May 2026 07:10:42 GMT
• Title: Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
• Abstract summary: Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems.<n>We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities.<n>We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek.
• Score: 0.016201337438609787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.

Related papers

• BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks [0.20524609401792396]
  LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight.<n>Previous work has documented reward hacking, bringing into question the validity of produced research and the broader safety case for AI R&D.<n>We introduce BAITBENCH, a suite of three synthetic tabular ML tasks that each contain a shortcut that allows agents to inflate the public test score but fail on a hidden test set.<n>Across seven frontier agents scored by our two-stage judge pipeline, 57.1% of runs exhibit reward hacking, with five of seven above 50%.
  arXiv  Detail & Related papers  (2026-08-31T12:59:33Z)
• Can escalation channels redirect reward hacking toward defect disclosure? [0.0]
  coding agents may reward-hack: hardcoding outputs or editing test files to pass tests they cannot legitimately satisfy.<n>We evaluate escalation channels, structured reporting tools available to the agent at the point of conflict.<n>A $2 times 2$ factorial separates the contributions of an escalation tool, a standalone anti-reward-hacking policy, and their combination.
  arXiv  Detail & Related papers  (2026-08-29T22:39:45Z)
• ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment [66.45065414461274]
  Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer.<n>We propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents.<n>ABC converts sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions.
  arXiv  Detail & Related papers  (2026-08-05T17:41:31Z)
• Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack [51.54835866517547]
  BenchJack is a system that drives coding agents to audit benchmarks and identify possible reward-hacking exploits.<n>We apply BenchJack to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations.<n> BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task.
  arXiv  Detail & Related papers  (2026-05-12T19:22:45Z)
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation [98.38575149237442]
  VLAA-GUI is a modular GUI framework built around three integrated components.<n>A mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step.<n>A mandatory Loop Breaker provides multi-tier switching interaction mode after repeated failures.
  arXiv  Detail & Related papers  (2026-04-23T07:42:37Z)
• Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories [30.901327091925385]
  We release Terminal Wrench, a subset of 331 terminal-agent benchmark environments.<n>The data set includes 3,632 hack trajectories and 2,352 legitimate baseline trajectories.<n>The exploits range from simple output spoofing to stack-frame introspection.
  arXiv  Detail & Related papers  (2026-04-19T20:04:02Z)
• Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities [0.0]
  LLM agents with tool access can exploit security vulnerabilities.<n>What is not known is which features of a system prompt trigger this behaviour, and which do not.<n>We present a systematic taxonomy based on 10,000 trials across seven models, 37 prompt conditions, and 12 hypothesized attack dimensions.
  arXiv  Detail & Related papers  (2026-04-06T09:44:34Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking [69.06218054848803]
  We propose Adrial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game.<n>ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations.<n>ARA achieves the best alignment-utility tradeoff among all baselines.
  arXiv  Detail & Related papers  (2026-02-02T07:34:57Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Detecting and Mitigating Reward Hacking in Reinforcement Learning Systems: A Comprehensive Empirical Study [2.1797343876622097]
  Reward hacking in Reinforcement Learning systems poses a critical threat to the deployment of autonomous agents.<n>This paper presents a large-scale empirical study of reward hacking across diverse RL environments and algorithms.
  arXiv  Detail & Related papers  (2025-07-08T03:00:02Z)
• Know the Ropes: A Heuristic Strategy for LLM-based Multi-Agent System Design [27.246783209927464]
  Know-The-Ropes is a framework that converts domain priors into an algorithmic blueprint hierarchy.<n>Three GPT-4o-mini agents raise accuracy from 3% zero-shot to 95% on size-5 instances after patching a single bottleneck agent.
  arXiv  Detail & Related papers  (2025-05-22T17:52:33Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.