Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

• URL: http://arxiv.org/abs/2605.12673v1
• Date: Tue, 12 May 2026 19:22:45 GMT
• Title: Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
• Abstract summary: BenchJack is a system that drives coding agents to audit benchmarks and identify possible reward-hacking exploits.<n>We apply BenchJack to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations.<n> BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task.
• Score: 51.54835866517547
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent benchmarks have become the de facto measure of frontier AI competence, guiding model selection, investment, and deployment. However, reward hacking, where agents maximize a score without performing the intended task, emerges spontaneously in frontier models without overfitting. We argue that benchmarks must be secure by design. From past incidents of reward hacks, we derive a taxonomy of eight recurring flaw patterns and compile them into the Agent-Eval Checklist for benchmark designers. We condense the insights into BenchJack, an automated red-teaming system that drives coding agents to audit benchmarks and identify possible reward-hacking exploits in a clairvoyant manner. Moreover, we extend BenchJack to an iterative generative-adversarial pipeline that discovers new flaws and patches them iteratively to improve benchmark robustness. We apply BenchJack to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations. BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task, surfacing 219 distinct flaws across the eight classes. Moreover, BenchJack's extended pipeline reduces the hackable-task ratio from near 100% to under 10% on four benchmarks without fatal design flaws, fully patching WebArena and OSWorld within three iterations. Our results show that evaluation pipelines have not internalized an adversarial mindset, and that proactive auditing could help close the security gap for the fast-paced benchmarking space.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops [30.90132709192538]
  Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking.<n>We audit 1,968 tasks across five terminal-agent benchmarks and find 323 (16%) hackable by frontier models given only the task description.<n>We introduce the hacker-fixer loop, a method for building exploit-resistant verifiers without per-task manual patching.
  arXiv  Detail & Related papers  (2026-06-08T03:00:56Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use [0.016201337438609787]
  Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems.<n>We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities.<n>We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek.
  arXiv  Detail & Related papers  (2026-05-03T07:10:42Z)
• BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks [26.58983143152204]
  BenchGuard is the first automated auditing framework for task-oriented, execution-based agent benchmarks.<n>It identifies 12 author-confirmed issues in ScienceAgentBench and 83.3% of expert-identified issues on the BIXBench Verified-50 subset.<n>A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review.
  arXiv  Detail & Related papers  (2026-04-27T19:51:25Z)
• ClawBench: Can AI Agents Complete Everyday Online Tasks? [50.958690494341106]
  ClawBench is an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work.<n>ClawBench operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction.<n>A lightweight interception layer captures and blocks only the final submission request, ensuring safe evaluation without real-world side effects.
  arXiv  Detail & Related papers  (2026-04-09T17:57:13Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards [44.03496012544118]
  OS-Themis is a scalable and accurate multi-agent critic framework.<n>It decomposes trajectories into verifiable milestones to isolate critical evidence for decision making.<n>It employs a review mechanism to strictly audit the evidence chain before making the final verdict.<n>Experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training.
  arXiv  Detail & Related papers  (2026-03-19T17:47:47Z)
• An Effective and Cost-Efficient Agentic Framework for Ethereum Smart Contract Auditing [8.735899453872966]
  Heimdallr is an automated auditing agent designed to overcome hurdles through four core innovations.<n>It minimizes context overhead while preserving essential business logic.<n>It then employs reasoning to detect complex vulnerabilities and automatically chain functional exploits.
  arXiv  Detail & Related papers  (2026-01-25T13:28:37Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks? [58.48689960350828]
  We show that a simple, modular and model-agnostic defense operating at the agent--tool interface achieves perfect security with high utility.<n>We employ a defense based on two firewalls: a Tool-Input Firewall (Minimizer) and a Tool-Output Firewall (Sanitizer)
  arXiv  Detail & Related papers  (2025-10-06T18:09:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.