SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents

• URL: http://arxiv.org/abs/2605.28122v1
• Date: Wed, 27 May 2026 08:14:07 GMT
• Title: SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents
• Abstract summary: A coding agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes.<n>We call this overeager behavior: the prompt is not adversarial and the run succeeds, yet an out-of-scope step can leak credentials or delete files.<n>Existing benchmarks miss it: task-completion suites credit any finished run, jailbreak suites probe adversarial prompts, and the one prior overeager benchmark applies a single fixed prompt to every agent-model pair.<n>We present SNARE, a pipeline that composes benign scenarios from reusable scope
• Score: 40.270213696031625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A coding agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes. We call this overeager behavior: the prompt is not adversarial and the run succeeds, yet an out-of-scope step can leak credentials or delete files. Existing benchmarks miss it: task-completion suites credit any finished run, jailbreak suites probe adversarial prompts, and the one prior overeager benchmark applies a single fixed prompt set to every agent-model pair, leaving its easiest and most resistant pairs under-measured. We present SNARE (Synthesizing Non-adversarial scenarios for Adaptive Reward-guided Elicitation), a pipeline that composes benign scenarios from reusable scope and trap fragments, scores each run with a judge-free oracle flagging trap-pattern matches and unsolicited file additions or deletions, and uses Thompson sampling to steer each pair's run budget toward the scenarios that most often trigger it. Instantiating it over 24 overeager archetypes yields OverEager, which we run across a 4x5 matrix of four coding agents and five base models. Across 10,000 benign runs, 19.51% trigger overeager behavior, with per-pair rates spanning 11.9x. This variation is driven by the agent framework, not the model: the framework accounts for 56% of it against the model's 21%, so any single-framework or single-model evaluation undercounts the matrix by about a fifth.

Related papers

• AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents [2.8000808592516293]
  AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions.<n>It reads only the recorded execution trace and does not interfere with how the agent runs.<n>We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks.
  arXiv  Detail & Related papers  (2026-09-09T08:29:16Z)
• Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep [0.3384279376065155]
  This pilot studies that choice on bounded cross-border VAT determination with reverse charge.<n>We hold the activity surface fixed (subtasks, tools, I/O schemas, validation checks, orchestrator, base model, and merge policy)<n>The program spans 4,400 runs: a 40-case, five-repeat main sweep, matched-token arms separating prompt-budget from agent-count effects, and three failure-injection arms.
  arXiv  Detail & Related papers  (2026-08-24T15:40:15Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions [9.297494684604763]
  EvoCode-Bench is a benchmark of 26 stateful coding tasks and 227 evaluated rounds.<n>Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior.<n>We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state.
  arXiv  Detail & Related papers  (2026-05-22T18:17:28Z)
• AgentAtlas: Beyond Outcome Leaderboards for LLM Agents [0.025718125188898048]
  AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol.<n>The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models.
  arXiv  Detail & Related papers  (2026-05-19T22:05:12Z)
• Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks [40.270213696031625]
  OverEager-Gen is a benchmark dedicated to overeager behavior on benign tasks.<n>On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1%.<n>OverEager-Gen therefore certifies each scenario's discriminative power before admission.
  arXiv  Detail & Related papers  (2026-05-18T16:00:41Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks [1.1435139523855764]
  This paper presents a novel multi-agent defense framework to detect and neutralize prompt injection attacks in real-time.<n>We evaluate our approach using two distinct architectures: a sequential chain-of-agents pipeline and a hierarchical coordinator-based system.
  arXiv  Detail & Related papers  (2025-09-16T19:11:28Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.