Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

• URL: http://arxiv.org/abs/2609.14400v1
• Date: Sun, 13 Sep 2026 10:00:13 GMT
• Title: Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
• Abstract summary: Benchmark developers should audit policies before collecting gold annotations.<n>A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness.
• Score: 2.0305676256390934
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two $τ^2$-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.

Related papers

• STAGE: Stateful Translation to Agentic Graph Execution with Policy-Scoped Context and Deterministic Control [0.29360071145551064]
  We present STAGE, a framework that confines model judgment to policy-scoped nodes while placing procedural control in deterministic code.<n>We evaluate STAGE on three public policy-following benchmarks and Smart Dispute, a proprietary banking benchmark.
  arXiv  Detail & Related papers  (2026-08-23T18:17:09Z)
• Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments [0.0]
  We evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions.<n>We characterize how agents behave when policy blocks their actions and decompose the failures hardening.<n>We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents.
  arXiv  Detail & Related papers  (2026-08-02T14:26:12Z)
• DRIP-R: A Benchmark for Decision-Making and Reasoning Under Real-World Policy Ambiguity in the Retail Domain [21.9900052246381]
  We introduce DRIPR, a benchmark that exploits realworld retail policy construct scenarios in which no single correct resolution exists.<n>DRIPR comprises a curated set of policyambiguous return scenarios paired with a realistic customer personas, a full conversational simulation tool-calling capabilities and a multi-judge evaluation framework.<n>Our experiments show that frontier models fundamentally disagree on identical policy ambiguities, confirming that ambiguity poses a genuine and systematic challenge to LLM decisionmaking.
  arXiv  Detail & Related papers  (2026-05-08T13:10:49Z)
• PolicyBank: Evolving Policy Understanding for LLM Agents [51.86716874651299]
  PolicyBank maintains structured, tool-level policy insights and iteratively refines them.<n>PolicyBank closes up to 82% of the gap toward a human oracle.
  arXiv  Detail & Related papers  (2026-04-16T20:29:30Z)
• Near-Miss: Latent Policy Failure Detection in Agentic Workflows [9.719140082591956]
  We introduce a novel metric for detecting latent policy failures in agent conversations traces.<n>Our results show that latent failures occur in 8-17% of trajectories involving mutating tool calls.
  arXiv  Detail & Related papers  (2026-03-31T12:26:35Z)
• Verification of Robust Properties for Access Control Policies [51.736723807086385]
  Existing methods for verifying access control policies require the policy to be complete and fully determined before verification can proceed.<n>We introduce robust property verification: the problem of determining what a policy's structure commits it to regardless of how pending decisions are resolved and regardless of subsequent extension.<n>We show that despite quantifying universally over all possible policy extensions the judgment reduces to proof search in a second-order logic programming language.
  arXiv  Detail & Related papers  (2026-03-13T17:14:38Z)
• Towards Policy-Compliant Agents: Learning Efficient Guardrails For Policy Violation Detection [25.53228630260007]
  PolicyGuardBench is a benchmark of about 60k examples for detecting policy violations in agent trajectories.<n>PolicyGuard-4B is a lightweight guardrail model that delivers strong detection accuracy across all tasks.<n>Together, PolicyGuardBench and PolicyGuard-4B provide the first comprehensive framework for studying policy compliance in web agent trajectories.
  arXiv  Detail & Related papers  (2025-10-03T20:03:19Z)
• Best-Effort Policies for Robust Markov Decision Processes [69.60742680559788]
  We study the common generalization of Markov decision processes (MDPs) with sets of transition probabilities, known as robust MDPs (RMDPs)<n>We call such a policy an optimal robust best-effort (ORBE) policy.<n>We prove that ORBE policies always exist, characterize their structure, and present an algorithm to compute them with a small overhead compared to standard robust value iteration.
  arXiv  Detail & Related papers  (2025-08-11T09:18:34Z)
• How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation [17.638831964639834]
  Behavior cloning policies are increasingly successful at solving complex tasks by learning from human demonstrations. We present a framework that provides a tight lower-bound on robot performance in an arbitrary environment. In experiments we evaluate policies for visuomotor manipulation in both simulation and hardware.
  arXiv  Detail & Related papers  (2024-05-08T22:00:35Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)
• Supervised Off-Policy Ranking [145.3039527243585]
  Off-policy evaluation (OPE) leverages data generated by other policies to evaluate a target policy. We propose supervised off-policy ranking that learns a policy scoring model by correctly ranking training policies with known performance. Our method outperforms strong baseline OPE methods in terms of both rank correlation and performance gap between the truly best and the best of the ranked top three policies.
  arXiv  Detail & Related papers  (2021-07-03T07:01:23Z)
• Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning [80.42316902296832]
  We study the efficient off-policy evaluation of natural policies, which are defined in terms of deviations from the behavior policy. This is a departure from the literature on off-policy evaluation where most work consider the evaluation of explicitly specified policies.
  arXiv  Detail & Related papers  (2020-06-06T15:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.