Policy-Invisible Violations in LLM-Based Agents

• URL: http://arxiv.org/abs/2604.12177v1
• Date: Tue, 14 Apr 2026 01:15:15 GMT
• Title: Policy-Invisible Violations in LLM-Based Agents
• Abstract summary: We call this failure mode policy-invisible violations: cases in which compliance depends on entity attributes, contextual state, or session history absent from the agent's visible context.<n>We present PhantomPolicy, a benchmark spanning eight violation categories with balanced violation and safe-control cases.<n>We introduce Sentinel, an enforcement framework based on counterfactual graph simulation.
• Score: 10.660248467840821
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents can execute actions that are syntactically valid, user-sanctioned, and semantically appropriate, yet still violate organizational policy because the facts needed for correct policy judgment are hidden at decision time. We call this failure mode policy-invisible violations: cases in which compliance depends on entity attributes, contextual state, or session history absent from the agent's visible context. We present PhantomPolicy, a benchmark spanning eight violation categories with balanced violation and safe-control cases, in which all tool responses contain clean business data without policy metadata. We manually review all 600 model traces produced by five frontier models and evaluate them using human-reviewed trace labels. Manual review changes 32 labels (5.3%) relative to the original case-level annotations, confirming the need for trace-level human review. To demonstrate what world-state-grounded enforcement can achieve under favorable conditions, we introduce Sentinel, an enforcement framework based on counterfactual graph simulation. Sentinel treats every agent action as a proposed mutation to an organizational knowledge graph, performs speculative execution to materialize the post-action world state, and verifies graph-structural invariants to decide Allow/Block/Clarify. Against human-reviewed trace labels, Sentinel substantially outperforms a content-only DLP baseline (68.8% vs. 93.0% accuracy) while maintaining high precision, though it still leaves room for improvement on certain violation categories. These results demonstrate what becomes achievable once policy-relevant world state is made available to the enforcement layer.

Related papers

• Grounded Normative Rule Generation with Structured Search [47.28595196084396]
  We formalize the problem as Grounded Normative Rule Synthesis (GNRS)<n>We introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG)<n>We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharterBench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses.
  arXiv  Detail & Related papers  (2026-08-23T05:43:42Z)
• FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphs [11.447078130049512]
  FAVA is a permission-carrying authorization framework for agent execution.<n>We evaluate FAVA across OpenAgentSafety, OctoBench, and ActPlane scenarios.
  arXiv  Detail & Related papers  (2026-07-29T09:41:07Z)
• Agentic Permissions Policy Algebra for Taint Confinement in LLM Agents [0.0]
  Traditional taint tracking permanently taints an agent's context upon reading unvetted data.<n>We present APPA, an IFC framework that resolves this usability bottleneck through engine-managed context branching.<n>We evaluate APPA on a multi-turn tool-chaining benchmark across four models.
  arXiv  Detail & Related papers  (2026-07-27T16:19:45Z)
• ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents [4.018837636907658]
  We introduce a sandboxed, trace-based benchmark that measures benchmark-defined endpoint policy compliance.<n>We find that aggregate logged-spread rankings change with evidence-stage composition and denominator choice.<n>These results show that a terminal policy label is not a sufficient statistic for operational post-exposure containment.
  arXiv  Detail & Related papers  (2026-07-27T04:51:20Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• SentinelAgent: Intent-Verified Delegation Chains for Securing Federal Multi-Agent AI Systems [0.0]
  This paper introduces SentinelAgent, a formal framework for verifiable delegation chains in federal multi-agent AI systems.<n>The Delegation Chain Calculus (DCC) defines seven properties - six deterministic (authority narrowing, policy preservation, forensic reconstructibility, cascade containment, scope-action conformance, output schema conformance) and one probabilistic (intent preservation)<n>The Intent-Preserving Delegation Protocol (I PDP) enforces all seven properties at runtime through a non-LLM Delegation Authority Service.
  arXiv  Detail & Related papers  (2026-04-03T06:25:18Z)
• Near-Miss: Latent Policy Failure Detection in Agentic Workflows [9.719140082591956]
  We introduce a novel metric for detecting latent policy failures in agent conversations traces.<n>Our results show that latent failures occur in 8-17% of trajectories involving mutating tool calls.
  arXiv  Detail & Related papers  (2026-03-31T12:26:35Z)
• TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models [59.13964209628383]
  Vision-Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors.<n>We propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies.<n>We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
  arXiv  Detail & Related papers  (2026-03-25T17:56:32Z)
• Agentproof: Static Verification of Agent Workflow Graphs [0.0]
  Agent frameworks increasingly encode tool-using behavior as explicit workflow graphs, yet safety enforcement remains a runtime concern.<n>This paper presents Agentproof, a system that automatically extracts a unified abstract graph model from four major agent frameworks.<n>Unlike general-purpose model checkers, Agentproof requires no manual modeling.
  arXiv  Detail & Related papers  (2026-03-20T13:56:20Z)
• Agent Control Protocol: Admission Control for Agent Actions [0.4929694290403903]
  Agent Control Protocol (ACP) is a formal specification for admission control governance of autonomous agents in B2B institutional environments.<n>ACP defines cryptographic identity, capability-based authorization, deterministic risk evaluation, chained delegation, and cryptographically-chained auditing.<n>ACP operates on top of RBAC and Zero Trust, addressing what neither model solves.
  arXiv  Detail & Related papers  (2026-03-19T12:28:28Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs [62.116710797795314]
  Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance.<n>We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context.
  arXiv  Detail & Related papers  (2025-11-18T21:51:23Z)
• Policy-Aware Generative AI for Safe, Auditable Data Access Governance [0.0]
  We present a policy aware controller that uses a large language model (LLM) to interpret natural language requests against written policies and metadata, not raw data.<n>The system, implemented with Google Gemini2.0 Flash, executes a six-stage reasoning framework with early hard policy gates and deny by default.
  arXiv  Detail & Related papers  (2025-10-27T16:10:55Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.