Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles

• URL: http://arxiv.org/abs/2605.27784v1
• Date: Wed, 27 May 2026 00:09:45 GMT
• Title: Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles
• Abstract summary: LLM agents are governed by long-lived natural-collision prompt policies, but individually reasonable standing rules can interact in uninspected ways.<n>We study live intra-policy rule-conflict diagnosis: finding rule pairs inside a single prompt policy that can co-govern a realistic state.<n>We introduce WIRE, a Witnessed Intra-policy Rule Evaluation pipeline.
• Score: 15.65681557926802
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: LLM agents are governed by long-lived natural-language prompt policies, but individually reasonable standing rules can interact in uninspected ways. We study live intra-policy rule-conflict diagnosis: finding rule pairs inside a single prompt policy that can co-govern a realistic state, and measuring how models resolve that pressure in responses or tool actions. We introduce WIRE, a Witnessed Intra-policy Rule Evaluation pipeline. WIRE extracts source-grounded rules, encodes them as PyRule clauses, uses satisfiability checks to retain same-surface hard-collision candidates, realizes those candidates as concrete co-governance witnesses, and judges model outputs against the original source-rule text. Across six public prompt policies, WIRE extracts 276 source rules and 560 atomic clauses, classifies 30,944 within-policy clause-pair comparisons, retains 170 encoded hard-collision candidate source-rule pairs, and realizes them as 1,402 concrete witnesses. In policy-only evaluation, these witnesses yield 13,335 post- generation trials where both source rules govern and both compliance labels are judgeable. Only 35.4% fall in joint compliance; 64.6% violate at least one governed source rule. These profiles are conditional diagnostics for WIRE-selected candidates, not deployment-frequency or causal excess failure estimates, but they reveal distinct policy, model, and tool-action resolution patterns.

Related papers

• PolicyMem: Geometric Policy Memory for LLM Governance [72.04962480384322]
  We introduce PolicyMem, a geometric policy memory that externalizes natural-language policies as reusable geometric memory objects.<n>A memory writer compiles natural-language policies into policy memory slots, and query-response pairs read the policy memory through projection energy.<n>The resulting policy-evidence profile directly mediates the safety verdict and is reused for policy attribution and post-intervention verification.
  arXiv  Detail & Related papers  (2026-09-12T06:07:43Z)
• Grounded Normative Rule Generation with Structured Search [47.28595196084396]
  We formalize the problem as Grounded Normative Rule Synthesis (GNRS)<n>We introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG)<n>We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharterBench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses.
  arXiv  Detail & Related papers  (2026-08-23T05:43:42Z)
• Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring [0.0]
  Current approaches to AI compliance treat conformity as a binary, audit-time verdict.<n>We argue that this compliance fiction is structurally ill-suited to the requirements of the EU AI Act.<n>We introduce governance from metrics, a principle whereby regulatory compliance is derived as a continuous signal from behavioural observability.
  arXiv  Detail & Related papers  (2026-05-23T21:21:33Z)
• RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderation [4.531887731522564]
  RuleSafe-VL is a benchmark for rule-conditioned decision reasoning in content moderation.<n>It formalizes 93 atomic rules and 92 typed rule relations, yielding 2,166 context-sensitive image-text cases.<n>It identifies activated rules, recover rule interactions, judge decision sufficiency, and resolve outcomes once missing context is supplied.
  arXiv  Detail & Related papers  (2026-05-08T14:05:00Z)
• Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI [0.6138671548064355]
  We formalize evaluation as policy-grounded correctness and introduce the Defensibility Index (DI) and Ambiguity Index (AI)<n>We validate the framework on 193,000+ Reddit moderation decisions across multiple communities and evaluation cohorts.
  arXiv  Detail & Related papers  (2026-04-22T18:05:29Z)
• Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies [0.0]
  LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect.<n>Existing benchmarks evaluate models against external standards but do not measure whether models understand and enforce their own boundaries.<n>We introduce the Symbolic-Neural Consistency Audit (SNCA), a framework that extracts a model's self-stated safety rules via structured prompts.
  arXiv  Detail & Related papers  (2026-04-10T10:18:45Z)
• Executable Governance for AI: Translating Policies into Rules Using LLMs [1.388831902854619]
  Policy-to-Tests (P2T) is a framework that converts natural policy documents into normalized, machine-readable rules.<n>To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards.<n>These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set.
  arXiv  Detail & Related papers  (2025-12-04T03:11:54Z)
• Few-shot Policy (de)composition in Conversational Question Answering [54.259440408606515]
  We propose a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting.<n>We show that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies.<n>We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning.
  arXiv  Detail & Related papers  (2025-01-20T08:40:15Z)
• Statistical Analysis of Policy Space Compression Problem [54.1754937830779]
  Policy search methods are crucial in reinforcement learning, offering a framework to address continuous state-action and partially observable problems. Reducing the policy space through policy compression emerges as a powerful, reward-free approach to accelerate the learning process. This technique condenses the policy space into a smaller, representative set while maintaining most of the original effectiveness.
  arXiv  Detail & Related papers  (2024-11-15T02:46:55Z)
• Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts [61.929388479847525]
  This paper focuses on the problem of detecting and reacting to changes in the distribution of a sensorimotor controller's observables. The key idea is the design of switching policies that can take conformal quantiles as input. We show how to design such policies by using conformal quantiles to switch between base policies with different characteristics.
  arXiv  Detail & Related papers  (2023-11-02T17:59:30Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)
• Policy Dispersion in Non-Markovian Environment [53.05904889617441]
  This paper tries to learn the diverse policies from the history of state-action pairs under a non-Markovian environment. We first adopt a transformer-based method to learn policy embeddings. Then, we stack the policy embeddings to construct a dispersion matrix to induce a set of diverse policies.
  arXiv  Detail & Related papers  (2023-02-28T11:58:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.