Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies

• URL: http://arxiv.org/abs/2604.09189v1
• Date: Fri, 10 Apr 2026 10:18:45 GMT
• Title: Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies
• Abstract summary: LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect.<n>Existing benchmarks evaluate models against external standards but do not measure whether models understand and enforce their own boundaries.<n>We introduce the Symbolic-Neural Consistency Audit (SNCA), a framework that extracts a model's self-stated safety rules via structured prompts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect. Existing benchmarks evaluate models against external standards but do not measure whether models understand and enforce their own stated boundaries. We introduce the Symbolic-Neural Consistency Audit (SNCA), a framework that (1) extracts a model's self-stated safety rules via structured prompts, (2) formalizes them as typed predicates (Absolute, Conditional, Adaptive), and (3) measures behavioral compliance via deterministic comparison against harm benchmarks. Evaluating four frontier models across 45 harm categories and 47,496 observations reveals systematic gaps between stated policy and observed behavior: models claiming absolute refusal frequently comply with harmful prompts, reasoning models achieve the highest self-consistency but fail to articulate policies for 29% of categories, and cross-model agreement on rule types is remarkably low (11%). These results demonstrate that the gap between what LLMs say and what they do is measurable and architecture-dependent, motivating reflexive consistency audits as a complement to behavioral benchmarks.

Related papers

• EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents [74.54929751174289]
  EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain.<n>It reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost.<n>It also achieves the best score on Agent-SafetyBench for every victim.
  arXiv  Detail & Related papers  (2026-09-05T05:56:41Z)
• Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment [0.0]
  Under value pluralism there is no correct target, but a shared prerequisite is that the system's behavior expresses a coherent policy.<n>We introduce four structural conditions for such coherent policies.<n>We show no model expresses a coherent policy across the three deployments.
  arXiv  Detail & Related papers  (2026-09-04T11:56:44Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Auditing Proprietary Alignment in Large Language Models: A Comparative Framework Without a Ground-Truth Standard [19.605143530514777]
  Large language models (LLMs) are increasingly released and deployed through opaque development and deployment pipelines.<n>Various models have been reported to generate responses reflecting proprietary rules and organizational interests.<n>We propose a statistical framework for detecting proprietary alignment in black-box language models via comparative behavioral analysis.
  arXiv  Detail & Related papers  (2026-06-07T00:20:55Z)
• ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails [57.25411733152009]
  Reasoning-based guardrails improve safety moderation by generating explicit rationales before issuing final decisions.<n>A model may recognize a harmful intent in its reasoning but still predict a safe label, or issue an unsafe decision without policy-grounded justification.<n>We propose ConsisGuard, a consistency-aware framework for reasoning-based guardrails.
  arXiv  Detail & Related papers  (2026-05-29T09:42:08Z)
• RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderation [4.531887731522564]
  RuleSafe-VL is a benchmark for rule-conditioned decision reasoning in content moderation.<n>It formalizes 93 atomic rules and 92 typed rule relations, yielding 2,166 context-sensitive image-text cases.<n>It identifies activated rules, recover rule interactions, judge decision sufficiency, and resolve outcomes once missing context is supplied.
  arXiv  Detail & Related papers  (2026-05-08T14:05:00Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluation [0.0]
  We study alignment evaluation through the lens of statistical identifiability under partial observability.<n>We formalize the Alignment Verifiability Problem and introduce Normative Indistinguishability.<n>Our results show that behavioral benchmarks provide necessary but insufficient evidence for latent alignment under evaluation awareness.
  arXiv  Detail & Related papers  (2026-02-05T13:40:56Z)
• COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs [19.742967013586927]
  We present the first systematic framework for evaluating whether large language models comply with organizational allowlist and denylist policies.<n>We show that models reliably handle legitimate requests but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations.
  arXiv  Detail & Related papers  (2026-01-05T06:57:45Z)
• SSR: Socratic Self-Refine for Large Language Model Reasoning [78.62319252287938]
  Socratic Self-Refine (SSR) is a novel framework for fine-grained evaluation and precise refinement of Large Language Models (LLMs)<n>Our proposed SSR decomposes model responses into verifiable (sub-question, sub-answer) pairs, enabling step-level confidence estimation.<n> Empirical results across five reasoning benchmarks and three LLMs show that SSR consistently outperforms state-of-the-art iterative self-refinement baselines.
  arXiv  Detail & Related papers  (2025-11-13T18:47:07Z)
• VAL-Bench: Measuring Value Alignment in Language Models [10.745372809345412]
  Large language models (LLMs) are increasingly used for tasks where outputs shape human decisions.<n>Existing benchmarks mostly track refusals or predefined safety violations but do not reveal whether a model upholds a coherent value system.<n>We introduce the Value ALignment Benchmark (VAL-Bench), which evaluates whether models maintain a stable value stance across paired that frame opposing sides of public debates.
  arXiv  Detail & Related papers  (2025-10-06T23:55:48Z)
• SpecEval: Evaluating Model Adherence to Behavior Specifications [63.13000010340958]
  We introduce an automated framework that audits models against their providers specifications.<n>Our central focus is on three way consistency between a provider specification, its model outputs, and its own models as judges.<n>We apply our framework to 16 models from six developers across more than 100 behavioral statements, finding systematic inconsistencies including compliance gaps of up to 20 percent across providers.
  arXiv  Detail & Related papers  (2025-09-02T16:18:40Z)
• Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities [49.09703018511403]
  Evaluations of large language model (LLM) risks and capabilities are increasingly being incorporated into AI risk management and governance frameworks.<n>Currently, most risk evaluations are conducted by designing inputs that elicit harmful behaviors from the system.<n>We propose evaluating LLMs with model tampering attacks which allow for modifications to latent activations or weights.
  arXiv  Detail & Related papers  (2025-02-03T18:59:16Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.