How Well Do Models Follow Their Constitutions?

• URL: http://arxiv.org/abs/2605.24229v1
• Date: Fri, 22 May 2026 21:17:16 GMT
• Title: How Well Do Models Follow Their Constitutions?
• Abstract summary: We propose a multi-method audit pipeline that treats each lab's published specification as an auditable target.<n>We find that models follow their own lab's specification substantially better with each generation.
• Score: 11.533708129682887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Frontier AI developers now train models against long written behavioral specifications, such as Anthropic's constitution (Anthropic, 2025a) and OpenAI's Model Spec (OpenAI, 2025a), integrated into post-training via methods like character training (Anthropic, 2024) and deliberative alignment (Guan et al., 2024). These documents serve a governance function, but it is unclear how well models actually follow them under adversarial, multi-turn pressure similar to what they would face in real-world deployment. We propose a multi-method audit pipeline that treats each lab's published specification as an auditable target: it decomposes the specification into atomic testable tenets (205 for Anthropic, 197 for OpenAI), generates multi-turn adversarial scenarios with the Petri auditing agent (Anthropic, 2025b), runs a modified SURF-style rubric search (Murray et al., 2026) to catch shallow single-turn failures Petri misses, validates flagged transcripts against the relevant specification, and compares the findings against the lab's own published system card. Applying the pipeline across seven models per specification, we find that models follow their own lab's specification substantially better with each generation. On Anthropic's constitution, the Claude family falls from a 15.0% violation rate (Sonnet 4) to 2.0% (Sonnet 4.6); on OpenAI's Model Spec, the GPT family falls from 11.7% (GPT-4o) to 3.6% (GPT-5.2 medium reasoning), with the severity ceiling falling from 10/10 to 7/10. We cannot externally isolate whether these gains come from specification-specific training, broader post-training improvements, or evaluation awareness. Remaining failures cluster around operator-imposed personas under AI-identity questioning, irreversible action in agentic deployments, and fabricated quantitative claims with false precision.

Related papers

• Persistent Semantic Entities in Tool-Augmented LLM Systems [3.556355987197792]
  Tool-augmented LLM agents can harbor implicit state that persists across sessions, activates through events, and propagates across agent boundaries.<n>We formalize this as Persistent Semantic Entities (PSEs)<n>We evaluate them across 24 models from 11 families (1.5B--1T parameters)
  arXiv  Detail & Related papers  (2026-08-08T06:31:00Z)
• Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation [0.0]
  We document a failure class in frontier large language models.<n>The failure reproduces at first observation, but its empirical surface is unstable.<n>For regulated, frontier-model accuracy is a moving compliance boundary that shifts without notice.
  arXiv  Detail & Related papers  (2026-07-25T22:40:02Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolution [0.0]
  Prediction markets aggregate collective intelligence to forecast uncertain events.<n>Existing oracle systems tradeoff fast but brittle automation against accurate but costly human arbitration.<n>We evaluate whether multi-agent LLM architectures can improve oracle resolution accuracy over single-model baselines.
  arXiv  Detail & Related papers  (2026-05-29T03:44:19Z)
• Philosophical Dispositions as Behavioral Constraints for AI-Assisted Code Review: An Empirical Study [0.0]
  We present a system that constrains AI reviewer behavior through philosophical dispositions.<n>Each disposition is defined apophatically (by what it refuses to do)<n>We evaluate this system on 50 merged pull requests across 7 repositories spanning 5 programming languages.
  arXiv  Detail & Related papers  (2026-05-21T23:57:25Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort [51.56484100374058]
  Spracklen et al. showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm.<n>We measure hallucination rates across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists.<n>We identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically.
  arXiv  Detail & Related papers  (2026-05-16T16:08:52Z)
• Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation [0.0]
  We compare tested models to the contemporaneous frontier on the Epoch AI Capabilities Index.<n>The gap between these answers is widening at +5.53 ECI/year.<n>Proposed remedies include API-access subsidies and editorial enforcement of reporting frameworks.
  arXiv  Detail & Related papers  (2026-05-05T17:58:35Z)
• When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors [66.18091962164219]
  Existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.<n>We propose two complementary metrics to isolate non-mandatory behavioral patterns: textbfResponse Pattern Similarity (RPS) for verbal alignment and textbfAction Graph Similarity (AGS) for tool-use habits modeled as directed graphs.
  arXiv  Detail & Related papers  (2026-04-23T03:48:56Z)
• LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning [46.294745464571456]
  LongCat-Flash-Prover is an open-source MoE model for agentic tool-integrated reasoning.<n>It sets a new state-of-the-art for open-weights models in both auto-formalization and theorem proving.<n>It achieves a 97.1% pass rate on MiniF2F-Test using only 72 inference budget per problem.
  arXiv  Detail & Related papers  (2026-03-22T05:16:09Z)
• Nonstandard Errors in AI Agents [6.890249567932368]
  We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results.<n>We find that AI agents exhibit sizable textitnonstandard errors (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices.<n>These findings have implications for the growing use of AI in automated policy evaluation and empirical research.
  arXiv  Detail & Related papers  (2026-03-17T16:21:22Z)
• ResearchGym: Evaluating Language Model Agents on Real-World AI Research [48.46915933681714]
  We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research.<n>To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL.<n>In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap.
  arXiv  Detail & Related papers  (2026-02-16T19:00:03Z)
• AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline [2.1787849426740364]
  We analyzed documentation from five frontier models (Gemini 3, Grok 4.1, Llama 4, GPT-5, and Claude 4.5) and 100 Hugging Face model cards.<n>We developed a weighted transparency framework with 8 sections and 23 subsections that prioritizes safety-critical disclosures.
  arXiv  Detail & Related papers  (2025-12-13T19:48:44Z)
• SpecEval: Evaluating Model Adherence to Behavior Specifications [63.13000010340958]
  We introduce an automated framework that audits models against their providers specifications.<n>Our central focus is on three way consistency between a provider specification, its model outputs, and its own models as judges.<n>We apply our framework to 16 models from six developers across more than 100 behavioral statements, finding systematic inconsistencies including compliance gaps of up to 20 percent across providers.
  arXiv  Detail & Related papers  (2025-09-02T16:18:40Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.