Structural Quality Gaps in Practitioner AI Governance Prompts: An Empirical Study Using a Five-Principle Evaluation Framework

• URL: http://arxiv.org/abs/2604.21090v1
• Date: Wed, 22 Apr 2026 21:18:28 GMT
• Title: Structural Quality Gaps in Practitioner AI Governance Prompts: An Empirical Study Using a Five-Principle Evaluation Framework
• Abstract summary: We introduce a framework for evaluating whether AI is structurally complete.<n>Our evaluation reveals that 37% of evaluated file-model pairs score below the threshold threshold.<n>We discuss implications for engineering requirements practice in AI-assisted development contexts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI governance programmes increasingly rely on natural language prompts to constrain and direct AI agent behaviour. These prompts function as executable specifications: they define the agent's mandate, scope, and quality criteria. Despite this role, no systematic framework exists for evaluating whether a governance prompt is structurally complete. We introduce a five-principle evaluation framework grounded in computability theory, proof theory, and Bayesian epistemology, and apply it to an empirical corpus of 34 publicly available AGENTS.md governance files sourced from GitHub. Our evaluation reveals that 37% of evaluated file-model pairs score below the structural completeness threshold, with data classification and assessment rubric criteria most frequently absent. These results suggest that practitioner-authored governance prompts exhibit consistent structural patterns that automated static analysis could detect and remediate. We discuss implications for requirements engineering practice in AI-assisted development contexts, identify a previously undocumented artefact classification gap in the AGENTS.md convention, and propose directions for tool support.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering [0.0]
  Large language models (LLMs) and agentic AI are shifting software engineering from manual coding toward intent specification, architecture, and governance.<n>Two paradigms have emerged: vibe coding, and an intuition-driven approach accepting AI artifacts via observed behavior, and Specification-Driven Development (SDD), which uses structured specifications as the authoritative source of truth.<n>This article makes three contributions. First, it identifies failure modes of ungoverned conversational generation: the productivity-reliability paradox, architectural erosion from limited context, security exposure, and technical debt.<n>Second, it introduces the Specification Governance Reference Model (SGRM), which formalize
  arXiv  Detail & Related papers  (2026-07-18T07:27:02Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Philosophical Dispositions as Behavioral Constraints for AI-Assisted Code Review: An Empirical Study [0.0]
  We present a system that constrains AI reviewer behavior through philosophical dispositions.<n>Each disposition is defined apophatically (by what it refuses to do)<n>We evaluate this system on 50 merged pull requests across 7 repositories spanning 5 programming languages.
  arXiv  Detail & Related papers  (2026-05-21T23:57:25Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems [0.0]
  The thesis argues that generative AI must be evaluated as a pluralist sociotechnical system.<n>MaSH Loops is a framework for tracing how models, users, and institutions co-construct meaning and values.<n>The World Values Benchmark introduces a distributional approach grounded in World Values Survey data.
  arXiv  Detail & Related papers  (2026-04-22T13:29:33Z)
• Governed Reasoning for Institutional AI [0.0]
  Cognitive Core is a governed decision substrate built from nine typed cognitive primitives.<n>Human review is a condition of execution rather than a post-hoc check.<n>We benchmark three systems on an 11-case balanced prior authorization appeal evaluation set.
  arXiv  Detail & Related papers  (2026-04-12T14:09:18Z)
• Reasoning Provenance for Autonomous AI Agents: Structured Behavioral Analytics Beyond State Checkpoints and Execution Traces [0.0]
  Agent Execution Record (AER) is a structured reasoning primitive that captures intent, observation, and inference as first-class queryable fields on every step.<n>We show how AERs enable population-level behavioral analytics: reasoning pattern mining, confidence calibration, cross-agent comparison, counterfactual regression testing via mock replay.
  arXiv  Detail & Related papers  (2026-03-23T08:27:54Z)
• Explainability and Certification of AI-Generated Educational Assessments [0.0]
  This chapter proposes a comprehensive framework for explainability and certification of AI-generated assessment items.<n>A structured certification metadata schema is introduced to capture provenance, alignment predictions, reviewer actions, and ethical indicators.<n>A proof-of-concept study on 500 AI-generated computer science questions demonstrates the framework's feasibility.
  arXiv  Detail & Related papers  (2026-03-18T11:33:58Z)
• Computable Gap Assessment of Artificial Intelligence Governance in Children's Centres: Evidence-Mechanism-Governance-Indicator Modelling of UNICEF's Guidance on AI and Children 3.0 Based on the Graph-GAP Framework [5.260137087369841]
  We propose a methodology that decomposes requirements from authoritative policy texts into a four layer graph of evidence, mechanism, governance, and indicator.<n>Using the UNICEF Innocenti Guidance on AI and Children 3.0 as primary material, we define reproducible extraction units, coding manuals, graph patterns, scoring scales, and consistency checks.<n>Results suggest that compared with privacy and data protection, requirements related to child well being and development, explainability and accountability, and cross agency implementation and resource allocation are more prone to indicator gaps and mechanism gaps.
  arXiv  Detail & Related papers  (2025-12-20T17:03:17Z)
• BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
  BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
  arXiv  Detail & Related papers  (2025-11-26T06:39:19Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.