Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

• URL: http://arxiv.org/abs/2605.02050v1
• Date: Sun, 03 May 2026 20:37:48 GMT
• Title: Principles and Guidelines for Randomized Controlled Trials in AI Evaluation
• Abstract summary: This work establishes a foundational framework for standardizing AI evaluation RCTs.<n>We adopt the four-validity framework and extend it with a fifth principle on transparency, repeatability, and verification.<n>We operationalize all five principles into 33 guidelines adapted for AI evaluation RCT contexts.
• Score: 34.88842637551998
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work establishes a foundational framework for standardizing AI evaluation RCTs (sometimes called human uplift studies). Drawing on established experimental practices from disciplines with established RCT traditions, including software engineering, economics, clinical and health sciences, and psychology, we adopt the (Shadish et al., 2002) four-validity framework and extend it with a fifth principle on transparency, repeatability, and verification adapted from the Transparency and Openness Promotion (TOP) Guidelines (Center for Open Science, 2025). We operationalize all five principles into 33 guidelines adapted for AI evaluation RCT contexts, expressed as requirements with rationales, implementation instructions, and evidence bases. We position the principles and guidelines as serving three key roles for AI evaluation RCTs: a design tool for planning studies, an evaluation rubric for assessing existing work, and a blueprint for standard setting as the field converges on norms. Our framework extends prior work by centering evaluation on human performance rather than model output alone, formalizing causal inference through RCT methodology for AI contexts, integrating heterogeneity analysis and practical significance assessment, implementing a graded transparency and repeatability framework, and addressing AI-specific challenges including model versioning, human-AI interaction dynamics, contamination and spillover effects, and equitable impact assessment.

Related papers

• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Practitioner Beliefs and Behaviors in AI-Enhanced Education: DOT Framework Survey Evidence [0.0]
  The study investigates AI familiarity, usage patterns, design-oriented practices, and pedagogical beliefs.<n>Results indicate that practitioners hold favorable views of AI as a pedagogical support while maintaining strong commitments to human oversight and critical evaluation.
  arXiv  Detail & Related papers  (2026-05-27T19:42:31Z)
• Structural Quality Gaps in Practitioner AI Governance Prompts: An Empirical Study Using a Five-Principle Evaluation Framework [0.0]
  We introduce a framework for evaluating whether AI is structurally complete.<n>Our evaluation reveals that 37% of evaluated file-model pairs score below the threshold threshold.<n>We discuss implications for engineering requirements practice in AI-assisted development contexts.
  arXiv  Detail & Related papers  (2026-04-22T21:18:28Z)
• SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution [82.31558282651811]
  Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents.<n>Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding.<n>We propose SAVOIR, a novel principled framework grounded in cooperative game theory.
  arXiv  Detail & Related papers  (2026-04-21T02:08:25Z)
• OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment [55.59322229889159]
  We propose OmniQuality-R, a unified reward modeling framework that transforms multi-task quality reasoning into continuous and interpretable reward signals.<n>We use a reasoning-enhanced reward modeling dataset to form a reliable chain-of-thought dataset for supervised fine-tuning.<n>We evaluate OmniQuality-R on three key IQA tasks: aesthetic quality assessment, technical quality evaluation, and text-image alignment.
  arXiv  Detail & Related papers  (2025-10-12T13:46:28Z)
• The 1st International Workshop on Disentangled Representation Learning for Controllable Generation (DRL4Real): Methods and Results [132.86866727471093]
  This paper reviews the 1st International Workshop on Disentangled Representation Learning for Controllable Generation (DRL4Real), held in conjunction with ICCV 2025.<n>DRL4Real focused on evaluating DRL methods in practical applications such as controllable generation, exploring advancements in model robustness, interpretability, and generalization.<n>The workshop accepted 9 papers covering a broad range of topics, including the integration of novel inductive biases (e.g., language), the application of diffusion models to DRL, 3D-aware disentanglement, and the expansion of DRL into specialized domains like autonomous driving and EEG analysis.
  arXiv  Detail & Related papers  (2025-08-15T16:35:41Z)
• A Systematic Review of User-Centred Evaluation of Explainable AI in Healthcare [1.57531613028502]
  This study aims to develop a framework of well-defined, atomic properties that characterise the user experience of XAI in healthcare.<n>We also provide context-sensitive guidelines for defining evaluation strategies based on system characteristics.
  arXiv  Detail & Related papers  (2025-06-16T18:30:00Z)
• Prioritization First, Principles Second: An Adaptive Interpretation of Helpful, Honest, and Harmless Principles [30.405680322319242]
  The Helpful, Honest, and Harmless (HHH) principle is a framework for aligning AI systems with human values.<n>We argue for an adaptive interpretation of the HHH principle and propose a reference framework for its adaptation to diverse scenarios.<n>This work offers practical insights for improving AI alignment, ensuring that HHH principles remain both grounded and operationally effective in real-world AI deployment.
  arXiv  Detail & Related papers  (2025-02-09T22:41:24Z)
• A Unified Understanding and Evaluation of Steering Methods [17.420727709895736]
  Steering methods provide a practical approach to controlling large language models by applying steering vectors to intermediate activations.<n>Despite their growing importance, the field lacks a unified understanding and consistent evaluation across tasks and datasets.<n>This paper introduces a unified framework for analyzing and evaluating steering methods, formalizing their core principles and offering theoretical insights into their effectiveness.
  arXiv  Detail & Related papers  (2025-02-04T20:55:24Z)
• Where Assessment Validation and Responsible AI Meet [0.0876953078294908]
  We propose a unified assessment framework that considers classical test validation theory and assessment-specific and domain-agnostic RAI principles and practice. The framework addresses responsible AI use for assessment that supports validity arguments, alignment with AI ethics to maintain human values and oversight, and broader social responsibility associated with AI use.
  arXiv  Detail & Related papers  (2024-11-04T20:20:29Z)
• A collection of principles for guiding and evaluating large language models [5.412690203810726]
  We identify and curate a list of 220 principles from literature, and derive a set of 37 core principles organized into seven categories. We conduct a small-scale expert survey, eliciting the subjective importance experts assign to different principles. We envision that the development of a shared model of principles can serve multiple purposes.
  arXiv  Detail & Related papers  (2023-12-04T12:06:12Z)
• Towards a General Framework for Continual Learning with Pre-training [55.88910947643436]
  We present a general framework for continual learning of sequentially arrived tasks with the use of pre-training. We decompose its objective into three hierarchical components, including within-task prediction, task-identity inference, and task-adaptive prediction. We propose an innovative approach to explicitly optimize these components with parameter-efficient fine-tuning (PEFT) techniques and representation statistics.
  arXiv  Detail & Related papers  (2023-10-21T02:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.