RCTs & Human Uplift Studies: Methodological Challenges and Practical Solutions for Frontier AI Evaluation

• URL: http://arxiv.org/abs/2603.11001v1
• Date: Wed, 11 Mar 2026 17:25:03 GMT
• Title: RCTs & Human Uplift Studies: Methodological Challenges and Practical Solutions for Frontier AI Evaluation
• Abstract summary: Human uplift studies measure AI effects on human performance relative to a status quo.<n>These studies are increasingly used to inform deployment, governance, and safety decisions for frontier AI systems.<n>While the methods underlying these studies are well-established, their interaction with the distinctive properties of frontier AI systems remains underexamined.
• Score: 7.662976269800995
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human uplift studies - or studies that measure AI effects on human performance relative to a status quo, typically using randomized controlled trial (RCT) methodology - are increasingly used to inform deployment, governance, and safety decisions for frontier AI systems. While the methods underlying these studies are well-established, their interaction with the distinctive properties of frontier AI systems remains underexamined, particularly when results are used to inform high-stakes decisions. We present findings from interviews with 16 expert practitioners with experience conducting human uplift studies in domains including biosecurity, cybersecurity, education, and labor. Across interviews, experts described a recurring tension between standard causal inference assumptions and the object of study itself. Rapidly evolving AI systems, shifting baselines, heterogeneous and changing user proficiency, and porous real-world settings strain assumptions underlying internal, external, and construct validity, complicating the interpretation and appropriate use of uplift evidence. We synthesize these challenges across key stages of the human uplift research lifecycle and map them to practitioner-reported solutions, clarifying both the limits and the appropriate uses of evidence from human uplift studies in high-stakes decision-making.

Related papers

• A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inference [72.19201465132619]
  We present feedback-driven Test-Time Intelligence (TTI) as a unified perspective for understanding such deployment-time improvement.<n>We use this view to relate test-time adaptation, test-time learning, and test-time scaling, highlighting both their distinctions and their growing overlap in hybrid systems.<n>Our goal is to provide a coherent foundation and research roadmap for the study of self-improving AI systems at test time.
  arXiv  Detail & Related papers  (2026-09-01T11:41:39Z)
• Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics [25.781418598536906]
  Safety risks of AI are becoming increasingly evident in human interactions with AI technologies.<n>To examine this apparent gap in the acceptance of human research, we conduct an expert survey (n=93) and expert interviews (n=17) with AI Safety & Ethics (AISE) researchers.
  arXiv  Detail & Related papers  (2026-08-06T06:52:52Z)
• Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
  SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
  arXiv  Detail & Related papers  (2026-06-10T22:55:30Z)
• Understanding Risk and Dependency in AI Chatbot Use from User Discourse [4.1957094635667875]
  We present a large-scale computational thematic analysis of posts collected between 2023 and 2025 from two communities, r/AIDangers and r/ChatbotAddiction.<n>We identify 14 recurring thematic categories and synthesize them into five higher-order experiential dimensions.<n>Our findings reveal five empirically derived experiential dimensions of AI-related psychological risk grounded in real-world user discourse.
  arXiv  Detail & Related papers  (2026-02-10T02:16:57Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• Measuring skill-based uplift from AI in a real biological laboratory [0.0]
  We report the results of a pilot study that attempted to empirically measure the magnitude of emphskills-based uplift caused by access to an AI reasoning model.<n>We discuss these results in the context of future studies of the evolving relationship between AI and global biosecurity.
  arXiv  Detail & Related papers  (2025-10-29T16:34:57Z)
• How Performance Pressure Influences AI-Assisted Decision Making [52.997197698288936]
  We show how pressure and explainable AI (XAI) techniques interact with AI advice-taking behavior.<n>Our results show complex interaction effects, with different combinations of pressure and XAI techniques either improving or worsening AI advice taking behavior.
  arXiv  Detail & Related papers  (2024-10-21T22:39:52Z)
• Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations [18.971689499890363]
  We identify and analyze 97core papers with human-based XAI evaluations over the past five years. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems. We propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners.
  arXiv  Detail & Related papers  (2022-10-20T20:53:00Z)
• Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions [6.9295094033607825]
  reinforcement learning (RL) has acquired a prominent position in health-related sequential decision-making problems. However, its real-life application is still limited and its potential is still to be realized.
  arXiv  Detail & Related papers  (2022-03-04T23:14:02Z)
• Decision Rule Elicitation for Domain Adaptation [93.02675868486932]
  Human-in-the-loop machine learning is widely used in artificial intelligence (AI) to elicit labels from experts. In this work, we allow experts to additionally produce decision rules describing their decision-making. We show that decision rule elicitation improves domain adaptation of the algorithm and helps to propagate expert's knowledge to the AI model.
  arXiv  Detail & Related papers  (2021-02-23T08:07:22Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.