Preregistration for Experiments with AI Agents

• URL: http://arxiv.org/abs/2606.11217v1
• Date: Sun, 03 May 2026 19:57:27 GMT
• Title: Preregistration for Experiments with AI Agents
• Abstract summary: "In silico" behavioral experiments use AI agents as proxies for human participants.<n>This paper argues that preregistration practices should now be extended to experiments with AI agents.<n>We propose a preregistration template tailored to experiments with AI agents and call on conferences, journals, and funding agencies to make preregistration standard practice.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has given rise to a rapidly growing methodological paradigm: "in silico" behavioral experiments. Originally conceived as a way to use AI agents as proxies for human participants in studies of cognition, decision-making, and social dynamics, this approach has taken on new significance -- as AI agents increasingly negotiate, transact, and make consequential decisions on behalf of people and organizations, understanding their behavior has become a research priority in its own right. While these experiments with AI agents offer unprecedented advantages in terms of scalability, cost efficiency, and experimental control, they also inherit, and in some cases amplify, methodological vulnerabilities that have long plagued human subjects research. To address these issues, this paper argues that preregistration practices -- central to improving the credibility of human subjects experiments -- should now be extended to experiments with AI agents. We systematically catalog the researcher degrees of freedom that experiments with AI agents introduce -- model selection, prompt wording, settings, and outcome-contingent redesign, for example -- and show how the low cost of iteration and lack of reporting norms make these choices both easy to exploit and difficult to detect. We propose a preregistration template tailored to experiments with AI agents and call on conferences, journals, and funding agencies to make preregistration standard practice for this emerging research paradigm.

Related papers

• Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
  SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
  arXiv  Detail & Related papers  (2026-06-10T22:55:30Z)
• Towards a Virtual Neuroscientist: Autonomous Neuroimaging Analysis via Multi-Agent Collaboration [53.772014300855375]
  We introduce NIAgent, a multi-agent system for autonomous end-to-end neuroimaging analysis.<n>Unlike conventional flat tool-calling agents, NIAgent adopts a code-centric execution paradigm.<n>We propose a hierarchical verification framework for autonomous quality control, integrating cohort-level metric screening with agentic visual inspection.
  arXiv  Detail & Related papers  (2026-05-10T06:30:19Z)
• Conformity and Social Impact on AI Agents [42.04722694386303]
  This study examines conformity, the tendency to align with group opinions under social pressure, in large multimodal language models functioning as AI agents.<n>Our experiments reveal that AI agents exhibit a systematic conformity bias, aligned with Social Impact Theory, showing sensitivity to group size, unanimity, task difficulty, and source characteristics.<n>These findings reveal fundamental security vulnerabilities in AI agent decision-making that could enable malicious manipulation, misinformation campaigns, and bias propagation in multi-agent systems.
  arXiv  Detail & Related papers  (2026-01-08T21:16:28Z)
• Enabling Ethical AI: A case study in using Ontological Context for Justified Agentic AI Decisions [0.0]
  Authors show how this process captures institutional knowledge, improves response quality and efficiency, and mitigates institutional amnesia.<n>We argue for a shift from post-hoc explanation to justifiable Agentic AI, where decisions are grounded in explicit, inspectable evidence and reasoning to both experts and non-specialists.
  arXiv  Detail & Related papers  (2025-12-04T14:06:35Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• AgentEvolver: Towards Efficient Self-Evolving Agent System [51.54882384204726]
  We present AgentEvolver, a self-evolving agent system that drives autonomous agent learning.<n>AgentEvolver introduces three synergistic mechanisms: self-questioning, self-navigating, and self-attributing.<n>Preliminary experiments indicate that AgentEvolver achieves more efficient exploration, better sample utilization, and faster adaptation compared to traditional RL-based baselines.
  arXiv  Detail & Related papers  (2025-11-13T15:14:47Z)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite [75.58737079136942]
  We present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research.<n>Our suite comes with the first scientific research environment with production-grade search tools.<n>Our evaluation of 57 agents across 22 agent classes reveals several interesting findings.
  arXiv  Detail & Related papers  (2025-10-24T17:10:26Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Will Agents Replace Us? Perceptions of Autonomous Multi-Agent AI [0.0]
  This study analyzes responses from 130 participants to a survey on the capabilities, impact, and governance of AI agents.<n>We explore expected timelines for AI replacing programmers, identify perceived barriers to deployment, and examine beliefs about responsibility when agents make critical decisions.
  arXiv  Detail & Related papers  (2025-06-01T11:02:52Z)
• Can AI Agents Design and Implement Drug Discovery Pipelines? [1.5848629658789695]
  Current AI agent-based systems demonstrate proficiency in solving programming challenges and conducting research.<n>This paper introduces DO Challenge, a benchmark designed to evaluate the decision-making abilities of AI agents.<n>We present the Deep Thought multi-agent system, which demonstrated strong performance on the benchmark, outperforming most human teams.
  arXiv  Detail & Related papers  (2025-04-28T15:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.