The Agentic Garden of Forking Paths

• URL: http://arxiv.org/abs/2607.01507v1
• Date: Wed, 01 Jul 2026 22:15:37 GMT
• Title: The Agentic Garden of Forking Paths
• Abstract summary: We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit.<n>Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions.<n>Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports.
• Score: 27.003267976848225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Empirical research rarely admits a unique analysis. Different analytical choices can lead to different conclusions from the same data, yet these hidden forking paths are difficult to observe. We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit. Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions from the same data and question, with findings systematically aligned with those beliefs. In a study in which 42 human research teams analyzed the same immigration dataset, AI agents reproduced 72% of the human ideological gap in reported effect estimates. Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports: 86% passed independent AI review and 78% passed majority human expert review. These findings suggest that the central challenge is often not flawed analyses, but selective exploration and reporting from a large space of methodologically defensible analyses. AI agents may amplify this longstanding problem by making such exploration inexpensive and scalable. To address this, we introduce the m-value (multiverse value), the probability that an analysis path would produce a claim at least as extreme as the reported one. We further introduce Agentic Bootstrap, which estimates the m-value by using AI agents to sample plausible analysis paths. Applied to the human immigration study, 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m<0.05). Scientific evidence should therefore be evaluated not only by a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. Agentic Bootstrap makes this distribution observable and turns it into a criterion for scientific credibility.

Related papers

• Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents [72.19038675107662]
  Mr. L is a benchmark for evaluating real-world deep research over long, irreducible chains of verifiable evidence.<n>Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions.
  arXiv  Detail & Related papers  (2026-09-10T09:47:44Z)
• EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [73.12811119273206]
  We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.<n>Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer.<n>Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%.
  arXiv  Detail & Related papers  (2026-08-24T17:29:16Z)
• Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis [59.44063760362954]
  We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment.<n>In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points.<n>In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred.
  arXiv  Detail & Related papers  (2026-08-20T11:13:27Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task [20.392151164052404]
  We evaluated whether autonomous access to a medical research skill package was associated with higher-quality AI-generated transcriptomic research-analysis outputs.<n>Skill-augmented outputs showed directionally higher expert overall quality than native-AI outputs.
  arXiv  Detail & Related papers  (2026-06-10T09:13:10Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• Nonstandard Errors in AI Agents [6.890249567932368]
  We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results.<n>We find that AI agents exhibit sizable textitnonstandard errors (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices.<n>These findings have implications for the growing use of AI in automated policy evaluation and empirical research.
  arXiv  Detail & Related papers  (2026-03-17T16:21:22Z)
• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse [22.927943525772857]
  We show that fully autonomous AI analysts built on large language models (LLMs) can reproduce a similar structured analytic diversity cheaply and at scale.<n>We show that the effects are emphsteerable: reassigning the analyst persona or LLM shifts the distribution of outcomes even after excluding methodologically deficient runs.
  arXiv  Detail & Related papers  (2026-02-21T04:10:21Z)
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis [3.6324565773746147]
  We conduct a so-called multiverse analysis on a published empirical SE paper.<n>We identify nine pivotal analytical decisions with at least one equally defensible alternative.<n>The overwhelming majority produced qualitatively different, and sometimes even opposite, findings.
  arXiv  Detail & Related papers  (2025-12-09T18:47:00Z)
• Language Agents Mirror Human Causal Reasoning Biases. How Can We Help Them Think Like Scientists? [42.29911505696807]
  Language model (LM) agents are increasingly used as autonomous decision-makers.<n>We examine LMs' ability to explore and infer causal relationships.<n>We find that LMs reliably infer the common, intuitive disjunctive causal relationships but systematically struggle with the unusual, yet equally, evidenced conjunctive ones.
  arXiv  Detail & Related papers  (2025-05-14T17:59:35Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.