Evaluating Stochasticity in Deep Research Agents

• URL: http://arxiv.org/abs/2602.23271v1
• Date: Thu, 26 Feb 2026 17:46:42 GMT
• Title: Evaluating Stochasticity in Deep Research Agents
• Abstract summary: Deep Research Agents (DRAs) are promising agentic systems to support research across domains such as financial decision-making, medical analysis, and scientific discovery.<n>Despite recent improvements in research quality, DRA system design often overlooks a critical barrier to real-world deployment:ity.<n>Under identical queries, repeated executions of DRAs can exhibit substantial variability in terms of research outcome, findings, and citations.
• Score: 34.59664118367103
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep Research Agents (DRAs) are promising agentic systems that gather and synthesize information to support research across domains such as financial decision-making, medical analysis, and scientific discovery. Despite recent improvements in research quality (e.g., outcome accuracy when ground truth is available), DRA system design often overlooks a critical barrier to real-world deployment: stochasticity. Under identical queries, repeated executions of DRAs can exhibit substantial variability in terms of research outcome, findings, and citations. In this paper, we formalize the study of stochasticity in DRAs by modeling them as information acquisition Markov Decision Processes. We introduce an evaluation framework that quantifies variance in the system and identify three sources of it: information acquisition, information compression, and inference. Through controlled experiments, we investigate how stochasticity from these modules across different decision steps influences the variance of DRA outputs. Our results show that reducing stochasticity can improve research output quality, with inference and early-stage stochasticity contributing the most to DRA output variance. Based on these findings, we propose strategies for mitigating stochasticity while maintaining output quality via structured output and ensemble-based query generation. Our experiments on DeepSearchQA show that our proposed mitigation methods reduce average stochasticity by 22% while maintaining high research quality.

Related papers

• Towards Trustworthy Report Generation: A Deep Research Agent with Progressive Confidence Estimation and Calibration [2.8268726984243497]
  We propose a novel deep research agent that incorporates progressive confidence estimation and calibration within the report generation pipeline.<n> Experimental results and case studies demonstrate that our method substantially improves interpretability and significantly increases user trust.
  arXiv  Detail & Related papers  (2026-04-07T14:46:26Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis [3.6324565773746147]
  We conduct a so-called multiverse analysis on a published empirical SE paper.<n>We identify nine pivotal analytical decisions with at least one equally defensible alternative.<n>The overwhelming majority produced qualitatively different, and sometimes even opposite, findings.
  arXiv  Detail & Related papers  (2025-12-09T18:47:00Z)
• Q-Sat AI: Machine Learning-Based Decision Support for Data Saturation in Qualitative Studies [0.0]
  The determination of sample size in qualitative research has traditionally relied on the subjective and often ambiguous principle of data saturation.<n>This study introduces a new, systematic model based on machine learning (ML) to make this process more objective.
  arXiv  Detail & Related papers  (2025-11-02T17:18:51Z)
• Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG [51.120170062795566]
  We propose Divide-Then-Align (DTA) to endow RAG systems with the ability to respond with "I don't know" when the query is out of the knowledge boundary.<n>DTA balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems.
  arXiv  Detail & Related papers  (2025-05-27T08:21:21Z)
• Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs [12.923119372847834]
  This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking.<n>Experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility.<n>Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.
  arXiv  Detail & Related papers  (2025-05-23T11:35:03Z)
• Identifying Trustworthiness Challenges in Deep Learning Models for Continental-Scale Water Quality Prediction [69.38041171537573]
  Water quality is foundational to environmental sustainability, ecosystem resilience, and public health.<n>Deep learning offers transformative potential for large-scale water quality prediction and scientific insights generation.<n>Their widespread adoption in high-stakes operational decision-making, such as pollution mitigation and equitable resource allocation, is prevented by unresolved trustworthiness challenges.
  arXiv  Detail & Related papers  (2025-03-13T01:50:50Z)
• Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain [26.72234494972736]
  Retrieval Augmented Generation (RAG) complements the knowledge of Large Language Models (LLMs) by leveraging external information to enhance response accuracy for queries.<n>Our study focuses on the impact of RAG, specifically examining whether RAG improves the confidence of LLM outputs in the medical domain.<n>We evaluate confidence by treating the model's predicted probability as its output and calculating several evaluation metrics which include calibration error method, entropy, the best probability, and accuracy.
  arXiv  Detail & Related papers  (2024-12-29T00:58:33Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications.<n>Our research identifies two critical latent factors affecting RAG's confidence in its predictions.<n>We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)
• Interpretable Causal Inference for Analyzing Wearable, Sensor, and Distributional Data [62.56890808004615]
  We develop an interpretable method for distributional data analysis that ensures trustworthy and robust decision-making. We demonstrate ADD MALTS' utility by studying the effectiveness of continuous glucose monitors in mitigating diabetes risks.
  arXiv  Detail & Related papers  (2023-12-17T00:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.