Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

• URL: http://arxiv.org/abs/2603.12180v1
• Date: Thu, 12 Mar 2026 17:11:22 GMT
• Title: Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
• Abstract summary: We introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents.<n>We show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning.<n>We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.
• Score: 37.38277822936901
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End? [28.383940617377856]
  We introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on operations research tasks.<n>It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema.<n>Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice.
  arXiv  Detail & Related papers  (2026-06-18T04:43:23Z)
• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics [24.125726163497742]
  FML-Bench is a benchmark of 18 fundamental ML research tasks across 10 domains.<n>It separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics.<n>We find greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse.
  arXiv  Detail & Related papers  (2026-05-17T10:30:38Z)
• Learning to Retrieve from Agent Trajectories [72.8923565916533]
  We argue that retrieval models for agentic search should be trained directly from agent interaction data.<n>We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions.<n>Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
  arXiv  Detail & Related papers  (2026-03-30T17:59:02Z)
• AI Planning Framework for LLM-Based Web Agents [2.9376953730570197]
  We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms.<n>We propose five novel evaluation metrics that assess trajectory quality beyond simple success rates.<n>Our results show that while the Step-by-Step agent aligns more closely with human gold trajectories, the Full-Plan-in-Advance agent excels in technical measures.
  arXiv  Detail & Related papers  (2026-03-13T06:46:32Z)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization [64.61432234404276]
  emphSearch More, Think Less (SMTL) is a framework for long-horizon agentic search that targets both efficiency and generalization.<n>We train an end-to-end agent using supervised fine-tuning and reinforcement learning, achieving strong and often state of the art performance across benchmarks.
  arXiv  Detail & Related papers  (2026-02-26T06:46:41Z)
• ResearchGym: Evaluating Language Model Agents on Real-World AI Research [48.46915933681714]
  We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research.<n>To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL.<n>In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap.
  arXiv  Detail & Related papers  (2026-02-16T19:00:03Z)
• Repurposing Synthetic Data for Fine-grained Search Agent Supervision [81.95597592711688]
  LLM-based search agents are increasingly trained on entity-centric synthetic data.<n> prevailing training methods discard this rich entity information, relying instead on sparse, outcome-based rewards.<n>We introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function.
  arXiv  Detail & Related papers  (2025-10-28T17:50:40Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Deep Reinforcement Learning Agents are not even close to Human Intelligence [25.836584192349907]
  Deep reinforcement learning (RL) agents achieve impressive results in a wide variety of tasks, but they lack zero-shot adaptation capabilities.<n>We introduce HackAtari, a set of task variations of the Arcade Learning Environments.<n>We use it to demonstrate that, contrary to humans, RL agents systematically exhibit huge performance drops on simpler versions of their training tasks.
  arXiv  Detail & Related papers  (2025-05-27T20:21:46Z)
• Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summaries [8.413049356622201]
  We present an adaptive search method for efficiently generating contrasting behaviour summaries. Our results indicate that adaptive search can efficiently identify informative contrasting scenarios that enable humans to accurately select the better performing agent.
  arXiv  Detail & Related papers  (2023-04-01T18:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.