SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
- URL: http://arxiv.org/abs/2607.20926v1
- Date: Thu, 23 Jul 2026 05:17:00 GMT
- Title: SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
- Abstract summary: We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents.<n> SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines.<n>We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured tasks.
- Score: 46.64739473662869
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.
Related papers
- Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
arXiv Detail & Related papers (2026-06-10T22:55:30Z) - From Text to Discovery: How Large Language Models Are Accelerating and Complicating Research Across Scientific and Humanistic Disciplines [4.953914351662556]
Large Language Models (LLMs) are rapidly reshaping academic research across the natural sciences, social sciences, and humanities.<n>This paper outlines an agenda for their responsible integration into scientific research.<n>We identify ten underexplored challenges, including the erosion of researcher autonomy, AI-driven confirmation bias, authorship ambiguity, and unequal access to these technologies.
arXiv Detail & Related papers (2026-06-07T16:38:27Z) - Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows [203.3527268311731]
We present an operational SGI definition grounded in the Practical Inquiry Model (PIM)<n>We operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning.<n>Our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.
arXiv Detail & Related papers (2025-12-18T12:44:36Z) - A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
arXiv Detail & Related papers (2025-08-28T18:30:52Z) - Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning [53.82037883518254]
We introduce SciReas, a diverse suite of existing benchmarks for scientific reasoning tasks.<n>We then propose KRUX, a probing framework for studying the distinct roles of reasoning and knowledge in scientific tasks.
arXiv Detail & Related papers (2025-08-26T17:04:23Z) - The Evolving Role of Large Language Models in Scientific Innovation: Evaluator, Collaborator, and Scientist [3.7803247326675162]
Scientific innovation is undergoing a paradigm shift driven by the rapid advancement of Large Language Models (LLMs)<n>This survey proposes a comprehensive framework to categorize the evolving roles of LLMs in scientific innovation across three hierarchical levels: Evaluator, Collaborator, and Scientist.
arXiv Detail & Related papers (2025-07-16T00:11:01Z) - The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers [11.88478056313921]
The problem is inherently more complex than general fact-checking since it must accommodate the evolving nature of scientific knowledge.<n>Existing approaches focus on simplified versions of the problem based on small-scale datasets consisting of abstracts.<n>This paper examines the limitations of current scientific fact-checking systems and reveals the potential features and resources that could be exploited to advance their performance.
arXiv Detail & Related papers (2025-06-25T21:29:33Z) - ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows [82.07367406991678]
Large Language Models (LLMs) have extended their impact beyond Natural Language Processing.<n>Among these, computer-using agents are capable of interacting with operating systems as humans do.<n>We introduce ScienceBoard, which encompasses a realistic, multi-domain environment featuring dynamic and visually rich scientific software.
arXiv Detail & Related papers (2025-05-26T12:27:27Z) - Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents [11.74019905854637]
Large language models (LLMs) are evolving into scientific agents that automate critical tasks.<n>Unlike general-purpose LLMs, specialized agents integrate domain-specific knowledge, advanced tool sets, and robust validation mechanisms.<n>We highlight why they differ from general agents and the ways in which they advance research across various scientific fields.
arXiv Detail & Related papers (2025-03-31T13:11:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.