SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores

• URL: http://arxiv.org/abs/2609.06192v1
• Date: Sat, 05 Sep 2026 17:29:15 GMT
• Title: SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores
• Abstract summary: We formulate evidence-grounded multimodal scientific analysis, requiring agents to produce executable analyses and claims supported by results and visualizations from the same run.<n>We introduce SciRIGOR, an evaluation framework and benchmark comprising 100 cases from scientific articles across six domains and 17 subfields.<n>On full-benchmark runs, claims agree with faithful and unfaithful results at nearly identical rates (91.8% versus 91.0%). Yet no system exceeds 62.6% on the soft evidence-chain score or 18.0% strict whole-chain success.
• Score: 11.097584790037454
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific coding agents produce interdependent code, results, figures, and claims, yet evaluating final outputs alone does not establish whether their conclusions are scientifically supported. We formulate evidence-grounded multimodal scientific analysis, requiring agents to produce executable analyses and claims supported by results and visualizations from the same run. We introduce SciRIGOR, an evaluation framework and benchmark comprising 100 cases from scientific articles across six domains and 17 subfields. The framework reconstructs typed evidence graphs, separates artifact fidelity from relational validity, and scores complete claim-support paths while localizing the earliest unsupported relation. Source-grounded alternative paths accommodate scientifically equivalent analyses and visualizations. We evaluate 11 agent/model configurations. On full-benchmark runs, claims agree with faithful and unfaithful results at nearly identical rates (91.8% versus 91.0%). Yet no system exceeds 62.6% on the soft evidence-chain score or 18.0% strict whole-chain success. These findings show that internal coherence does not establish scientific correctness: evaluation must verify support along the complete data-to-claim path.

Related papers

• Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents [72.19038675107662]
  Mr. L is a benchmark for evaluating real-world deep research over long, irreducible chains of verifiable evidence.<n>Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions.
  arXiv  Detail & Related papers  (2026-09-10T09:47:44Z)
• Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents [56.93800079330285]
  Existing multimodal benchmarks largely evaluate final-answer accuracy.<n>We introduce Sci-MMR, a benchmark for multi-step evidence-grounded scientific reasoning.<n>We find that answer accuracy consistently exceeds complete-evidence recovery rate by more than 20%.
  arXiv  Detail & Related papers  (2026-09-10T08:41:07Z)
• TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents [11.367029696505357]
  We present TruthInsightBench, a benchmark configured for discovery.<n>Its 40 blind tasks, drawn from 40 peer-reviewed studies across 10 scientific domains, expose only a neutral scientific objective and frozen data.<n>A fixed LLM-based judge scores the evidentiary maturity of an agent's own claims along six dimensions.
  arXiv  Detail & Related papers  (2026-09-04T12:37:20Z)
• FrontierChallenge: Evaluating Scientific Workflow Completion [59.601983661596286]
  We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific deliverables.<n>We release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.<n>We find that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered.
  arXiv  Detail & Related papers  (2026-08-25T16:50:58Z)
• EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [73.12811119273206]
  We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.<n>Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer.<n>Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%.
  arXiv  Detail & Related papers  (2026-08-24T17:29:16Z)
• Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis [59.44063760362954]
  We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment.<n>In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points.<n>In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred.
  arXiv  Detail & Related papers  (2026-08-20T11:13:27Z)
• EviGraph: Evidence-Guided Autonomous Research Agents [12.29802995923839]
  We introduce EviGraph, an autonomous research framework that represents the research process as a typed evidence graph.<n>EviGraph inspects evidence chains for missing dependencies, semantic misalignment, and result-claim inconsistencies.<n>It localizes the earliest weak node, and regenerates its affected downstream subgraph.
  arXiv  Detail & Related papers  (2026-08-05T12:02:55Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• Can AI Agents Synthesize Scientific Conclusions? [35.01145833955259]
  We introduce SciConBench, a benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis.<n>We also introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.<n>Our results show that reliable synthesis of scientific conclusions remains an open challenge, and clean-room evaluation is essential for assessing open-domain AI agents.
  arXiv  Detail & Related papers  (2026-06-09T18:16:04Z)
• RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation [29.44948404858214]
  RIGOURATE retrieves supporting evidence from a paper's body and assigns each claim an overstatement score.<n>The framework consists of a dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers.
  arXiv  Detail & Related papers  (2026-01-07T19:36:08Z)
• SciFact-Open: Towards open-domain scientific claim verification [61.288725621156864]
  We present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems. We collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1.
  arXiv  Detail & Related papers  (2022-10-25T05:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.