Related papers: SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

URL: http://arxiv.org/abs/2606.13020v1
Date: Thu, 11 Jun 2026 07:54:22 GMT
Title: SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Abstract summary: SciR is a benchmark that combines multi-paradigm reasoning with controllable scientific rendering.<n>To our knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty.
Score: 14.183744057358547
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on these in scientific settings is currently out of reach: scientific benchmarks built on human annotations are costly and lack mechanistic ground truth, while synthetic logical-reasoning benchmarks do not resemble real scientific documents. We introduce SciR, a benchmark that combines multi-paradigm reasoning with controllable scientific rendering, anchored on three paradigmatic scientific problems. Tasks are generated from formal objects (deduction tree, inductive rule hypothesis, causal graph) to guarantee verifiable answers, then rendered into multi-document scientific discourse via per-track domain-tuned genres. The construction lets us independently vary two difficulty axes: how hard it is to extract the key information needed for inference, and how hard the principled inference itself is. We test six models. Both axes hurt every model, and their effects compound. The rendering even hurts neurosymbolic pipelines, which hand inference to a verified solver. The two axes yield a per-model extraction-vs-inference profile: for instance, reasoning models like deepseek-r1 mostly surpass non-reasoning instruct models on the inference axis. To our knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty.

Related papers

Can LLMs Discover Scientific Laws in Real and Parallel Worlds? [58.76147897928744]
We introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data.<n>It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines.<n>We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck.
arXiv Detail & Related papers (2026-09-01T17:17:59Z)
SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature [26.176405135367038]
We introduce SPARK, a paper-oriented synthesis framework built on Sci-Base.<n> SPARK distills each paper into a compact reasoning skeleton capturing its central claims and supporting evidence.<n>We construct Spark-234K, a scientific reasoning dataset with substantially higher difficulty and diversity than existing resources.
arXiv Detail & Related papers (2026-08-31T03:55:30Z)
Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations [27.330265411994258]
SCHEMA is an evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents.<n>We show that hallucinations concentrate at a small set of highly connected knowledge hubs.<n>Results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability.
arXiv Detail & Related papers (2026-08-01T15:19:10Z)
Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility [57.83550091882176]
We study scientific image synthesis across generation paradigms, evaluation, and downstream use.<n>We introduce SciGenBench, which evaluates generated images based on information utility and logical validity.<n>We show that fine-tuning Large Multimodal Models on rigorously verified synthetic scientific images yields consistent reasoning gains.
arXiv Detail & Related papers (2026-01-17T14:18:36Z)
ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction [70.53044880892196]
We introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT)<n>To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints.<n> Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain.
arXiv Detail & Related papers (2025-11-16T07:37:09Z)
LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models [20.800445482814958]
Large Language Models (LLMs) have gained interest for their potential to leverage embedded scientific knowledge for hypothesis generation.<n>Existing benchmarks often rely on common equations that are susceptible to memorization by LLMs, leading to inflated performance metrics that do not reflect discovery.<n>In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains.<n>Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorized forms, and LSR- Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning
arXiv Detail & Related papers (2025-04-14T17:00:13Z)
Context-Aware Reasoning On Parametric Knowledge for Inferring Causal Variables [49.31233968546582]
We introduce a novel benchmark where the objective is to complete a partial causal graph.<n>We show the strong ability of LLMs to hypothesize the backdoor variables between a cause and its effect.<n>Unlike simple memorization of fixed associations, our task requires the LLM to reason according to the context of the entire graph.
arXiv Detail & Related papers (2024-09-04T10:37:44Z)
Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.<n>We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.<n>Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
arXiv Detail & Related papers (2024-05-27T13:26:34Z)
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
arXiv Detail & Related papers (2023-09-06T05:19:41Z)
RerrFact: Reduced Evidence Retrieval Representations for Scientific Claim Verification [4.052777228128475]
We propose a modular approach that sequentially carries out binary classification for every prediction subtask. We carry out two-step stance predictions that first differentiate non-relevant rationales and then identify supporting or refuting rationales for a given claim. Experimentally, our system RerrFact with no fine-tuning, simple design, and a fraction of model parameters fairs competitively on the leaderboard.
arXiv Detail & Related papers (2022-02-05T21:52:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.