Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

• URL: http://arxiv.org/abs/2604.18786v1
• Date: Mon, 20 Apr 2026 19:48:33 GMT
• Title: Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models
• Abstract summary: We evaluate large language models (LLMs) under controlled knowledge conditions.<n>We probe robustness by removing portions of the experimental and/or outcome context.<n>Findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.
• Score: 17.31622097939325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.

Related papers

• EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation [68.80901568810616]
  EvoSCM aims to equip scientific agents with explicit structural causal models that evolve as new evidence is collected.<n>We evaluate EvoSCM onPhysics, a benchmark requiring agents to uncover the hidden dynamics of noncanonical physical worlds through experimentation.
  arXiv  Detail & Related papers  (2026-09-01T16:55:56Z)
• Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery [41.32793603340132]
  Large language models (LLMs) are increasingly involved in scientific discovery.<n>It remains unclear whether they can support complex real laboratory science.<n> Evaluation of 19 large language models (MLLMs) shows that even the best-performing model reaches only 48.7% accuracy.
  arXiv  Detail & Related papers  (2026-08-07T08:06:31Z)
• Knowing When Not to Answer: Abstention-Aware Scientific Reasoning [2.680633756465714]
  In scientific settings, unsupported or uncertain conclusions can be more harmful than abstaining.<n>We study this problem through an abstention-aware verification framework.<n>We evaluate this framework across two scientific benchmarks: SciFact and PubMedQA.
  arXiv  Detail & Related papers  (2026-02-15T15:29:43Z)
• IV Co-Scientist: Multi-Agent LLM Framework for Causal Instrumental Variable Discovery [61.15184885636171]
  In the presence of confounding between an endogenous variable and the outcome, instrumental variables (IVs) are used to isolate the causal effect of the endogenous variable.<n>We investigate whether large language models (LLMs) can aid in this task.<n>We introduce IV Co-Scientist, a multi-agent system that proposes, critiques, and refines IVs for a given treatment-outcome pair.
  arXiv  Detail & Related papers  (2026-02-08T12:28:29Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback [136.27567671480156]
  We introduce experiment-guided ranking, which prioritizes hypotheses based on feedback from prior tests.<n>We frame experiment-guided ranking as a sequential decision-making problem.<n>Our approach significantly outperforms pre-experiment baselines and strong ablations.
  arXiv  Detail & Related papers  (2025-05-23T13:24:50Z)
• Prediction-Powered Causal Inferences [59.98498488132307]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n>We first show that conditional calibration guarantees valid PPCI at population level.<n>We then introduce a sufficient representation constraint transferring validity across experiments.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery [32.395366135636046]
  We introduce BoxingGym, a benchmark with 10 environments for evaluating experimental design and model discovery.<n>We compute the expected information gain (EIG), an information-theoretic quantity which measures how much an experiment reduces uncertainty about the parameters of a generative model.<n>We find that current LLMs, such as GPT-4o, struggle with both experimental design and model discovery.
  arXiv  Detail & Related papers  (2025-01-02T21:15:57Z)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees [41.78390564658645]
  Large Language Models (LLMs) to generate hallucinations and non-factual content undermines their reliability in high-stakes domains. We introduce FactTest, a novel framework that statistically assesses whether a LLM can confidently provide correct answers to given questions. We show that FactTest effectively detects hallucinations and improves the model's ability to abstain from answering unknown questions, leading to an over 40% accuracy improvement.
  arXiv  Detail & Related papers  (2024-11-04T20:53:04Z)
• A Double Machine Learning Approach to Combining Experimental and Observational Data [58.05402364136958]
  We propose a double machine learning approach to combine experimental and observational studies.<n>Our framework proposes a falsification test for external validity and ignorability under milder assumptions.
  arXiv  Detail & Related papers  (2023-07-04T02:53:11Z)
• Sources of Irreproducibility in Machine Learning: A Review [3.905855359082687]
  There exist no theoretical framework that relates experiment design choices to potential effects on the conclusions. The objective of this paper is to develop a framework that enables applied data science practitioners and researchers to understand which experiment design choices can lead to false findings.
  arXiv  Detail & Related papers  (2022-04-15T18:26:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.