A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation

• URL: http://arxiv.org/abs/2603.25780v1
• Date: Thu, 26 Mar 2026 16:03:52 GMT
• Title: A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation
• Abstract summary: Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems.<n>We show that classical mathematical validation can be fully automated by a Judge Agent.<n>72 blinded tasks submitted by 12 independent scientists yield an 89% success rate with automated error bounds, versus 53% without the Judge.
• Score: 1.7259824817932294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.

Related papers

• Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis [59.44063760362954]
  We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment.<n>In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points.<n>In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred.
  arXiv  Detail & Related papers  (2026-08-20T11:13:27Z)
• SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models [1.9873319610438742]
  SciCode is the standard measure of the scientific-coding ability of language models.<n>It is a component of the Artificial Analysis Intelligence Index.<n>State-of-the-art models are far more proficient in scientific coding than SciCode has suggested.
  arXiv  Detail & Related papers  (2026-08-05T15:45:55Z)
• Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects [2.9713899640333348]
  Integrating AI-powered coding agents into Continuous Integration/Continuous Delivery pipelines has fundamentally altered how software verification is conducted.<n>While these agents successfully automate the test generation, current evaluation benchmarks largely focus on pass-rates rather than the intrinsic quality of the generated tests.<n>We compare 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset.
  arXiv  Detail & Related papers  (2026-07-13T18:40:24Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• Science Earth: Towards A Planet-Scale Operating System for AI-Native Scientific Discovery [51.70607714288616]
  Science Earth is a planet-scale scientific runtime in which any capability can connect to any other.<n>Its underlying EACN protocol lets capabilities discover one another, negotiate task ownership, and adjudicate across incompatible evidentiary standards.<n>Two runs validate this under structurally distinct conditions.
  arXiv  Detail & Related papers  (2026-05-31T16:05:41Z)
• Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning [16.398764816978584]
  Lean is increasingly used to judge natural-language mathematical answers, but its signal is partial.<n>We show this signal is (i) sharply coverage-dependent, that is the proof-winning answer is correct 96% of the time at high proved coverage but 20% at low.<n>A 7B autoformalizer proves a class for only 28% of problems, and a manual audit finds only approximately 43% of those faithful.<n>We propose COVCAL, a selector over Lean-trace diagnostics that certifies a finite-sample selective-risk bound on accepted answers or abstains.
  arXiv  Detail & Related papers  (2026-05-27T11:59:28Z)
• MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models [0.0]
  MIRROR is a benchmark that evaluates whether large language models can use self-knowledge to make better decisions.<n>We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances.
  arXiv  Detail & Related papers  (2026-04-15T08:41:12Z)
• A Physics-Informed Neuro-Fuzzy Framework for Quantum Error Attribution [0.4511923587827302]
  We present a neuro-fuzzy framework that addresses the attribution problem by combining Adaptive Neuro-Fuzzy Inference Systems with physics-grounded feature engineering.<n>We introduce the Bhattacharyya Veto, a hard physical constraint grounded in the Data Processing Inequality.<n>This work establishes a robust, interpretable diagnostic layer that prevents error mitigation techniques from being applied to logically flawed circuits.
  arXiv  Detail & Related papers  (2026-02-22T16:19:51Z)
• Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis [0.0]
  Adversarial comments produce small, statistically non-significant effects on detection accuracy.<n>Complex adversarial strategies offer no advantage over simple manipulative comments.<n>Comment stripping reduces detection for weaker models by removing helpful context.
  arXiv  Detail & Related papers  (2026-02-18T00:34:17Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• MSC-180: A Benchmark for Automated Formal Theorem Proving from Mathematical Subject Classification [21.9173105378467]
  Current large language model (LLM)-based theorem provers suffer from limitations such as restricted domain coverage and weak generalization in mathematical reasoning.<n>We propose MSC-180, a benchmark for evaluation based on the MSC 2020 mathematical subject classification.<n>It comprises 180 formal verification problems, 3 advanced problems from each of 60 mathematical branches, spanning from undergraduate to graduate levels.
  arXiv  Detail & Related papers  (2025-12-20T07:39:19Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research [19.97666809905332]
  Large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.<n>Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.
  arXiv  Detail & Related papers  (2025-05-17T05:45:16Z)
• Intelligent Fault Diagnosis of Type and Severity in Low-Frequency, Low Bit-Depth Signals [0.6144680854063939]
  The research leverages sound data from the imbalanced MaFaulDa dataset, aiming to strike a balance between high performance and low resource consumption. We achieved an impressive accuracy of 99.54% and an F-Beta score of 99.52% with just 6 boosting trees at an 8 kHz, 8-bit configuration.
  arXiv  Detail & Related papers  (2024-11-09T22:01:11Z)
• Common 7B Language Models Already Possess Strong Math Capabilities [61.61442513067561]
  This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities. The potential for extensive scaling is constrained by the scarcity of publicly available math questions.
  arXiv  Detail & Related papers  (2024-03-07T18:00:40Z)
• Uncertainty-inspired Open Set Learning for Retinal Anomaly Identification [71.06194656633447]
  We establish an uncertainty-inspired open-set (UIOS) model, which was trained with fundus images of 9 retinal conditions. Our UIOS model with thresholding strategy achieved an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set. UIOS correctly predicted high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images.
  arXiv  Detail & Related papers  (2023-04-08T10:47:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.