gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy

• URL: http://arxiv.org/abs/2605.11269v1
• Date: Mon, 11 May 2026 21:47:22 GMT
• Title: gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy
• Abstract summary: Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort.<n>We study whether state-of-the-art LLM coding agents can perform such end-to-end scientific modeling.<n>We introduce gwBenchmarks, a suite of eight tasks grounded in gravitational wave analytic calculations and numerical simulations collectively representing over $108$ core-hours of compute.
• Score: 2.316169407465118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort, including building fast waveform surrogates from expensive numerical relativity simulations, modeling orbital dynamics of black holes, fitting merger remnant properties and constructing template banks. These problems demand extreme precision to support detection and parameter inference, with state-of-the-art models achieving $\lesssim 10^{-4}$ relative error. We study whether state-of-the-art LLM coding agents can perform such end-to-end scientific modeling, where success requires constructing models with stringent accuracy criteria and reasoning about physical systems. We introduce gwBenchmarks, a suite of eight tasks grounded in gravitational wave analytic calculations and numerical simulations collectively representing over $10^8$ core-hours of compute. The tasks span interpolation, regression, and high-dimensional time-series modeling, requiring a combination of numerical methods, machine learning, and physics-informed approaches. In preliminary experiments, agents frequently relied on proxy metrics, partial evaluation, or fabricated results to spuriously complete tasks. We therefore implement an external pre-defined framework to gauge agent progress. Evaluating twelve coding agents, we find no consistent winner. On the easiest task, multiple agents converge to the same cubic spline solution, with one rediscovering a coordinate transformation widely used in the literature. On harder tasks like analytic waveform modeling, all agents fall 1-2 orders of magnitude short of domain requirements and exhibit systematic failures, including metric misuse, constraint violations, and result fabrication. Our code, data, and website are publicly available.

Related papers

• Model-Driven Discipline for Multi-Agent LLMs: Requirement-to-Verification Generation of Traceable System Models [62.62160094849002]
  RADIANT is an engineering methodology that combines MDE with Multi-Agent Large Language Models.<n>It automatically generates heterogeneous models across engineering phases.<n>It provides exact, automated change-impact analysis.
  arXiv  Detail & Related papers  (2026-07-18T08:50:55Z)
• Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction [0.02446672595462589]
  We introduce Collider-Bench, a benchmark for evaluating whether language-model agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software.<n>Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps.<n>Our results show that on average no agent reliably beats the physicist-in-the-loop solution.
  arXiv  Detail & Related papers  (2026-05-13T18:00:00Z)
• Discovery of Interpretable Surrogates via Agentic AI: Application to Gravitational Waves [0.0]
  We present textttGWAgent, a large language model (LLM)-based workflow that constructs interpretable analytic surrogates directly from simulation data.<n>We show that providing the agent with a physics-informed domain ansatz substantially improves output model accuracy.<n>The resulting analytic surrogate attains a median Advanced LIGO mismatch of $6.9times10-4$ together with an $sim 8.4times$ speedup in waveform evaluation.
  arXiv  Detail & Related papers  (2026-05-11T22:09:34Z)
• Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints [48.80158223838359]
  We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks.<n>Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases.<n>Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints.
  arXiv  Detail & Related papers  (2026-04-17T03:38:50Z)
• From Paper to Program: Accelerating Quantum Many-Body Algorithm Development via a Multi-Stage LLM-Assisted Workflow [4.866193347313755]
  Large language models (LLMs) can generate code rapidly but remain unreliable for scientific algorithms.<n>We introduce a multi-stage LLM-assisted workflow that separates theory extraction, formal specification, and code implementation.
  arXiv  Detail & Related papers  (2026-04-05T12:12:54Z)
• Hierarchical Inference and Closure Learning via Adaptive Surrogates for ODEs and PDEs [15.38864225184245]
  Inverse problems are the task of calibrating models to match data.<n>We develop a principled methodology for leveraging data from collections of distinct yet related physical systems.<n>We learn the shared unknown dynamics in the form of an ML-based closure model.
  arXiv  Detail & Related papers  (2026-03-04T10:30:08Z)
• An Agentic Framework for Autonomous Materials Computation [70.24472585135929]
  Large Language Models (LLMs) have emerged as powerful tools for accelerating scientific discovery.<n>Recent advances integrate LLMs into agentic frameworks, enabling retrieval, reasoning, and tool use for complex scientific experiments.<n>Here, we present a domain-specialized agent designed for reliable automation of first-principles materials computations.
  arXiv  Detail & Related papers  (2025-12-22T15:03:57Z)
• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges [72.19809898215857]
  We introduce ModelingBench, a novel benchmark featuring real-world-inspired, open-ended problems from math modeling competitions across diverse domains.<n>These tasks require translating natural language into formal mathematical formulations, applying appropriate tools, and producing structured, defensible reports.<n>We also present ModelingAgent, a multi-agent framework that coordinates tool use, supports structured, creative solutions, and generates well-grounded, creative solutions.
  arXiv  Detail & Related papers  (2025-05-21T03:33:23Z)
• Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries [23.111935712144277]
  Rapid and accurate simulations of fluid dynamics around complicated geometric bodies are critical in a variety of engineering and scientific applications.<n>While scientific machine learning (SciML) has shown considerable promise, most studies in this field are limited to simple geometries.<n>This paper addresses this gap by benchmarking diverse SciML models for fluid flow prediction over intricate geometries.
  arXiv  Detail & Related papers  (2024-12-31T00:23:15Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.