MDGYM: Benchmarking AI Agents on Molecular Simulations

• URL: http://arxiv.org/abs/2605.08941v1
• Date: Sat, 09 May 2026 13:21:51 GMT
• Title: MDGYM: Benchmarking AI Agents on Molecular Simulations
• Abstract summary: We introduce MDGYM, a benchmark of 169 expert-curated MD simulations spanning LAMMPS and GROMACS.<n>We evaluate three agentic frameworks -- Claude Code, Codex, and OpenHands -- with four LLMs, and find that all perform poorly.<n>These failure modes are qualitatively distinct from those observed in general software engineering benchmarks, indicating that fluent code generation does not transfer to grounded physical reasoning.
• Score: 25.37963706902467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The promise of AI-driven scientific discovery hinges on whether AI agents can autonomously design and execute the computational workflows that underpin modern science. Molecular dynamics (MD) simulation presents a natural test bed to stress-test this claim; it requires translating physical intuition into syntactically and semantically correct input scripts, reasoning about initial and boundary conditions, diagnosing numerically unstable trajectories, and interpreting outputs against known physical behavior and laws. We introduce MDGYM, a benchmark of 169 expert-curated MD simulations spanning LAMMPS and GROMACS, two widely used MD packages, across three increasing difficulty levels. We evaluate three agentic frameworks -- Claude Code, Codex, and OpenHands -- with four LLMs, and find that all perform poorly: even the strongest agent solves only 21\% of easy-level tasks, with less than 10\% at higher difficulties. Trajectory analysis reveals a characteristic pattern of failure -- agents successfully invoke simulation machinery but produce physically unstable configurations, fabricate numerical outputs without executing the underlying computation, or abandon tasks prematurely rather than iterating through simulation-specific errors. These failure modes are qualitatively distinct from those observed in general software engineering benchmarks, indicating that fluent code generation does not transfer to grounded physical reasoning.

Related papers

• From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities [103.60384544335072]
  Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence.<n>Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities.<n> Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles.
  arXiv  Detail & Related papers  (2026-08-27T10:52:06Z)
• gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy [2.316169407465118]
  Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort.<n>We study whether state-of-the-art LLM coding agents can perform such end-to-end scientific modeling.<n>We introduce gwBenchmarks, a suite of eight tasks grounded in gravitational wave analytic calculations and numerical simulations collectively representing over $108$ core-hours of compute.
  arXiv  Detail & Related papers  (2026-05-11T21:47:22Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows [10.078675569996992]
  We evaluate CMBAgent across two workflow paradigms and eighteen astrophysical tasks.<n>In the One-Shot setting, access to domain-specific context yields an approximately 6x performance improvement.<n>In the Deep Research setting, the system frequently exhibits silent failures across stress tests, producing physically inconsistent posteriors without self-diagnosis.
  arXiv  Detail & Related papers  (2026-04-28T08:01:23Z)
• PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement [35.63080643326192]
  Physics-aware symbolic simulation of 3D scenes is critical for robotics, embodied AI, and scientific computing.<n>We introduce PhysCodeBench, the first comprehensive benchmark for evaluating physics-aware symbolic simulation.<n>Our evaluation framework measures both code executability and physical accuracy through automated and visual assessment.
  arXiv  Detail & Related papers  (2026-04-26T07:37:20Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• PhyNiKCE: A Neurosymbolic Agentic Framework for Autonomous Computational Fluid Dynamics [0.0]
  This work introduces PhyNiKCE, a neurosymbolic agentic framework for trustworthy engineering.<n>Unlike standard black-box agents, PhyNiKCE decouples neural planning from symbolic validation.<n>This architecture offers a scalable, auditable paradigm for Trustworthy Artificial Intelligence in broader industrial automation.
  arXiv  Detail & Related papers  (2026-02-12T07:37:56Z)
• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• Chain of Unit-Physics: A Primitive-Centric Approach to Scientific Code Synthesis [0.0]
  This work conceptualizes an inverse approach to code design: a first-principles (or primitives)-centric, multi-agent system in which human expert knowledge is encoded as unit-physics tests that explicitly constrain code generation.<n>As datasets and models evolve, zero-shot code accuracy will improve; however, the Chain of Unit-Physics framework goes further by embedding first-principles analysis that is foundational to scientific codes.
  arXiv  Detail & Related papers  (2025-11-30T18:16:50Z)
• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• Multi-fidelity Hierarchical Neural Processes [79.0284780825048]
  Multi-fidelity surrogate modeling reduces the computational cost by fusing different simulation outputs. We propose Multi-fidelity Hierarchical Neural Processes (MF-HNP), a unified neural latent variable model for multi-fidelity surrogate modeling. We evaluate MF-HNP on epidemiology and climate modeling tasks, achieving competitive performance in terms of accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2022-06-10T04:54:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.