MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows

• URL: http://arxiv.org/abs/2608.02642v1
• Date: Fri, 31 Jul 2026 13:48:07 GMT
• Title: MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows
• Abstract summary: We introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects.<n>We evaluate six model/harness configurations spanning Codex and OpenCode.<n>Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict successes@1 (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50.
• Score: 20.54127577222597
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accelerating scientific discovery is among the most consequential applications of AI, and computational biomolecular simulation stands out as a particularly promising target within this broader effort. Coding agents promise to automate significant portions of this workflow, yet their reliability on realistic molecular dynamics (MD) tasks remains poorly characterized. To address this issue, we introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects, spanning 29 molecular systems and 14 broad research protocols, including trajectory analysis, complex system preparation, free-energy protocols, and enhanced sampling. We evaluate six model/harness configurations spanning Codex and OpenCode. Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50. Average correctness and process rewards are substantially higher than strict success rates across all configurations, indicating that agents frequently make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows. Hard tasks remain largely unsolved, particularly membrane-protein system preparation and alchemical free-energy setup, both unsolved or near-unsolved by every evaluated configuration. MDArena thus exposes a substantial gap between the usefulness of coding agents as supervised assistants and their reliability as autonomous MD researchers, while providing a reproducible and extensible platform for tracking progress toward closing it.

Related papers

• OpenAI4S: Code as Action, Science as Sessions [57.428071188574]
  We present OpenAI4S, an open-source scientific research agent built around the principle of emphCode as Action, Science as Sessions.<n>OpenAI4S combines a persistent computing runtime with research-session management orchestration is handled through structured tool calls.<n>We evaluate OpenAI4S on 36 research scenarios spanning retrosynthesis, molecular protein binder design, protein mutation, catalyst screening, and mineral spectroscopy.
  arXiv  Detail & Related papers  (2026-09-14T06:16:59Z)
• Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs [78.01752802974855]
  We propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference.<n>We show that Mix-Quant preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
  arXiv  Detail & Related papers  (2026-05-19T17:50:17Z)
• MDGYM: Benchmarking AI Agents on Molecular Simulations [25.37963706902467]
  We introduce MDGYM, a benchmark of 169 expert-curated MD simulations spanning LAMMPS and GROMACS.<n>We evaluate three agentic frameworks -- Claude Code, Codex, and OpenHands -- with four LLMs, and find that all perform poorly.<n>These failure modes are qualitatively distinct from those observed in general software engineering benchmarks, indicating that fluent code generation does not transfer to grounded physical reasoning.
  arXiv  Detail & Related papers  (2026-05-09T13:21:51Z)
• Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Research [2.4251063179584436]
  Mimosa is an evolving multi-agent framework that automatically synthesizes task-specific multi-agents and iteratively refines them.<n>Released as a fully open-source platform, Mimosa aims to provide an open foundation for community-driven ASR.
  arXiv  Detail & Related papers  (2026-03-30T20:35:57Z)
• DynaMate: An Autonomous Agent for Protein-Ligand Molecular Dynamics Simulations [13.253932177045842]
  Force field-based molecular dynamics (MD) simulations are indispensable for probing the structure, dynamics, and functions of biomolecular systems.<n>The technical complexity of MD setup, encompassing parameterization, input preparation, and software configuration, remains a major barrier for widespread and efficient usage.<n>Here, we present DynaMate, a modular multi-agent framework that autonomously designs and executes complete MD for both protein and protein-ligand systems.
  arXiv  Detail & Related papers  (2025-12-10T19:40:51Z)
• Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization [69.36509281190662]
  Adapting production-level computer vision tools to bespoke scientific datasets is a critical "last mile" bottleneck.<n>We consider using AI agents to automate this manual coding, and focus on the open question of optimal agent design.<n>We demonstrate that a simple agent framework consistently generates adaptation code that outperforms human-expert solutions.
  arXiv  Detail & Related papers  (2025-12-02T18:42:26Z)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization [103.74675519953898]
  Long-chain reflective reasoning is a prerequisite for solving complex real-world problems.<n>We build a benchmark consisting 1,260 samples of 42 challenging synthetic tasks.<n>We generate post-training data and explore learning paradigms for exploiting such data.
  arXiv  Detail & Related papers  (2025-10-09T17:53:58Z)
• CoDA: Agentic Systems for Collaborative Data Visualization [57.270599188947294]
  Deep research has revolutionized data analysis, yet data scientists still devote substantial time to manually crafting visualizations.<n>Existing approaches, including simple single- or multi-agent systems, often oversimplify the task.<n>We introduce CoDA, a multi-agent system that employs specialized LLM agents for metadata analysis, task planning, code generation, and self-reflection.
  arXiv  Detail & Related papers  (2025-10-03T17:30:16Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Superior Molecular Representations from Intermediate Encoder Layers [0.006860025762294234]
  We analyze the information flow in five diverse molecular encoders and find that intermediate layers retain more general-purpose features.<n>We find that using frozen embeddings from optimal intermediate layers improves downstream performance by an average of 5.4%, up to 28.6%, compared to the final-layer.
  arXiv  Detail & Related papers  (2025-06-06T18:03:51Z)
• MDCrow: Automating Molecular Dynamics Workflows with Large Language Models [0.6130124744675498]
  We introduce MDCrow, an agentic LLM assistant capable of automating Molecular dynamics simulations.<n>We assess MDCrow's performance across 25 tasks of varying required subtasks and difficulty, and we evaluate the agent's robustness to both difficulty and prompt style.
  arXiv  Detail & Related papers  (2025-02-13T18:19:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.