Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS

• URL: http://arxiv.org/abs/2603.20630v1
• Date: Sat, 21 Mar 2026 03:58:40 GMT
• Title: Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS
• Abstract summary: Large language models (LLMs) are changing the way researchers interact with code and data in scientific computing.<n>We propose an evaluation procedure that enables domain experts to assess the validity of LLM-generated input files for LAMMPS, a widely used molecular dynamics (MD) code.
• Score: 41.99844472131922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are changing the way researchers interact with code and data in scientific computing. While their ability to generate general-purpose code is well established, their effectiveness in producing scientifically valid code/input scripting for domain-specific languages (DSLs) remains largely unexplored. We propose an evaluation procedure that enables domain experts (who may not be experts in the DSL) to assess the validity of LLM-generated input files for LAMMPS, a widely used molecular dynamics (MD) code, and to use those assessments to evaluate the performance of state-of-the-art LLMs and identify common issues. Key to the evaluation procedure are a normalization step to generate canonical files and an extensible parser for syntax analysis. The following steps isolate common errors without incurring costly tests (in time and computational resources). Once a working input file is generated, LLMs can accelerate verification tests. Our findings highlight limitations of LLMs in generating scientific DSLs and a practical path forward for their integration into domain-specific computational ecosystems by domain experts.

Related papers

• Toward Automated and Trustworthy Scientific Analysis and Visualization with LLM-Generated Code [6.068120728706316]
  Large language models (LLMs) offer a promising solution by generating code from natural language descriptions.<n>We construct a benchmark suite of domain-inspired prompts that reflect real-world research tasks.<n>Our findings show that, without human intervention, the reliability of LLM-generated code is limited.
  arXiv  Detail & Related papers  (2025-11-26T21:27:03Z)
• Can LLMs Replace Humans During Code Chunking? [2.4056836012742]
  Large language models (LLMs) have become essential tools in computer science, especially for tasks involving code understanding and generation.<n>This paper examines the application of LLMs in the modernization of legacy government code written in ALC and MUMPS.
  arXiv  Detail & Related papers  (2025-06-24T13:02:35Z)
• MoRE-LLM: Mixture of Rule Experts Guided by a Large Language Model [54.14155564592936]
  We propose a Mixture of Rule Experts guided by a Large Language Model (MoRE-LLM)<n>MoRE-LLM steers the discovery of local rule-based surrogates during training and their utilization for the classification task.<n>LLM is responsible for enhancing the domain knowledge alignment of the rules by correcting and contextualizing them.
  arXiv  Detail & Related papers  (2025-03-26T11:09:21Z)
• CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings [32.72039589832989]
  Large language models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency.<n>These advancements challenge programming skills, ethics, and assessment integrity, making the detection of LLM-generated code essential for maintaining accountability and standards.<n>We propose a framework capable of distinguishing between human- and LLM-written code across multiple programming languages, code generators, and domains.
  arXiv  Detail & Related papers  (2025-03-17T21:41:37Z)
• DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation [48.11754113512047]
  This study includes a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains. Our pipeline works in a fully automated manner, enabling a push-bottom construction from code repositories into formatted subjects under study. The contributions of this study include a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL.
  arXiv  Detail & Related papers  (2024-08-23T16:33:58Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• Large Language Models for Code Analysis: Do LLMs Really Do Their Job? [13.48555476110316]
  Large language models (LLMs) have demonstrated significant potential in the realm of natural language understanding and programming code processing tasks. This paper offers a comprehensive evaluation of LLMs' capabilities in performing code analysis tasks.
  arXiv  Detail & Related papers  (2023-10-18T22:02:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.