Related papers: SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

URL: http://arxiv.org/abs/2603.20253v1
Date: Wed, 11 Mar 2026 05:00:48 GMT
Title: SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs
Abstract summary: We introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations.<n>SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost.<n>Each simulator's cost is analytically defined and platform-independent.
Score: 56.07550353240028
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,916 single-round (initial guess) and 1,900 multi-round (adjustment by trial-and-error) tasks across 12 simulators from fluid dynamics, solid mechanics, and plasma physics. Each simulator's cost is analytically defined and platform-independent. Frontier LLMs achieve 46--64% success rates in single-round mode, dropping to 35--54% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 71--80%, but LLMs are 1.5--2.5x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench.

Related papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
arXiv Detail & Related papers (2026-03-11T19:12:31Z)
Simulating Environments with Reasoning Models for Agent Training [55.98861707136674]
Building bespoke environments for training is heavy, brittle, and limits progress.<n>We propose two frameworks: Simia-SFT and Simia-RL.<n>Simia-SFT and Simia-RL enable scalable agent training without environment engineering.
arXiv Detail & Related papers (2025-11-03T18:29:57Z)
SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
arXiv Detail & Related papers (2025-10-20T13:14:38Z)
Efficient dataset construction using active learning and uncertainty-aware neural networks for plasma turbulent transport surrogate models [0.0]
This work demonstrates a proof-of-principle for using uncertainty-aware architectures to construct efficient datasets for surrogate model generation.<n>This strategy was applied again to the plasma turbulent transport problem within tokamak fusion plasmas, specifically the QuaLiKiz quasilinear electrostatic gyrokinetic turbulent transport code.<n>With 45 active learning iterations, moving from a small initial training set of $102$ to a final set of $104$, the resulting models reached a $F_1$ classification performance of 0.8 and a $R2$ regression performance of 0.75 on an independent test set
arXiv Detail & Related papers (2025-07-21T18:15:12Z)
G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration [48.948187359727996]
G-Sim is a hybrid framework that automates simulator construction with rigorous empirical calibration.<n>It produces reliable, causally-informed simulators, mitigating data-inefficiency and enabling robust system-level interventions.
arXiv Detail & Related papers (2025-06-10T22:14:34Z)
Transfer learning for multifidelity simulation-based inference in cosmology [0.0]
Pre-training on dark-matter-only $N$-body simulations reduces the required number of high-fidelity hydrodynamical simulations by a factor between $8$ and $15$.<n>By leveraging cheaper simulations, our approach enables performant and accurate inference on high-fidelity models while substantially reducing computational costs.
arXiv Detail & Related papers (2025-05-27T14:04:30Z)
Cosmological Analysis with Calibrated Neural Quantile Estimation and Approximate Simulators [0.0]
We introduce a new Simulation-Based Inference ( SBI) method that leverages a large number of approximate simulations for training and a small number of high-fidelity simulations for calibration. As a proof of concept, we demonstrate that cosmological parameters can be inferred at field level from projected 2-dim dark matter density maps up to $k_rm maxsim1.5,h$/Mpc at $z=0$. The calibrated posteriors closely match those obtained by directly training on $sim104$ expensive Particle-Particle (PP) simulations, but at a fraction of the computational cost
arXiv Detail & Related papers (2024-11-22T05:53:46Z)
INSIGHT: Universal Neural Simulator for Analog Circuits Harnessing Autoregressive Transformers [13.94505840368669]
INSIGHT is an effective universal neural simulator in the analog front-end design automation loop. It accurately predicts the performance metrics of analog circuits with just a few microseconds of inference time.
arXiv Detail & Related papers (2024-07-10T03:52:53Z)
LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
arXiv Detail & Related papers (2024-05-09T11:49:05Z)
Fast emulation of cosmological density fields based on dimensionality reduction and supervised machine-learning [0.0]
We show that it is possible to perform fast dark matter density field emulations with competitive accuracy using simple machine-learning approaches. New density cubes for different cosmological parameters can be estimated without relying directly on new N-body simulations.
arXiv Detail & Related papers (2023-04-12T18:29:26Z)
AutoSimulate: (Quickly) Learning Synthetic Data Generation [70.82315853981838]
We propose an efficient alternative for optimal synthetic data generation based on a novel differentiable approximation of the objective. We demonstrate that the proposed method finds the optimal data distribution faster (up to $50times$), with significantly reduced training data generation (up to $30times$) and better accuracy ($+8.7%$) on real-world test datasets than previous methods.
arXiv Detail & Related papers (2020-08-16T11:36:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.