Physics simulation capabilities of LLMs

• URL: http://arxiv.org/abs/2312.02091v2
• Date: Mon, 2 Sep 2024 10:02:51 GMT
• Title: Physics simulation capabilities of LLMs
• Authors: Mohamad Ali-Dib, Kristen Menou,
• Abstract summary: Large Language Models (LLMs) can solve some undergraduate-level to graduate-level physics textbook problems and are proficient at coding. We present an evaluation of state-of-the-art (SOTA) LLMs on PhD-level to research-level computational physics problems.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: [Abridged abstract] Large Language Models (LLMs) can solve some undergraduate-level to graduate-level physics textbook problems and are proficient at coding. Combining these two capabilities could one day enable AI systems to simulate and predict the physical world. We present an evaluation of state-of-the-art (SOTA) LLMs on PhD-level to research-level computational physics problems. We condition LLM generation on the use of well-documented and widely-used packages to elicit coding capabilities in the physics and astrophysics domains. We contribute $\sim 50$ original and challenging problems in celestial mechanics (with REBOUND), stellar physics (with MESA), 1D fluid dynamics (with Dedalus) and non-linear dynamics (with SciPy). Since our problems do not admit unique solutions, we evaluate LLM performance on several soft metrics: counts of lines that contain different types of errors (coding, physics, necessity and sufficiency) as well as a more "educational" Pass-Fail metric focused on capturing the salient physical ingredients of the problem at hand. As expected, today's SOTA LLM (GPT4) zero-shot fails most of our problems, although about 40\% of the solutions could plausibly get a passing grade. About $70-90 \%$ of the code lines produced are necessary, sufficient and correct (coding \& physics). Physics and coding errors are the most common, with some unnecessary or insufficient lines. We observe significant variations across problem class and difficulty. We identify several failure modes of GPT4 in the computational physics domain. Our reconnaissance work provides a snapshot of current computational capabilities in classical physics and points to obvious improvement targets if AI systems are ever to reach a basic level of autonomy in physics simulation capabilities.

Related papers

• UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models [39.917074900737575]
  Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks. The domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics.
  arXiv  Detail & Related papers  (2025-02-01T06:42:02Z)
• Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models [41.88825441287559]
  Existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. We propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.
  arXiv  Detail & Related papers  (2024-12-18T12:33:50Z)
• Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback [33.000541253136745]
  Large Language Models (LLMs) have demonstrated strong capabilities in text-based tasks but struggle with the complex reasoning required for physics problems. This paper presents a novel approach to improving LLM performance on physics questions using Reinforcement Learning with Human and Artificial Intelligence Feedback (RLHAIF)
  arXiv  Detail & Related papers  (2024-12-06T21:17:47Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)
• Using Large Language Model to Solve and Explain Physics Word Problems Approaching Human Level [0.0]
  Large language model (LLM) pre-trained on texts can not only solve pure math word problems, but also physics word problems. Our work is the first research to focus on the automatic solving, explanation, and generation of physics word problems.
  arXiv  Detail & Related papers  (2023-09-15T06:13:06Z)
• Learning Controllable Adaptive Simulation for Multi-resolution Physics [86.8993558124143]
  We introduce Learning controllable Adaptive simulation for Multi-resolution Physics (LAMP) as the first full deep learning-based surrogate model. LAMP consists of a Graph Neural Network (GNN) for learning the forward evolution, and a GNN-based actor-critic for learning the policy of spatial refinement and coarsening. We demonstrate that our LAMP outperforms state-of-the-art deep learning surrogate models, and can adaptively trade-off computation to improve long-term prediction error.
  arXiv  Detail & Related papers  (2023-05-01T23:20:27Z)
• Physics Embedded Machine Learning for Electromagnetic Data Imaging [83.27424953663986]
  Electromagnetic (EM) imaging is widely applied in sensing for security, biomedicine, geophysics, and various industries. It is an ill-posed inverse problem whose solution is usually computationally expensive. Machine learning (ML) techniques and especially deep learning (DL) show potential in fast and accurate imaging. This article surveys various schemes to incorporate physics in learning-based EM imaging.
  arXiv  Detail & Related papers  (2022-07-26T02:10:15Z)
• A Review of Physics-based Machine Learning in Civil Engineering [0.0]
  Machine learning (ML) is a significant tool that can be applied across many disciplines. ML for civil engineering applications that are simulated in the lab often fail in real-world tests. This paper reviews the history of physics-based ML and its application in civil engineering.
  arXiv  Detail & Related papers  (2021-10-09T15:50:21Z)
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics [89.81550748680245]
  We introduce a new differentiable physics benchmark called PasticineLab. In each task, the agent uses manipulators to deform the plasticine into the desired configuration. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark.
  arXiv  Detail & Related papers  (2021-04-07T17:59:23Z)
• Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation [49.631034790080406]
  We present a task for navigating a marble to the center of a circular maze. We present a model that learns to move a marble in the complex environment within minutes of interacting with the real system.
  arXiv  Detail & Related papers  (2020-11-14T02:03:08Z)
• Scalable Differentiable Physics for Learning and Control [99.4302215142673]
  Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions.
  arXiv  Detail & Related papers  (2020-07-04T19:07:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.