Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models

• URL: http://arxiv.org/abs/2412.13791v1
• Date: Wed, 18 Dec 2024 12:33:50 GMT
• Title: Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models
• Authors: Xinyu Pang, Ruixin Hong, Zhanke Zhou, Fangrui Lv, Xinwei Yang, Zhilong Liang, Bo Han, Changshui Zhang,
• Abstract summary: Existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application.<n>We propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs.<n>Given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning.<n> Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.
• Score: 41.88825441287559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physics problems constitute a significant aspect of reasoning, necessitating complicated reasoning ability and abundant physics knowledge. However, existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. To mitigate these issues, we propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Specifically, the proposed framework constructs a comprehensive formula set to provide explicit physics knowledge and utilizes checklists containing detailed instructions to guide effective knowledge application. Namely, given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. During the process, checklists are employed to enhance LLMs' self-improvement in the analysis and reasoning stages. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.

Related papers

• PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning [36.193595420239845]
  We present PhysReason, a 1,200-problem benchmark for evaluating large language models. Problems require an average of 8.1 solution steps, with hard requiring 15.6. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on answer-level evaluation.
  arXiv  Detail & Related papers  (2025-02-17T17:24:14Z)
• MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations [90.07275414500154]
  We observe significant performance drops on MATH-P-Hard across various models. We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills.
  arXiv  Detail & Related papers  (2025-02-10T13:31:46Z)
• UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models [39.917074900737575]
  Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks. The domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics.
  arXiv  Detail & Related papers  (2025-02-01T06:42:02Z)
• Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents [27.112239616508834]
  Mixture of Refinement Agents (MoRA) is a novel agentic refinement framework for large language models (LLMs)<n>MoRA iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs.<n>We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA)
  arXiv  Detail & Related papers  (2024-12-01T14:15:55Z)
• SimLM: Can Language Models Infer Parameters of Physical Systems? [56.38608628187024]
  We investigate the performance of Large Language Models (LLMs) at performing parameter inference in the context of physical systems. Our experiments suggest that they are not inherently suited to this task, even for simple systems. We propose a promising direction of exploration, which involves the use of physical simulators to augment the context of LLMs.
  arXiv  Detail & Related papers  (2023-12-21T12:05:19Z)
• Physics simulation capabilities of LLMs [0.0]
  Large Language Models (LLMs) can solve some undergraduate-level to graduate-level physics textbook problems and are proficient at coding. We present an evaluation of state-of-the-art (SOTA) LLMs on PhD-level to research-level computational physics problems.
  arXiv  Detail & Related papers  (2023-12-04T18:06:41Z)
• Structured Chemistry Reasoning with Large Language Models [70.13959639460015]
  Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in chemistry. We introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Tests across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30% peak improvement.
  arXiv  Detail & Related papers  (2023-11-16T08:20:36Z)
• Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models [49.23348672822087]
  We propose Knowledge Crosswords, a benchmark consisting of incomplete knowledge networks bounded by structured factual constraints. The novel setting of geometric knowledge reasoning necessitates new LM abilities beyond existing atomic/linear multi-hop QA. We conduct extensive experiments to evaluate existing LLMs and approaches on Knowledge Crosswords.
  arXiv  Detail & Related papers  (2023-10-02T15:43:53Z)
• Using Large Language Model to Solve and Explain Physics Word Problems Approaching Human Level [0.0]
  Large language model (LLM) pre-trained on texts can not only solve pure math word problems, but also physics word problems. Our work is the first research to focus on the automatic solving, explanation, and generation of physics word problems.
  arXiv  Detail & Related papers  (2023-09-15T06:13:06Z)
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models [70.5763210869525]
  We introduce an expansive benchmark suite SciBench for Large Language Model (LLM) SciBench contains a dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%.
  arXiv  Detail & Related papers  (2023-07-20T07:01:57Z)
• Solving Quantitative Reasoning Problems with Language Models [53.53969870599973]
  We introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences.
  arXiv  Detail & Related papers  (2022-06-29T18:54:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.