FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

• URL: http://arxiv.org/abs/2604.03893v1
• Date: Sat, 04 Apr 2026 23:18:58 GMT
• Title: FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
• Abstract summary: We introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks.<n>It is designed to evaluate AI's capacity for multistep diagrammatic reasoning.<n>Our database spans the electromagnetic, weak, and strong interactions of the Standard Model.
• Score: 11.098160996983417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Breakthroughs in frontier theory often depend on the combination of concrete diagrammatic notations with rigorous logic. While multimodal large language models (MLLMs) show promise in general scientific tasks, current benchmarks often focus on local information extraction rather than the global structural logic inherent in formal scientific notations. In this work, we introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks. It is designed to evaluate AI's capacity for multistep diagrammatic reasoning, which requires satisfying conservation laws and symmetry constraints, identifying graph topology, converting between diagrammatic and algebraic representations, and constructing scattering amplitudes under specific conventions and gauges. To support large-scale and reproducible evaluation, we developed an automated pipeline producing diverse Feynman diagrams along with verifiable topological annotations and amplitude results. Our database spans the electromagnetic, weak, and strong interactions of the Standard Model, encompasses over 100 distinct types and includes more than 2000 tasks. Experiments on state-of-the-art MLLMs reveal systematic failure modes, including unstable enforcement of physical constraints and violations of global topological conditions, highlighting the need for physics-grounded benchmarks for visual reasoning over scientific notation. FeynmanBench provides a logically rigorous test of whether AI can effectively engage in scientific discovery, particularly within theoretical physics.

Related papers

• Towards Physics-Faithful Generation of Scientific Diagrams [60.50428624576465]
  Princigram is a physics-faithful scientific-diagram generator.<n>We present Princigram, a physics-faithful scientific-diagram generator.
  arXiv  Detail & Related papers  (2026-08-13T11:40:33Z)
• OMNIFLOW: A Physics-Grounded Multimodal Agent for Generalized Scientific Reasoning [51.33849811496781]
  Large Language Models (LLMs) have demonstrated exceptional logical reasoning capabilities but frequently struggle with the continuous dynamics by Partial Differential Equations (PDEs)<n>We propose OMNIFLOW, a multimodal-symbolic architecture designed to ground frozen LLMs in fundamental physical laws without requiring domain-specific parameter updates.<n>We evaluate it on a benchmark spanning microscopic turbulence, theoretical Navier-Stokes, and macroscopic global weather forecasting.
  arXiv  Detail & Related papers  (2026-03-16T18:29:01Z)
• Physics-based phenomenological characterization of cross-modal bias in multimodal models [11.525886296936413]
  multimodal large language models (MLLMs) are breaking new ground in multimodal understanding, reasoning, and generation.<n>We argue that inconspicuous distortions arising from complex multimodal interaction dynamics can lead to systematic bias.
  arXiv  Detail & Related papers  (2026-02-24T07:21:08Z)
• P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiads [91.05736019384489]
  We introduce P1-VL, a family of open-source vision-language models engineered for advanced scientific reasoning.<n>Our flagship P1-VL-235B-A22B becomes the first open-source Vision-Language Model to secure 12 gold medals and achieves the state-of-the-art performance in the open-source models.
  arXiv  Detail & Related papers  (2026-02-10T06:28:08Z)
• SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence [60.202862987441684]
  We introduce scientific instruction following: the capability to solve problems while strictly adhering to the constraints that establish scientific validity.<n>Specifically, we introduce SciIF, a multi-discipline benchmark that evaluates this capability by pairing university-level problems with a fixed catalog of constraints.<n>By measuring both solution correctness and multi-constraint adherence, SciIF enables finegrained diagnosis of compositional reasoning failures.
  arXiv  Detail & Related papers  (2026-01-08T09:45:58Z)
• RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature [25.978951548176706]
  We introduce RxnBench, a benchmark designed to rigorously evaluate MLLMs on chemical reaction understanding from scientific PDFs.<n> RxnBench comprises two tasks: Single-Figure QA (SF-QA), which tests fine-grained visual perception and mechanistic reasoning, and Full-Document QA (FD-QA), which challenges models to synthesize information from 108 articles.<n>Our evaluation of MLLMs reveals a critical capability gap: while models excel at extracting explicit text, they struggle with deep chemical logic and precise structural recognition.
  arXiv  Detail & Related papers  (2025-12-29T16:05:38Z)
• PRiSM: An Agentic Multimodal Benchmark for Scientific Reasoning via Python-Grounded Evaluation [7.0748516420242495]
  PRiSM is a synthetic, fully dynamic, and multimodal benchmark for evaluating scientific reasoning via grounded Python code.<n> PRiSM includes over 24,750 university-level physics and math problems, and it leverages our scalable agent-based pipeline, PrismAgent.<n>We propose five targeted evaluation tasks covering perturbation, symbolic program synthesis, robustness, reasoning correction, and ambiguity resolution.
  arXiv  Detail & Related papers  (2025-12-05T18:14:55Z)
• PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning [57.868248683256574]
  PRISM-Physics is a process-level evaluation framework and benchmark for complex physics reasoning problems.<n> Solutions are represented as directed acyclic graphs (DAGs) of formulas.<n>Results show that our evaluation framework is aligned with human experts' scoring.
  arXiv  Detail & Related papers  (2025-10-03T17:09:03Z)
• PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models [69.73115077227969]
  We present PhysUniBench, a large-scale benchmark designed to evaluate and improve the reasoning capabilities of large language models (MLLMs)<n>PhysUniBench consists of 3,304 physics questions spanning 8 major sub-disciplines of physics, each accompanied by one visual diagram.<n>The benchmark's construction involved a rigorous multi-stage process, including multiple roll-outs, expert-level evaluation, automated filtering of easily solved problems, and a nuanced difficulty grading system with five levels.
  arXiv  Detail & Related papers  (2025-06-21T09:55:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.