Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

• URL: http://arxiv.org/abs/2607.06820v1
• Date: Tue, 07 Jul 2026 21:29:59 GMT
• Title: Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
• Abstract summary: We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath.<n>We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark.
• Score: 1.166361205377345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation. We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set. Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7~pp on average, the gains range from 1.5~pp to 27.8~pp and narrow the gap between open-weight and closed models. Qwen~3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of $75.2\%$ and the lowest token usage among tool-enabled configurations. Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.

Related papers

• AMTFV: Agentic Mathematical Tool-Flow Verification for LLM Self-Correction [58.27846292449026]
  Large language models have demonstrated strong mathematical problem-solving capabilities, yet reliably verifying their candidate answers remains challenging.<n>We propose AMTFV (Agentic Mathematical Tool-Flow Verification), which decouples verification modeling from concrete execution.<n>We evaluate AMTFV on five challenging mathematical reasoning datasets with seven model configurations from DeepSeek, GPT, and Gemini.
  arXiv  Detail & Related papers  (2026-07-31T15:42:00Z)
• PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models [31.37252086950609]
  PyraMathBench is a comprehensive benchmark with 32,505 questions derived from 7,404 math word problems.<n>Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions.<n>We propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO) to enhance LLMs' numerical-mathematical synergy.
  arXiv  Detail & Related papers  (2026-06-02T16:32:53Z)
• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent [80.83250816918861]
  Large Reasoning Models (LRMs) like o3 and DeepSeek-R1 have achieved remarkable progress in natural language reasoning with long chain-of-thought.<n>However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations.<n>We present AgentMath, an agent framework that seamlessly integrates language models' reasoning capabilities with code interpreters' computational precision.
  arXiv  Detail & Related papers  (2025-12-23T19:57:49Z)
• StreetMath: Study of LLMs' Approximation Behaviors [1.4119508208285607]
  We introduce StreetMath, a benchmark designed to evaluate models' approximation abilities under real-world approximation scenarios.<n>Our analysis reveals that LLMs generally attempt to compute exact values or external tools even in tasks that call for approximation.<n>We argue that LLMs do not exhibit cognitive miserliness in the same way humans do in street math settings.
  arXiv  Detail & Related papers  (2025-10-27T05:16:00Z)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools [65.73762766854192]
  Agent Distillation is a framework for transferring reasoning capability and task-solving behavior from large language models into small language models.<n>Our results show that sLMs as small as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next-tier larger 1.5B, 3B, 7B models.
  arXiv  Detail & Related papers  (2025-05-23T08:20:15Z)
• BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search [22.672130194493793]
  Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. They still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. We propose a novel approach, BEATS, to enhance mathematical problem-solving abilities.
  arXiv  Detail & Related papers  (2024-09-26T15:47:42Z)
• Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On [55.449818944278526]
  We introduce the Skywork-Math model series, supervised fine-tuned (SFT) on common 7B language models. Skywork-Math 7B has achieved impressive accuracies of 51.2% on the competition-level MATH benchmark. We provide several practical takeaways to enhance math reasoning abilities in LLMs for both research and industry applications.
  arXiv  Detail & Related papers  (2024-07-11T09:56:51Z)
• MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time [51.5039731721706]
  MindStar is a purely inference-based searching method for large language models. It formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. It significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1.
  arXiv  Detail & Related papers  (2024-05-25T15:07:33Z)
• MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning [2.9104279358536647]
  We present MathSensei, a tool-augmented large language model for mathematical reasoning. We study the complementary benefits of the tools - knowledge retriever (Bing Web Search), program generator + executor (Python), and symbolic equation solver (Wolfram-Alpha API)
  arXiv  Detail & Related papers  (2024-02-27T05:50:35Z)
• Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions [47.83142414018448]
  We focus on two popular reasoning tasks: arithmetic reasoning and code generation. We introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets. We show a significant performance drop across all the models against perturbed questions.
  arXiv  Detail & Related papers  (2024-01-17T18:13:07Z)
• Modeling Complex Mathematical Reasoning via Large Language Model based MathAgent [15.81048994298046]
  Large language models (LLMs) face challenges in solving complex mathematical problems. We propose a formal description of the mathematical solving and extend LLMs with an agent-based zero-shot framework. Experiments on miniF2F and MATH have demonstrated the effectiveness of PRER and proposed MathAgents.
  arXiv  Detail & Related papers  (2023-12-14T13:33:50Z)
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct [130.37945867605302]
  We present WizardMath, which enhances the mathematical CoT reasoning abilities of large language models (LLMs) without using external python tools.<n>Remarkably, WizardMath-Mistral 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency.<n>Our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance.
  arXiv  Detail & Related papers  (2023-08-18T14:23:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.