Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification

• URL: http://arxiv.org/abs/2609.15145v1
• Date: Mon, 14 Sep 2026 07:20:44 GMT
• Title: Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
• Abstract summary: We define errors in the reasoning process as reasoning errors and manually analyze the features of reasoning errors.<n>We design a prompt that explicitly focuses on eight error classes.<n>The results suggest that the frequent reasoning errors identified in this paper are common across LLMs of comparable scale.
• Score: 12.689084576723351
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The reasoning ability of large language models (LLMs) is a critical factor for practical LLM-based applications. To investigate the current reasoning capability of LLMs, we clarify the types of errors that arise in LLMs' reasoning processes on mathematical datasets. We focus on problems where LLMs produce an incorrect answer. We define errors in the reasoning process as reasoning errors and manually analyze the features of reasoning errors. We defined and classified 21 error classes and identified the frequently occurring classes among them. Beyond qualitative evaluation, we leverage the evaluation results to improve the reasoning capability. We designed a prompt that explicitly focuses on eight error classes. The experiments demonstrate that this prompt effectively improves reasoning performance. Furthermore, the results suggest that the frequent reasoning errors identified in this paper are common across LLMs of comparable scale.

Related papers

• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• CounterBench: A Benchmark for Counterfactuals Reasoning in Large Language Models [5.409370027524351]
  We evaluate the performance of large language models (LLMs) in counterfactual reasoning.<n>We introduce a new benchmark dataset, CounterBench, comprising 1K counterfactual reasoning questions.
  arXiv  Detail & Related papers  (2025-02-16T06:19:37Z)
• No Need for Explanations: LLMs can implicitly learn from mistakes in-context [15.70441204657966]
  We study why Large Language Models learn from mistakes more effectively without explicit corrective rationales.<n>We find evidence that, while incorrect answers are more beneficial for LLM learning, explicit corrective rationales over-constrain the model.
  arXiv  Detail & Related papers  (2025-02-12T16:31:21Z)
• Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning [34.427730009102966]
  We develop an automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs.<n>Our work will also serve as an evaluation framework that can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks.
  arXiv  Detail & Related papers  (2025-02-08T19:49:32Z)
• Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying [0.3659498819753633]
  State-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning.<n>This paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation.<n>We show that employing these critical questions can improve the reasoning capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-12-19T18:51:30Z)
• Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment [53.17596274334017]
  We evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs.<n>Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic.
  arXiv  Detail & Related papers  (2024-10-06T08:33:39Z)
• Not All LLM Reasoners Are Created Equal [58.236453890457476]
  We study the depth of grade-school math problem-solving capabilities of LLMs. We evaluate their performance on pairs of existing math word problems together.
  arXiv  Detail & Related papers  (2024-10-02T17:01:10Z)
• Automated Theorem Provers Help Improve Large Language Model Reasoning [0.18416014644193066]
  We show how accuracy can be improved with a neuro-symbolic architecture. We define a framework of syntactic and semantic error categories. We extend our method with capabilities for automatically correcting syntactic and semantic errors.
  arXiv  Detail & Related papers  (2024-08-07T01:03:56Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)
• LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models [63.14196038655506]
  We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) Our methodology reveals significant gaps in LLMs' learning of logical rules, with identified reasoning failures ranging from 29% to 90% across different models. We leverage these findings to construct targeted demonstration examples and fine-tune data, notably enhancing logical reasoning in models like GPT-4o by up to 5%.
  arXiv  Detail & Related papers  (2024-01-01T13:53:53Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• Improving Large Language Models in Event Relation Logical Prediction [33.88499005859982]
  Event relation extraction is a challenging task that demands thorough semantic understanding and rigorous logical reasoning. In this paper, we conduct an in-depth investigation to systematically explore the capability of LLMs in understanding and applying event relation logic. Our study reveals that LLMs are not logically consistent reasoners, which results in their suboptimal performance on tasks that need rigorous reasoning.
  arXiv  Detail & Related papers  (2023-10-13T14:53:06Z)
• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning [55.76083560152823]
  SelfCheck is a general-purpose zero-shot verification schema for recognizing errors in step-by-step reasoning. We test SelfCheck on three datasets (GSM8K, MathQA, and MATH) and find that it successfully recognizes errors and, in turn, increases final answer accuracies.
  arXiv  Detail & Related papers  (2023-08-01T10:31:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.