Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms

• URL: http://arxiv.org/abs/2604.15842v1
• Date: Fri, 17 Apr 2026 08:44:36 GMT
• Title: Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
• Abstract summary: Large language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling reasoning-intensive tasks remain underexplored.<n>We present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution.<n>Our experiments reveal that while the models recognize arithmetic tasks early, correct result occurs only in the final layers.
• Score: 10.546694596577057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling reasoning-intensive tasks remain underexplored. To advance the understanding of model-internal processing mechanisms, we present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution. Using early decoding, we trace how next-token predictions are constructed across layers. Our experiments reveal that while the models recognize arithmetic tasks early, correct result generation occurs only in the final layers. Notably, models proficient in arithmetic exhibit a clear division of labor between attention and MLP modules, where attention propagates input information and MLP modules aggregate it. This division is absent in less proficient models. Furthermore, successful models appear to process more challenging arithmetic tasks functionally, suggesting reasoning capabilities beyond factual recall.

Related papers

• FormuEvo: LLM-Guided Evolution for Discovering Solver-Efficient Mixed-Integer Programming Formulations [66.32852384262038]
  Mixed-integer programming (MIP) lies at the core of operations research and industrial optimization.<n>We propose FormuEvo, an evolutionary framework for automated discovery of solver-efficient MIP formulations.<n>FormuEvo frames MIP formulation design as evolutionary optimization over the symbolic space of MIP formulations.
  arXiv  Detail & Related papers  (2026-08-24T15:01:28Z)
• Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B [51.10897657510313]
  Large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction.<n>We investigate Llama 3.1-8B from a mechanistic interpretability point of view.<n>We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure.
  arXiv  Detail & Related papers  (2026-08-19T01:09:51Z)
• Large Language Models and Algorithm Execution: Application to an Arithmetic Function [0.0]
  We introduce a training model called LLM-DAL (Large Language Model - Decompositional Algorithmic Learning)<n>We demonstrate that LLMs' ability to perform complex algorithmic inferences and generalize can be significantly improved when the training method is properly designed.
  arXiv  Detail & Related papers  (2026-01-12T12:27:59Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Interpreting and Improving Large Language Models in Arithmetic Calculation [72.19753146621429]
  Large language models (LLMs) have demonstrated remarkable potential across numerous applications. In this work, we delve into uncovering a specific mechanism by which LLMs execute calculations. We investigate the potential benefits of selectively fine-tuning these essential heads/MLPs to boost the LLMs' computational performance.
  arXiv  Detail & Related papers  (2024-09-03T07:01:46Z)
• Towards Modeling Learner Performance with Large Language Models [7.002923425715133]
  This paper investigates whether the pattern recognition and sequence modeling capabilities of LLMs can be extended to the domain of knowledge tracing. We compare two approaches to using LLMs for this task, zero-shot prompting and model fine-tuning, with existing, non-LLM approaches to knowledge tracing. While LLM-based approaches do not achieve state-of-the-art performance, fine-tuned LLMs surpass the performance of naive baseline models and perform on par with standard Bayesian Knowledge Tracing approaches.
  arXiv  Detail & Related papers  (2024-02-29T14:06:34Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.