What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code
- URL: http://arxiv.org/abs/2605.19762v1
- Date: Tue, 19 May 2026 12:37:01 GMT
- Title: What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code
- Abstract summary: We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation.<n>Code substantially improves programming ability but does not act as a general reasoning enhancer.<n>Cross-domain structured reasoning traces, such as code-text and math-text mixtures, are better explained.
- Score: 72.9921566968371
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Code has become a standard component of modern foundation language model (LM) training, yet its role beyond programming remains unclear. We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation. Our findings are threefold. First, when code is restricted to standalone executable programs and Code-NL data are controlled for, code substantially improves programming ability but does not act as a general reasoning enhancer; instead, it competes with knowledge-intensive tasks, especially complex mathematical reasoning. Second, the reasoning gains often attributed to code are better explained by cross-domain structured reasoning traces, such as code-text and math-text mixtures, rather than by executable code alone. Third, increasing the density of structured math-domain samples within a fixed math budget yields substantial gains on difficult mathematical reasoning while largely preserving programming performance, suggesting that cognitive scaffolds offer a targeted way to mitigate cross-domain trade-offs. Finally, routing analyses show that data-composition effects are reflected in expert-activation patterns, providing mechanism-level evidence for competitive and synergistic interactions across domains. Our results clarify which data characteristics transfer across capability dimensions and point to more precise data-centric optimization strategies.
Related papers
- The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis [50.473217152006875]
MIMIC is a framework that leverages executable code as a rigorous medium for reasoning data synthesis.<n>MIMIC transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation.<n>Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks.
arXiv Detail & Related papers (2026-09-13T17:04:16Z) - BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
arXiv Detail & Related papers (2025-11-26T06:39:19Z) - Chain of Execution Supervision Promotes General Reasoning in Large Language Models [48.100128916029064]
We introduce TracePile, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought-style rationales.<n>We evaluate TracePile using three training setups: continue-pretraining, instruction tuning after pretraining, and two-stage finetuning.<n> Notably, TracePile boosts LLaMA3.1-8B by 7.1% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and MMLU.
arXiv Detail & Related papers (2025-10-24T02:21:11Z) - Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code [29.382261465478248]
We introduce executable counterfactuals, a framework that operationalizes causal reasoning through code and math problems.<n>Our results reveal substantial drop in accuracy (25-40%) from interventional to counterfactual reasoning for SOTA models like o4-mini and Claude-4-Sonnet.<n>We also test whether a model trained on code would generalize to counterfactual math word problems.
arXiv Detail & Related papers (2025-10-02T00:26:35Z) - Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions [8.540135660509058]
Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding.<n>We leverage influence functions to attribute LLMs' reasoning ability on math and coding to individual training examples, sequences, and tokens.<n>High-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning.
arXiv Detail & Related papers (2025-05-26T13:15:26Z) - Is Compression Really Linear with Code Intelligence? [60.123628177110206]
textitFormat Annealing is a lightweight, transparent training methodology designed to assess the intrinsic capabilities of pre-trained models equitably.<n>Our empirical results reveal a fundamental logarithmic relationship between measured code intelligence and bits-per-character (BPC)<n>Our work provides a more nuanced understanding of compression's role in developing code intelligence and contributes a robust evaluation framework in the code domain.
arXiv Detail & Related papers (2025-05-16T16:59:14Z) - When Do Program-of-Thoughts Work for Reasoning? [51.2699797837818]
We propose complexity-impacted reasoning score (CIRS) to measure correlation between code and reasoning abilities.
Specifically, we use the abstract syntax tree to encode the structural information and calculate logical complexity.
Code will be integrated into the EasyInstruct framework at https://github.com/zjunlp/EasyInstruct.
arXiv Detail & Related papers (2023-08-29T17:22:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.