Arithmetic Pedagogy for Language Models

• URL: http://arxiv.org/abs/2606.05106v1
• Date: Wed, 03 Jun 2026 17:09:25 GMT
• Title: Arithmetic Pedagogy for Language Models
• Abstract summary: We investigate whether methods of human mathematics pedagogy can guide the training of language models toward arithmetic reasoning.<n>We operationalize each operation as a computational procedure whose execution trace is serialized into natural-language Chain-of-Thought (CoT) supervision.<n>A small GPT-2 decoder (86M parameters) with a syllabic-agglutinative TOBA tokenizer for Indonesian is trained from scratch on this data.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate whether methods of human mathematics pedagogy can guide the training of language models toward arithmetic reasoning. Building on the GASING method -- an Indonesian pedagogy that solves basic arithmetic through a left-to-right procedure aligned with the causal order of token generation -- we operationalize each operation as a computational procedure whose execution trace is serialized into natural-language Chain-of-Thought (CoT) supervision. A small GPT-2 decoder (86M parameters) with a syllabic-agglutinative TOBA tokenizer for Indonesian is trained from scratch on this data using only a next-token prediction objective, without reinforcement learning or reward-based optimization. Monitoring training reveals three distinct learning phases, and mechanistic analyses -- attention-masking interventions on the CoT information graph, residual-stream probing, and logit-lens inspection -- show that the model first internalizes a procedural pathway and subsequently develops an associative, ``mental-arithmetic'' capacity that retrieves intermediate results without explicit step-by-step computation. The trained model reaches over 80% accuracy on held-out problems and attains competitive performance against substantially larger language models, indicating that targeted, pedagogically grounded training can yield strong and economical arithmetic capability at small scale.

Related papers

• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent [80.83250816918861]
  Large Reasoning Models (LRMs) like o3 and DeepSeek-R1 have achieved remarkable progress in natural language reasoning with long chain-of-thought.<n>However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations.<n>We present AgentMath, an agent framework that seamlessly integrates language models' reasoning capabilities with code interpreters' computational precision.
  arXiv  Detail & Related papers  (2025-12-23T19:57:49Z)
• Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence [113.48450265178953]
  We study how to convert existing pretrained non-recurrent language models into depth-recurrent models.<n>We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost.
  arXiv  Detail & Related papers  (2025-11-10T18:43:07Z)
• Domain-Adaptive Continued Pre-Training of Small Language Models [0.0]
  Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources.<n>I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch.<n>My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation.
  arXiv  Detail & Related papers  (2025-04-13T18:40:32Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback [71.95402654982095]
  We propose Math-Minos, a natural language feedback-enhanced verifier. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier.
  arXiv  Detail & Related papers  (2024-06-20T06:42:27Z)
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision [23.807288360423193]
  We propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named textitOmegaPRM for the efficient collection of high-quality process supervision data.<n>We are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs)<n>This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances.
  arXiv  Detail & Related papers  (2024-06-05T19:25:40Z)
• AlphaMath Almost Zero: Process Supervision without Process [6.318873143509028]
  We propose an innovative framework, AlphaMath, that bypasses the need for process annotations by leveraging Monte Carlo Tree Search (MCTS) This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-05-06T15:20:30Z)
• Reverse That Number! Decoding Order Matters in Arithmetic Learning [49.5504492920404]
  Our work introduces a novel strategy that reevaluates the digit order by prioritizing output from the least significant digit. Compared to the previous state-of-the-art (SOTA) method, our findings reveal an overall improvement of in accuracy while requiring only a third of the tokens typically used during training.
  arXiv  Detail & Related papers  (2024-03-09T09:04:53Z)
• Headless Language Models: Learning without Predicting with Contrastive Weight Tying [0.11510009152620666]
  Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. We propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in a contrastive fashion via Constrastive Weight Tying (CWT) We observe a significant +1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.
  arXiv  Detail & Related papers  (2023-09-15T12:20:00Z)
• Arithmetic with Language Models: from Memorization to Computation [3.077668143048211]
  This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing.
  arXiv  Detail & Related papers  (2023-08-02T13:58:37Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.