On-Policy Delta Distillation for Multilingual Math Reasoning

• URL: http://arxiv.org/abs/2608.05802v1
• Date: Thu, 06 Aug 2026 09:37:49 GMT
• Title: On-Policy Delta Distillation for Multilingual Math Reasoning
• Abstract summary: We study On-Policy Distillation (OPD) for mathematical reasoning in English, Korean, and Japanese.<n>OPD$2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese.<n>English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English.
• Score: 61.76889440384448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.

Related papers

• Crosslingual On-Policy Self-Distillation for Multilingual Reasoning [48.68444770923683]
  Crosslingual On-Policy Self-Distillation (COPSD) transfers a model's own high-resource reasoning behavior to low-resource languages.<n>Experiments on 17 low-resource African languages show that COPSD consistently improves low-resource mathematical reasoning across model sizes.
  arXiv  Detail & Related papers  (2026-05-10T14:06:09Z)
• CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training [25.752819016822574]
  Cross-Lingual Enhancement in Retrieval via Reverse-training (CLEAR) is a novel loss function utilizing a reverse training scheme to improve retrieval performance.<n>CLEAR achieves notable improvements in cross-lingual scenarios, with gains up to 15%, particularly in low-resource languages.
  arXiv  Detail & Related papers  (2026-04-07T12:54:38Z)
• Do LLMs Need Inherent Reasoning Before Reinforcement Learning? A Study in Korean Self-Correction [7.756650000650388]
  We investigate whether reinforcement learning can enhance Korean reasoning abilities to a degree comparable to English.<n>Our findings reveal that RL alone yields limited improvements when applied to models lacking inherent Korean reasoning capabilities.<n>We show that aligning the model's internal reasoning processes with Korean inputs-particularly by tuning Korean-specific neurons in early layers-is key to unlocking RL's effectiveness.
  arXiv  Detail & Related papers  (2026-01-09T01:17:31Z)
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective [52.452449102961225]
  This study proposes a novel cross-linguistic perspective to investigate reasoning generalization.<n>Our findings reveal that cross-lingual transferability varies significantly across initial model, target language, and training paradigm.<n>Our study challenges the assumption that LRM reasoning mirrors human cognition, providing critical insights for the development of more language-agnostic LRMs.
  arXiv  Detail & Related papers  (2025-10-02T17:49:49Z)
• ShifCon: Enhancing Non-Dominant Language Capabilities with a Shift-based Multilingual Contrastive Framework [78.07201802874529]
  ShifCon is a Shift-based multilingual Contrastive framework that aligns the internal forward process of other languages toward that of the dominant one.<n>Experiments demonstrate that our ShifCon framework significantly enhances the performance of non-dominant languages.
  arXiv  Detail & Related papers  (2024-10-25T10:28:59Z)
• MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing [78.62611800987817]
  Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. We propose a method called MoE-LPR (Mixture-of-Experts with Language Priors) to enhance the multilingual capability.
  arXiv  Detail & Related papers  (2024-08-21T07:43:49Z)
• RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining [0.0]
  We present RedWhale, a model specifically tailored for Korean language processing. RedWhale is developed using an efficient continual pretraining approach that includes a comprehensive Korean corpus preprocessing pipeline. Experimental results demonstrate that RedWhale outperforms other leading models on Korean NLP benchmarks.
  arXiv  Detail & Related papers  (2024-08-21T02:49:41Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities [20.40712512748528]
  Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. We constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus.
  arXiv  Detail & Related papers  (2024-04-27T06:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.