RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

• URL: http://arxiv.org/abs/2608.06347v1
• Date: Thu, 06 Aug 2026 17:52:06 GMT
• Title: RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
• Abstract summary: RP-OPSD is Reasoning-Pivot-guided On-Policy Self-Distillation.<n>We show that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens.
• Score: 31.19784811323349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.

Related papers

• Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs [45.13932034141772]
  We propose textbfEcho- GRPO, a framework that lets the model reason in the words it speaks.<n>We instantiate this framework as textbfVideoEcho-R1 for video reasoning distillation.<n>Our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation.
  arXiv  Detail & Related papers  (2026-08-27T06:41:08Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Efficient Multilingual Reasoning Transfer via Progressive Code-Switching [51.49388869532955]
  Transfer approaches typically rely on distilled target-language reasoning traces from stronger LRMs or online supervision from external judge models.<n>We propose PCS (Progressive Code-Switching), a more efficient transfer framework that requires only lightweight translation.<n> PCS substantially narrows the performance gap between target-language and English reasoning, yielding more language-consistent reasoning.
  arXiv  Detail & Related papers  (2026-07-01T06:13:16Z)
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance [77.58408743830314]
  Reinforcement learning has proven effective for enhancing multi-step reasoning in large language models.<n>But its benefits have not fully translated to multilingual contexts.<n>We develop a novel framework that leverages language-conditioned hints to guide exploration in non-English reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-21T14:47:52Z)
• Crosslingual On-Policy Self-Distillation for Multilingual Reasoning [48.68444770923683]
  Crosslingual On-Policy Self-Distillation (COPSD) transfers a model's own high-resource reasoning behavior to low-resource languages.<n>Experiments on 17 low-resource African languages show that COPSD consistently improves low-resource mathematical reasoning across model sizes.
  arXiv  Detail & Related papers  (2026-05-10T14:06:09Z)
• Align to the Pivot: Dual Alignment with Self-Feedback for Multilingual Math Reasoning [71.4175109189942]
  We present Pivot-Aligned Self-Feedback Multilingual Reasoning (PASMR)<n>This approach designates the model's primary language as the pivot language.<n>It establishes a cross-lingual self-feedback mechanism without relying on external correct answers or reward models.
  arXiv  Detail & Related papers  (2026-01-25T03:20:00Z)
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective [52.452449102961225]
  This study proposes a novel cross-linguistic perspective to investigate reasoning generalization.<n>Our findings reveal that cross-lingual transferability varies significantly across initial model, target language, and training paradigm.<n>Our study challenges the assumption that LRM reasoning mirrors human cognition, providing critical insights for the development of more language-agnostic LRMs.
  arXiv  Detail & Related papers  (2025-10-02T17:49:49Z)
• Towards Better Understanding of Program-of-Thought Reasoning in Cross-Lingual and Multilingual Environments [38.191619790402655]
  Multi-step reasoning is essential for large language models (LLMs), yet multilingual performance remains challenging.<n>While Chain-of-Thought (CoT) prompting improves reasoning, it struggles with non-English languages due to the entanglement of reasoning and execution.<n>We propose a framework to evaluate Program-of-Thought (PoT) prompting, offering a promising alternative but shifting the challenge to generating programs from non-English questions.
  arXiv  Detail & Related papers  (2025-02-25T08:27:28Z)
• AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thought [40.16140566668239]
  We introduce AdaMCOT, a framework that enhances multilingual factual reasoning.<n>AdaMCOT dynamically routing thought processes in intermediary "thinking languages" before generating target-language responses.<n>Our evaluation demonstrates substantial improvements in both factual reasoning quality and cross-lingual consistency.
  arXiv  Detail & Related papers  (2025-01-27T15:48:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.