PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners

• URL: http://arxiv.org/abs/2604.26573v1
• Date: Wed, 29 Apr 2026 11:56:07 GMT
• Title: PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners
• Abstract summary: Large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level.<n>Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context.<n>We propose PAINT, which masks the verified solution according to rollout-reference overlap and applies a small energy-space on a sparse set of entropy-mismatch token positions.
• Score: 9.324642081509756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Improving large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level. Reinforcement learning with verifiable rewards provides on-policy exploration but offers sparse, high-variance credit; supervised fine-tuning and distillation provide dense targets but often train on fixed trajectories or rely on stronger teachers. Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context. We revisit this setting through a contextual re-scoring lens: for reasoning, the important choices are not only whether privileged context is available, but how much of it should be revealed and where its distribution should shape the student. We propose PAINT (Partial-solution Adaptive INterpolated Training), which masks the verified solution according to rollout-reference overlap and applies a small energy-space interpolation on a sparse set of entropy-mismatch token positions. Across competition-level math benchmarks, PAINT consistently improves over a strong prior on-policy self-distillation baseline at all three Qwen3 scales. On Qwen3-8B, it raises macro Avg@12 by 2.1 points over this prior baseline and 2.9 points over GRPO.

Related papers

• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity [5.35852583397955]
  On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student.<n>We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten.
  arXiv  Detail & Related papers  (2026-06-24T17:59:02Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning [12.363923974156636]
  Reinforcement learning from verifiable rewards assigns a single scalar to each rollout.<n>On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information.<n>We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout.
  arXiv  Detail & Related papers  (2026-06-14T03:37:27Z)
• Self-Distilled Policy Gradient [51.95967256628261]
  On-policy self-distillation is a promising source of dense supervision for sparse-reward reinforcement learning.<n>We propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation.
  arXiv  Detail & Related papers  (2026-06-02T02:31:13Z)
• Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
  Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
  arXiv  Detail & Related papers  (2026-05-06T15:31:50Z)
• Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance [8.66471442661456]
  We show that hidden-state distributions contain a useful signal for local reasoning quality.<n>Within each GRPO group, the Wasserstein distance between span-level hidden state distributions increases around regions where their local reasoning quality diverges.<n>We propose textbfSpan-level textbfHidden state textbfEd textbfAdvantage textbfReweighting.
  arXiv  Detail & Related papers  (2026-04-25T14:11:23Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Find, Fix, Reason: Context Repair for Video Reasoning [45.021693494492666]
  Reinforcement learning has advanced video reasoning in large multi-modal models.<n>We introduce an observation-level intervention: a frozen, tool-integrated teacher identifies the missingtemporal dependency and provides a minimal evidence patch.<n>We propose a Robust Improvement Reward (RIR) that aligns optimization with two goals: outcome validity through correct answers and dependency alignment rationales.
  arXiv  Detail & Related papers  (2026-04-17T17:04:19Z)
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting [17.504616835765617]
  We propose a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths.<n>SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.
  arXiv  Detail & Related papers  (2026-04-12T15:26:14Z)
• Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models [38.47027398567909]
  Perception-Grounded Policy Optimization (PGPO) is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level.<n>We show that PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors.<n>Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2026-04-02T09:53:20Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)
• Self-Point-Flow: Self-Supervised Scene Flow Estimation from Point Clouds with Optimal Transport and Random Walk [59.87525177207915]
  We develop a self-supervised method to establish correspondences between two point clouds to approximate scene flow. Our method achieves state-of-the-art performance among self-supervised learning methods.
  arXiv  Detail & Related papers  (2021-05-18T03:12:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.