On the Geometry of On-Policy Distillation

• URL: http://arxiv.org/abs/2606.07082v2
• Date: Wed, 10 Jun 2026 09:18:33 GMT
• Title: On the Geometry of On-Policy Distillation
• Abstract summary: We study on-policy distillation (OPD) to improve large language model reasoning.<n>We compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR)<n>We find that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.
• Score: 22.873898953554605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.

Related papers

• Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing [16.45880716714667]
  Domain-aware Relaxed Orthogonal Subspace adaptation (DROS)<n>DROS reformulates low-rank adaptation as data-conditioned subspace learning and flexible subspace adaptation.<n>Experiments on multiple remote sensing benchmarks demonstrate that DROS achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-09-12T02:19:11Z)
• Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging [35.66009296523317]
  Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT)<n>Recent studies reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT.<n>We propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces.
  arXiv  Detail & Related papers  (2026-06-16T22:22:40Z)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation [51.210887267509854]
  We argue that OPD's efficiency stems from a form of foresight'': it establishes a stable update trajectory toward the final model early in training.<n>We propose textbfEffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction.
  arXiv  Detail & Related papers  (2026-05-12T08:19:15Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration [88.7332439088792]
  scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm.<n>We propose the textbfNonlinear textbfExtrapolation of low-rank trajectories (textbfNExt)<n>Our method reduces computational overhead by approximately 37.5% while remaining compatible with a wide range of RLVR algorithms and tasks.
  arXiv  Detail & Related papers  (2026-04-13T13:28:12Z)
• On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning [14.815508281465272]
  We show that forgetting is strongly influenced by the geometry and parameterization of the update subspace.<n>Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.
  arXiv  Detail & Related papers  (2026-03-10T13:53:25Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• Trust-Region Adaptive Policy Optimization [82.09255251747818]
  Post-training methods play an important role in improving large language models' (LLMs) complex reasoning abilities.<n>We introduce TRAPO, a framework that interleavesSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) within each training instance.<n>Experiments on five mathematical reasoning benchmarks show that TRAPO consistently surpasses standard SFT, RL, and SFT-then-RL pipelines.
  arXiv  Detail & Related papers  (2025-12-19T14:37:07Z)
• The Path Not Taken: RLVR Provably Learns Off the Principals [85.41043469428365]
  We show that sparsity is a surface artifact of a model-conditioned optimization bias.<n>We mechanistically explain these dynamics with a Three-Gate Theory.<n>We provide a parameter-level characterization of RLVR's learning dynamics.
  arXiv  Detail & Related papers  (2025-11-11T18:49:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.