Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

• URL: http://arxiv.org/abs/2602.02994v1
• Date: Tue, 03 Feb 2026 02:05:48 GMT
• Title: Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation
• Abstract summary: Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG)<n>We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation.
• Score: 29.755136665244805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG) due to its on-policy optimization, yet existing GRPO-based methods remain fundamentally constrained by sparse reward signals and substantial computational overhead. We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation. Video-OPD optimizes trajectories sampled directly from the current policy, thereby preserving alignment between training and inference distributions, while a frontier teacher supplies dense, token-level supervision via a reverse KL divergence objective. This formulation preserves the on-policy property critical for mitigating distributional shift, while converting sparse, episode-level feedback into fine-grained, step-wise learning signals. Building on Video-OPD, we introduce Teacher-Validated Disagreement Focusing (TVDF), a lightweight training curriculum that iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency. Empirical results demonstrate that Video-OPD consistently outperforms GRPO while achieving substantially faster convergence and lower computational cost, establishing on-policy distillation as an effective alternative to conventional reinforcement learning for TVG.

Related papers

• Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models [58.45264308304815]
  On-policy self-distillation (OPSD) has recently emerged as an effective post-training paradigm that improves policy optimization through dense token-level supervision from a privileged self-teacher.<n>We present $textbfVideo-OPSD$, an OPSD framework that exploits privileged visual evidence for both self-teacher construction and knowledge transfer.
  arXiv  Detail & Related papers  (2026-08-27T12:51:32Z)
• Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies [5.170807667319543]
  We propose a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding.<n>Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering.
  arXiv  Detail & Related papers  (2026-05-31T10:40:28Z)
• Extreme Region Policy Distillation [36.61472284280031]
  We show that aggressive multi-step optimization brings rapid initial gains, but excessive updates cause trajectory probabilities to deviate and entropy to collapse.<n>This motivates Extreme Region Policy Distillation (ERPD), a two-stage framework that decouples sample efficiency from KL efficiency.
  arXiv  Detail & Related papers  (2026-05-25T08:32:24Z)
• RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO [53.38929612273108]
  Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content.<n>We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints.
  arXiv  Detail & Related papers  (2026-05-14T17:59:30Z)
• A Systematic Post-Train Framework for Video Generation [76.26555417456773]
  Large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content.<n>A significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs.<n>We propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages.
  arXiv  Detail & Related papers  (2026-04-28T09:34:51Z)
• Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [29.56905427210088]
  gradient-ARM is a framework that jointly optimize a rubric generator and a judge using reinforcement learning from preference feedback.<n>We show that gradient-ARM achieves state-of-the-art performance among baselines on benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
  arXiv  Detail & Related papers  (2026-02-02T00:50:53Z)
• TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment [28.18756041538092]
  We present TAGRPO, a robust framework for I2V models inspired by contrastive learning.<n>Our approach is grounded in the observation that rollout videos generated from identical initial noise provide superior guidance for optimization.
  arXiv  Detail & Related papers  (2026-01-09T11:15:27Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency [56.475612147721264]
  We propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals.<n>We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA.<n>Results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T17:28:26Z)
• VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization [59.39976343879587]
  VerIPO aims to gradually improve video LLMs' capacity for generating deep, long-term reasoning chains.<n>The training loop benefits from GRPO's expansive search and DPO's targeted optimization.<n>Our trained models exceed the direct inference of large-scale instruction-tuned Video-LLMs.
  arXiv  Detail & Related papers  (2025-05-25T06:41:28Z)
• Reinforcement Learning with Action-Free Pre-Training from Videos [95.25074614579646]
  We introduce a framework that learns representations useful for understanding the dynamics via generative pre-training on videos. Our framework significantly improves both final performances and sample-efficiency of vision-based reinforcement learning.
  arXiv  Detail & Related papers  (2022-03-25T19:44:09Z)
• Accelerating Deep Reinforcement Learning With the Aid of Partial Model: Energy-Efficient Predictive Video Streaming [97.75330397207742]
  Predictive power allocation is conceived for energy-efficient video streaming over mobile networks using deep reinforcement learning. To handle the continuous state and action spaces, we resort to deep deterministic policy gradient (DDPG) algorithm. Our simulation results show that the proposed policies converge to the optimal policy that is derived based on perfect large-scale channel prediction.
  arXiv  Detail & Related papers  (2020-03-21T17:36:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.