Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation

• URL: http://arxiv.org/abs/2608.13391v1
• Date: Thu, 13 Aug 2026 15:51:31 GMT
• Title: Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation
• Abstract summary: Context-Matched Distillation (CMD) is a causal DMD framework that aligns teacher supervision with the information available when each target is generated.<n>Prefix Corruption stabilizes training by perturbing unreliable prefixes produced early in training while preserving this target-context alignment.<n> Experiments demonstrate state-of-the-art aggregate performance among autoregressive methods on both short- and long-video benchmarks.
• Score: 97.02922768457239
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive autoregressive video generation demands both low-latency rollouts and precise online control. Few-step distillation accelerates generation by reducing denoising steps, while online control imposes a causal constraint: frames and blocks should depend on history and controls available during generation. Existing video distribution matching distillation (DMD) pipelines, however, often supervise causal few-step students using bidirectional teachers that score complete clips. The score for a target can therefore depend on future frames and controls that were unavailable when the student generated it, misaligning teacher supervision with the student's causal information set. We introduce Context-Matched Distillation (CMD), a causal DMD framework that aligns teacher supervision with the information available when each target is generated. CMD replaces bidirectional full-clip scoring with a causal teacher that evaluates each target without access to future frames or controls. The same causal teacher initializes the few-step student, establishing a consistent causal formulation across teacher training, student distillation, and inference. Beyond aligning the temporal information boundary, Prefix Scoring matches supervision to the student's realized rollout context by evaluating each target under the cached student-generated prefix that produced it. Prefix Corruption further stabilizes training by perturbing unreliable prefixes produced early in training while preserving this target-context alignment. With a simple causal formulation, CMD naturally extends to frame-wise and chunk-wise generation, long video distillation, and camera-conditioned distillation. Experiments demonstrate state-of-the-art aggregate performance among autoregressive methods on both short- and long-video benchmarks, together with substantially improved adherence to time-varying camera controls.

Related papers

• OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models [39.15861870749305]
  We propose OPTD, On-Policy Transition Distillation with consistency-guided adaptive compression.<n>It samples partial states from the few-step student's own trajectories, uses a frozen, question-only teacher to identify outcome-aligned future candidates, and orders them by current-state confidence.<n>It consistently improves the quality--efficiency trade-off and attains the strongest overall quality-constrained AUP among the evaluated few-step baselines.
  arXiv  Detail & Related papers  (2026-08-03T23:09:43Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• On-Policy Adversarial Flow Distillation for Autoregressive Video Generation [34.70887671905755]
  Adversarial Flow Distillation (AFD) is an on-policy framework for heterogeneous black-box video distillation.<n>AFD queries the teacher and rolls out the current student on the same prompts.<n>AFD provides dense velocity-field supervision while requiring no teacher scores, latents, denoising trajectories, step alignment, or reverse-chain reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-25T17:58:12Z)
• Distribution Corrected Offline Data Distillation for Large Language Models [8.778789513095907]
  Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings.<n>Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift.<n>We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher-student distribution drift.
  arXiv  Detail & Related papers  (2026-05-13T19:47:31Z)
• Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning [2.9029480541171107]
  We propose adaptive teacher exposure as an effective new axis for reasoning self-distillation.<n>We show that Adaptive Teacher Exposure for Self-Distillation (ATESD) consistently outperforms competitive self-distillation and RL baselines.
  arXiv  Detail & Related papers  (2026-05-12T03:15:58Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models [50.986189632485285]
  We introduce Backwards Aggregation (BAgger), a self-supervised scheme that constructs corrective trajectories from the model's own rollouts.<n>Unlike prior approaches that rely on few-step distillation and distribution-matching losses, BAgger trains with standard score or flow matching objectives.<n>We instantiate BAgger on causal diffusion transformers and evaluate on text-to-video, video extension, and multi-prompt generation.
  arXiv  Detail & Related papers  (2025-12-12T23:02:02Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer [14.714768026997534]
  This paper aims to stabilize DETR training through the online distillation. It utilizes a teacher model, accumulated by Exponential Moving Average (EMA) Experiments show that the proposed OD-DETR successfully stabilizes the training, and significantly increases the performance without bringing in more parameters.
  arXiv  Detail & Related papers  (2024-06-09T14:07:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.