Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models

• URL: http://arxiv.org/abs/2603.17051v1
• Date: Tue, 17 Mar 2026 18:32:18 GMT
• Title: Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models
• Abstract summary: Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
• Score: 58.3184497327891
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-distillation or solver-coupled reverse-process optimization that introduces considerable memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this alignment to long videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR video models, serving as a robust and scalable alignment solution.

Related papers

• Scaling Reinforcement Learning for Diffusion Models via Velocity Matching [66.34541945134916]
  RewardBench fine-tuning is an important tool for adapting diffusion models to human preferences and task-specific objectives.<n>Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples.<n>We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field.
  arXiv  Detail & Related papers  (2026-08-24T17:13:43Z)
• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• Explicit Critic Guidance for Aligning Diffusion Models [27.6704888666835]
  We propose a state-aligned latent actor-critic framework for diffusion post-training.<n>Our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T22:20:51Z)
• RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO [53.38929612273108]
  Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content.<n>We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints.
  arXiv  Detail & Related papers  (2026-05-14T17:59:30Z)
• Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers [12.948398661304184]
  Diffusion-APO is a trajectory-aware algorithm for aligning video diffusion models with human intent.<n>We introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction.<n>We demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following.
  arXiv  Detail & Related papers  (2026-05-08T09:37:46Z)
• Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model [26.672519467929686]
  We present the first autoregressive (AR) models tailored for streaming Target Speaker Extraction (TSE)<n>Our approach introduces a Chunk-wise Interleaved Splicing Paradigm that ensures highly efficient and stable streaming inference.<n>Experiments on Libri2Mix show that while AR generative baseline exhibits performance degradation at low latencies, our approach maintains 100% stability and superior intelligibility.
  arXiv  Detail & Related papers  (2026-04-21T16:25:22Z)
• Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance [46.06527859746679]
  We introduce Reinforcement Learning Guidance (RLG), an inference-time method that adapts Dejin-Free Guidance (CFG)<n>RLG consistently improves the performance of RL fine-tuned models across various, RL algorithms, and downstream tasks, including human preferences, compositional control, compress, and text rendering.<n>Our approach provides a practical and theoretically sound solution for enhancing and controlling diffusion model alignment inference.
  arXiv  Detail & Related papers  (2025-08-28T17:18:31Z)
• SwiftVideo: A Unified Framework for Few-Step Video Generation through Trajectory-Distribution Alignment [76.60024640625478]
  Diffusion-based or flow-based models have achieved significant progress in video synthesis but require multiple iterative sampling steps.<n>We propose a unified and stable distillation framework that combines the advantages of trajectory-preserving and distribution-matching strategies.<n>Our method maintains high-quality video generation while substantially reducing the number of inference steps.
  arXiv  Detail & Related papers  (2025-08-08T07:26:34Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [67.94300151774085]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• ROCM: RLHF on consistency models [8.905375742101707]
  We propose a reward optimization framework for applying RLHF to consistency models.<n>We investigate various $f$-divergences as regularization strategies, striking a balance between reward and model consistency.
  arXiv  Detail & Related papers  (2025-03-08T11:19:48Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.