SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning

• URL: http://arxiv.org/abs/2603.17426v1
• Date: Wed, 18 Mar 2026 07:04:02 GMT
• Title: SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning
• Abstract summary: We study the problem of motion alignment in video diffusion models post-training. To address this, we introduce pixel-motion rewards based on pixel flux dynamics.<n>We propose Smooth Hybrid Fine-tuning (SHIFT), a scalable reward-driven fine-tuning framework for video diffusion models.
• Score: 47.36837823291006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-conditioned Video diffusion models achieve impressive visual realism but often suffer from weakened motion fidelity, e.g., reduced motion dynamics or degraded long-term temporal coherence, especially after fine-tuning. We study the problem of motion alignment in video diffusion models post-training. To address this, we introduce pixel-motion rewards based on pixel flux dynamics, capturing both instantaneous and long-term motion consistency. We further propose Smooth Hybrid Fine-tuning (SHIFT), a scalable reward-driven fine-tuning framework for video diffusion models. SHIFT fuses the normal supervised fine-tuning and advantage weighted fine-tuning into a unified framework. Benefiting from novel adversarial advantages, SHIFT improves convergence speed and mitigates reward hacking. Experiments show that our approach efficiently resolves dynamic-degree collapse in modern video diffusion models supervised fine-tuning.

Related papers

• SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models [53.82339966194578]
  Motion entanglement is the unintended coupling of independent motion sources, such as camera movement and object motion.<n>We propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos.
  arXiv  Detail & Related papers  (2026-06-26T05:39:40Z)
• TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion [30.705936075333465]
  We show that incoherent videos consistently exhibit irregular, fragmented temporal diagonals in their intermediate self-attention maps.<n>We introduce TeDiO, a training-free, inference-time method that reinforces temporal consistency by regularizing these internal attention patterns.
  arXiv  Detail & Related papers  (2026-05-13T21:39:50Z)
• Causal Motion Diffusion Models for Autoregressive Motion Generation [19.61051102039212]
  Causal Motion Diffusion Models (CMDM) is a unified framework for autoregressive motion generation.<n> CMDM builds upon a Motion-Language-Aligned Causal VAE (MAC-VAE), which encodes motion sequences into temporally causal latent representations.<n>Experiments on HumanML3D and SnapMoGen demonstrate CMDM outperforms existing diffusion and autoregressive models in both semantic fidelity and temporal smoothness.
  arXiv  Detail & Related papers  (2026-02-26T03:58:25Z)
• MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models [50.162882483045045]
  We propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder.<n>This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics.<n>We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos.
  arXiv  Detail & Related papers  (2025-10-21T19:05:23Z)
• Image Motion Blur Removal in the Temporal Dimension with Video Diffusion Models [3.052019331122618]
  We propose a novel single-image deblurring approach that treats motion blur as a temporal averaging phenomenon.<n>Our core innovation lies in leveraging a pre-trained video diffusion transformer model to capture diverse motion dynamics.<n> Empirical results on synthetic and real-world datasets demonstrate that our method outperforms existing techniques in deblurring complex motion blur scenarios.
  arXiv  Detail & Related papers  (2025-01-22T03:01:54Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms.<n> SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics.<n>Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• LaMD: Latent Motion Diffusion for Image-Conditional Video Generation [63.34574080016687]
  latent motion diffusion (LaMD) framework consists of a motion-decomposed video autoencoder and a diffusion-based motion generator.<n>LaMD generates high-quality videos on various benchmark datasets, including BAIR, Landscape, NATOPS, MUG and CATER-GEN.
  arXiv  Detail & Related papers  (2023-04-23T10:32:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.