Motion-Aware Generative Frame Interpolation

• URL: http://arxiv.org/abs/2501.03699v2
• Date: Sat, 08 Mar 2025 08:22:34 GMT
• Title: Motion-Aware Generative Frame Interpolation
• Authors: Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, Limin Wang,
• Abstract summary: Flow-based frame methods ensure motion stability through estimated intermediate flow but often introduce severe artifacts in complex motion regions.<n>Recent generative approaches, boosted by large-scale pre-trained video generation models, show promise in handling intricate scenes.<n>We propose Motion-aware Generative frame (MoG) that synergizes intermediate flow guidance with generative capacities to enhance fidelity.
• Score: 23.380470636851022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Flow-based frame interpolation methods ensure motion stability through estimated intermediate flow but often introduce severe artifacts in complex motion regions. Recent generative approaches, boosted by large-scale pre-trained video generation models, show promise in handling intricate scenes. However, they frequently produce unstable motion and content inconsistencies due to the absence of explicit motion trajectory constraints. To address these challenges, we propose Motion-aware Generative frame interpolation (MoG) that synergizes intermediate flow guidance with generative capacities to enhance interpolation fidelity. Our key insight is to simultaneously enforce motion smoothness through flow constraints while adaptively correcting flow estimation errors through generative refinement. Specifically, we first introduce a dual guidance injection that propagates condition information using intermediate flow at both latent and feature levels, aligning the generated motion with flow-derived motion trajectories. Meanwhile, we implemented two critical designs, encoder-only guidance injection and selective parameter fine-tuning, which enable dynamic artifact correction in the complex motion regions. Extensive experiments on both real-world and animation benchmarks demonstrate that MoG outperforms state-of-the-art methods in terms of video quality and visual fidelity. Our work bridges the gap between flow-based stability and generative flexibility, offering a versatile solution for frame interpolation across diverse scenarios.

Related papers

• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech. The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture. To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• MotionDiff: Training-free Zero-shot Interactive Motion Editing via Flow-assisted Multi-view Diffusion [20.142107033583027]
  MotionDiff is a training-free zero-shot diffusion method that leverages optical flow for complex multi-view motion editing. It outperforms other physics-based generative motion editing methods in achieving high-quality multi-view consistent motion results. MotionDiff does not require retraining, enabling users to conveniently adapt it for various down-stream tasks.
  arXiv  Detail & Related papers  (2025-03-22T08:32:56Z)
• Leader and Follower: Interactive Motion Generation under Trajectory Constraints [42.90788442575116]
  This paper explores the motion range refinement process in interactive motion generation. It proposes a training-free approach, integrating a Pace Controller and a Kinematic Synchronization Adapter. Experimental results show that the proposed approach, by better leveraging trajectory information, outperforms existing methods in both realism and accuracy.
  arXiv  Detail & Related papers  (2025-02-17T08:52:45Z)
• Generalizable Implicit Motion Modeling for Video Frame Interpolation [51.966062283735596]
  Motion is critical in flow-based Video Frame Interpolation (VFI) We introduce General Implicit Motion Modeling (IMM), a novel and effective approach to motion modeling VFI. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion.
  arXiv  Detail & Related papers  (2024-07-11T17:13:15Z)
• Disentangled Motion Modeling for Video Frame Interpolation [40.83962594702387]
  Video Frame Interpolation (VFI) aims to synthesize intermediate frames between existing frames to enhance visual smoothness and quality.<n>We introduce disentangled Motion Modeling (MoMo), a diffusion-based approach for VFI that enhances visual quality by focusing on intermediate motion modeling.
  arXiv  Detail & Related papers  (2024-06-25T03:50:20Z)
• Motion-aware Latent Diffusion Models for Video Frame Interpolation [51.78737270917301]
  Motion estimation between neighboring frames plays a crucial role in avoiding motion ambiguity. We propose a novel diffusion framework, motion-aware latent diffusion models (MADiff) Our method achieves state-of-the-art performance significantly outperforming existing approaches.
  arXiv  Detail & Related papers  (2024-04-21T05:09:56Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models [75.20168902300166]
  We propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects. generated video sequences by our TrackDiffusion can be used as training data for visual perception models.
  arXiv  Detail & Related papers  (2023-12-01T15:24:38Z)
• Enhanced Bi-directional Motion Estimation for Video Frame Interpolation [0.05541644538483946]
  We present a novel yet effective algorithm for motion-based video frame estimation. Our method achieves excellent performance on a broad range of video frame benchmarks.
  arXiv  Detail & Related papers  (2022-06-17T06:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.