LumosFlow: Motion-Guided Long Video Generation

• URL: http://arxiv.org/abs/2506.02497v1
• Date: Tue, 03 Jun 2025 06:25:00 GMT
• Title: LumosFlow: Motion-Guided Long Video Generation
• Authors: Jiahao Chen, Hangjie Yuan, Yichen Qian, Jingyun Liang, Jiazheng Xing, Pengwei Liu, Weihua Chen, Fan Wang, Bing Su,
• Abstract summary: Long video generation has gained increasing attention due to its widespread applications in fields such as entertainment and simulation.<n>We revisit the hierarchical long video generation pipeline and introduce LumosFlow, a framework introduce motion guidance explicitly.<n>Compared with traditional video frame, we achieve 15x, ensuring reasonable and continuous motion between adjacent frames.
• Score: 31.63126037070182
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long video generation has gained increasing attention due to its widespread applications in fields such as entertainment and simulation. Despite advances, synthesizing temporally coherent and visually compelling long sequences remains a formidable challenge. Conventional approaches often synthesize long videos by sequentially generating and concatenating short clips, or generating key frames and then interpolate the intermediate frames in a hierarchical manner. However, both of them still remain significant challenges, leading to issues such as temporal repetition or unnatural transitions. In this paper, we revisit the hierarchical long video generation pipeline and introduce LumosFlow, a framework introduce motion guidance explicitly. Specifically, we first employ the Large Motion Text-to-Video Diffusion Model (LMTV-DM) to generate key frames with larger motion intervals, thereby ensuring content diversity in the generated long videos. Given the complexity of interpolating contextual transitions between key frames, we further decompose the intermediate frame interpolation into motion generation and post-hoc refinement. For each pair of key frames, the Latent Optical Flow Diffusion Model (LOF-DM) synthesizes complex and large-motion optical flows, while MotionControlNet subsequently refines the warped results to enhance quality and guide intermediate frame generation. Compared with traditional video frame interpolation, we achieve 15x interpolation, ensuring reasonable and continuous motion between adjacent frames. Experiments show that our method can generate long videos with consistent motion and appearance. Code and models will be made publicly available upon acceptance. Our project page: https://jiahaochen1.github.io/LumosFlow/

Related papers

• Video Motion Graphs [17.57582826585202]
  We present Motion Graphs, a system designed to generate realistic human motion videos.<n>The system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating frames to seamlessly connect clip boundaries.
  arXiv  Detail & Related papers  (2025-03-26T04:20:14Z)
• Generative Inbetweening through Frame-wise Conditions-Driven Video Generation [63.43583844248389]
  generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input.<n>We propose a Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames.<n>Our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear curves.
  arXiv  Detail & Related papers  (2024-12-16T13:19:41Z)
• Anchored Diffusion for Video Face Reenactment [17.343307538702238]
  We introduce Anchored Diffusion, a novel method for synthesizing relatively long and seamless videos. We train our model on video sequences with random non-uniform temporal spacing and incorporate temporal information via external guidance. During inference, we leverage the transformer architecture to modify the diffusion process, generating a batch of non-uniform sequences anchored to a common frame.
  arXiv  Detail & Related papers  (2024-07-21T13:14:17Z)
• MoVideo: Motion-Aware Video Generation with Diffusion Models [97.03352319694795]
  We propose a novel motion-aware generation (MoVideo) framework that takes motion into consideration from two aspects: video depth and optical flow. MoVideo achieves state-of-the-art results in both text-to-video and image-to-video generation, showing promising prompt consistency, frame consistency and visual quality.
  arXiv  Detail & Related papers  (2023-11-19T13:36:03Z)
• VideoComposer: Compositional Video Synthesis with Motion Controllability [52.4714732331632]
  VideoComposer allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. We introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs.
  arXiv  Detail & Related papers  (2023-06-03T06:29:02Z)
• ControlVideo: Training-free Controllable Text-to-Video Generation [117.06302461557044]
  ControlVideo is a framework to enable natural and efficient text-to-video generation. It generates both short and long videos within several minutes using one NVIDIA 2080Ti.
  arXiv  Detail & Related papers  (2023-05-22T14:48:53Z)
• Towards Smooth Video Composition [59.134911550142455]
  Video generation requires consistent and persistent frames with dynamic content over time. This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs) We show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality.
  arXiv  Detail & Related papers  (2022-12-14T18:54:13Z)
• TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation [50.49396123016185]
  Video frame (VFI) aims to synthesize an intermediate frame between two consecutive frames. We propose a novel Trajectory-aware Transformer for Video Frame Interpolation (TTVFI) Our method outperforms other state-of-the-art methods in four widely-used VFI benchmarks.
  arXiv  Detail & Related papers  (2022-07-19T03:37:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.