LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation

• URL: http://arxiv.org/abs/2508.03694v1
• Date: Tue, 05 Aug 2025 17:59:58 GMT
• Title: LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation
• Abstract summary: LongVie is an end-to-end autoregressive framework for controllable long video generation.<n>LongVie achieves state-of-the-art performance in long-range controllability, consistency, and quality.
• Score: 90.02820796070135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable ultra-long video generation is a fundamental yet challenging task. Although existing methods are effective for short clips, they struggle to scale due to issues such as temporal inconsistency and visual degradation. In this paper, we initially investigate and identify three key factors: separate noise initialization, independent control signal normalization, and the limitations of single-modality guidance. To address these issues, we propose LongVie, an end-to-end autoregressive framework for controllable long video generation. LongVie introduces two core designs to ensure temporal consistency: 1) a unified noise initialization strategy that maintains consistent generation across clips, and 2) global control signal normalization that enforces alignment in the control space throughout the entire video. To mitigate visual degradation, LongVie employs 3) a multi-modal control framework that integrates both dense (e.g., depth maps) and sparse (e.g., keypoints) control signals, complemented by 4) a degradation-aware training strategy that adaptively balances modality contributions over time to preserve visual quality. We also introduce LongVGenBench, a comprehensive benchmark consisting of 100 high-resolution videos spanning diverse real-world and synthetic environments, each lasting over one minute. Extensive experiments show that LongVie achieves state-of-the-art performance in long-range controllability, consistency, and quality.

Related papers

• FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matching [68.01498128172214]
  We propose a novel but simple inference-time approach for long video generation that is architecture-agnostic and requires no additional training.<n>Our method generates long videos via overlapping sliding windows, where predicted clean samples from adjacent windows are blended via emphTweedie matching to enforce both textbfmanifold constraint and temporal consistency across overlap regions.<n>Our method generates videos several times longer than the native window length while outperforming both training-free and autoregressive baselines in temporal consistency and visual quality, and further extends to audio-video joint generation and text-to-3DGS without any
  arXiv  Detail & Related papers  (2026-05-20T08:55:37Z)
• UCM: Unifying Camera Control and Memory with Time-aware Positional Encoding Warping for World Models [54.564740558030245]
  We present UCM, a novel framework that unifies long-term memory and precise camera control via a time-aware positional encoding warping mechanism.<n>We also introduce a scalable data curation strategy utilizing point-cloud-based rendering to simulate scene revisiting.
  arXiv  Detail & Related papers  (2026-02-26T12:54:46Z)
• LongVie 2: Multimodal Controllable Ultra-Long Video World Model [94.85126937285527]
  LongVie 2 is an end-to-end autoregressive framework trained in three stages.<n>LongVie 2 achieves state-of-the-art performance in long-range controllability, temporal coherence, and visual fidelity.
  arXiv  Detail & Related papers  (2025-12-15T17:59:58Z)
• KlingAvatar 2.0 Technical Report [43.949604396366425]
  Our model effectively addresses the challenges of efficient, multimodally aligned long-form high-resolution video generation.<n>It delivers enhanced visual clarity, realistic lip-teeth rendering with accurate lip synchronization, strong identity preservation, and coherent multimodal instruction following.
  arXiv  Detail & Related papers  (2025-12-15T13:30:51Z)
• FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion [24.48220892418698]
  FreeLong is a training-free framework designed to balance the frequency distribution of long video features during the denoising process.<n>FreeLong achieves this by blending global low-frequency features, which capture holistic semantics across the full video, with local high-frequency features extracted from short temporal windows.<n>FreeLong++ extends FreeLong into a multi-branch architecture with multiple attention branches, each operating at a distinct temporal scale.
  arXiv  Detail & Related papers  (2025-06-30T18:11:21Z)
• DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer [56.98400572837792]
  DiVE produces high-fidelity, temporally coherent, and cross-view consistent multi-view videos.<n>These innovations collectively achieve a 2.62x speedup with minimal quality degradation.
  arXiv  Detail & Related papers  (2025-04-28T09:20:50Z)
• Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Sampling [81.37449968164692]
  We propose Synchronized Coupled Sampling (SynCoS), a novel inference framework that synchronizes denoising paths across the entire video.<n>Our approach combines two complementary sampling strategies, which ensure seamless local transitions and enforce global coherence.<n>Extensive experiments show that SynCoS significantly improves multi-event long video generation, achieving smoother transitions and superior long-range coherence.
  arXiv  Detail & Related papers  (2025-03-11T16:43:45Z)
• Tuning-Free Long Video Generation via Global-Local Collaborative Diffusion [22.988212617368095]
  We propose GLC-Diffusion, a tuning-free method for long video generation.<n>It models the long video denoising process by establishing Global-Local Collaborative Denoising.<n>We also propose a Video Motion Consistency Refinement (VMCR) module that computes the gradient of pixel-wise and frequency-wise losses.
  arXiv  Detail & Related papers  (2025-01-08T05:49:39Z)
• FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention [57.651429116402554]
  This paper investigates a straightforward and training-free approach to extend an existing short video diffusion model for consistent long video generation. We find that directly applying the short video diffusion model to generate long videos can lead to severe video quality degradation. Motivated by this, we propose a novel solution named FreeLong to balance the frequency distribution of long video features during the denoising process.
  arXiv  Detail & Related papers  (2024-07-29T11:52:07Z)
• Encoding and Controlling Global Semantics for Long-form Video Question Answering [40.129800076300434]
  We introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length.
  arXiv  Detail & Related papers  (2024-05-30T06:10:10Z)
• ControlVideo: Training-free Controllable Text-to-Video Generation [117.06302461557044]
  ControlVideo is a framework to enable natural and efficient text-to-video generation. It generates both short and long videos within several minutes using one NVIDIA 2080Ti.
  arXiv  Detail & Related papers  (2023-05-22T14:48:53Z)
• Generating Long Videos of Dynamic Scenes [66.56925105992472]
  We present a video generation model that reproduces object motion, changes in camera viewpoint, and new content that arises over time. A common failure case is for content to never change due to over-reliance on inductive biases to provide temporal consistency.
  arXiv  Detail & Related papers  (2022-06-07T16:29:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.