RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers

• URL: http://arxiv.org/abs/2502.15894v1
• Date: Fri, 21 Feb 2025 19:28:05 GMT
• Title: RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers
• Authors: Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, Jun Zhu,
• Abstract summary: RIFLEx is a free lunch--achieving high-quality $2times$ extrapolation on state-of-the-art video diffusion transformers.<n>It enhances quality and enables $3times$ extrapolation by minimal fine-tuning without long videos.
• Score: 29.663251658875673
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in video generation have enabled models to synthesize high-quality, minute-long videos. However, generating even longer videos with temporal coherence remains a major challenge, and existing length extrapolation methods lead to temporal repetition or motion deceleration. In this work, we systematically analyze the role of frequency components in positional embeddings and identify an intrinsic frequency that primarily governs extrapolation behavior. Based on this insight, we propose RIFLEx, a minimal yet effective approach that reduces the intrinsic frequency to suppress repetition while preserving motion consistency, without requiring any additional modifications. RIFLEx offers a true free lunch--achieving high-quality $2\times$ extrapolation on state-of-the-art video diffusion transformers in a completely training-free manner. Moreover, it enhances quality and enables $3\times$ extrapolation by minimal fine-tuning without long videos. Project page and codes: \href{https://riflex-video.github.io/}{https://riflex-video.github.io/.}

Related papers

• FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion [24.48220892418698]
  FreeLong is a training-free framework designed to balance the frequency distribution of long video features during the denoising process.<n>FreeLong achieves this by blending global low-frequency features, which capture holistic semantics across the full video, with local high-frequency features extracted from short temporal windows.<n>FreeLong++ extends FreeLong into a multi-branch architecture with multiple attention branches, each operating at a distinct temporal scale.
  arXiv  Detail & Related papers  (2025-06-30T18:11:21Z)
• DiffuseSlide: Training-Free High Frame Rate Video Generation Diffusion [4.863177884263436]
  We present a training-free approach for high FPS video generation using pre-trained diffusion models.<n>Our method, DiffuseSlide, introduces a new pipeline that leverages key frames from low FPS videos and applies innovative techniques, including noise re-injection and sliding window latent denoising.<n>Through extensive experiments, we demonstrate that our approach significantly improves video quality, offering enhanced temporal coherence and spatial fidelity.
  arXiv  Detail & Related papers  (2025-06-02T09:12:41Z)
• SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models [10.66567645920237]
  Given an input video of a person and a new garment, the objective of this paper is to synthesize a new video where the person is wearing the garment while maintaining temporal consistency.<n>We reconceptualize video virtual try-on as a conditional video inpainting task, with garments serving as input conditions.<n>Specifically, our approach enhances image diffusion models by incorporating temporal attention layers to improve temporal coherence.
  arXiv  Detail & Related papers  (2024-12-13T14:50:26Z)
• MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion [3.7270979204213446]
  We present four key contributions to address the challenges of video processing.<n>First, we introduce the 3D Inverted Vector-Quantization Variencoenco Autocoder.<n>Second, we present MotionAura, a text-to-video generation framework.<n>Third, we propose a spectral transformer-based denoising network.<n>Fourth, we introduce a downstream task of Sketch Guided Videopainting.
  arXiv  Detail & Related papers  (2024-10-10T07:07:56Z)
• FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention [57.651429116402554]
  This paper investigates a straightforward and training-free approach to extend an existing short video diffusion model for consistent long video generation. We find that directly applying the short video diffusion model to generate long videos can lead to severe video quality degradation. Motivated by this, we propose a novel solution named FreeLong to balance the frequency distribution of long video features during the denoising process.
  arXiv  Detail & Related papers  (2024-07-29T11:52:07Z)
• Anchored Diffusion for Video Face Reenactment [17.343307538702238]
  We introduce Anchored Diffusion, a novel method for synthesizing relatively long and seamless videos. We train our model on video sequences with random non-uniform temporal spacing and incorporate temporal information via external guidance. During inference, we leverage the transformer architecture to modify the diffusion process, generating a batch of non-uniform sequences anchored to a common frame.
  arXiv  Detail & Related papers  (2024-07-21T13:14:17Z)
• CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion [15.013908857230966]
  "Look-back" mechanism enhances the fine-grained scene transition between different video clips. Long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one. Experiments have shown the effectiveness of the strategies by performing long-video generation under both single- and multi-text prompt conditions.
  arXiv  Detail & Related papers  (2024-06-07T16:56:42Z)
• StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text [58.49820807662246]
  We introduce StreamingT2V, an autoregressive approach for long video generation of 80, 240, 600, 1200 or more frames with smooth transitions. Our code will be available at: https://github.com/Picsart-AI-Research/StreamingT2V.
  arXiv  Detail & Related papers  (2024-03-21T18:27:29Z)
• Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis [69.83405335645305]
  We argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. We show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity.
  arXiv  Detail & Related papers  (2024-02-22T18:55:08Z)
• FlashVideo: A Framework for Swift Inference in Text-to-Video Generation [9.665089218030086]
  This paper introduces FlashVideo, a novel framework tailored for swift Text-to-Video generation. FlashVideo reduces the time complexity of inference from $mathcalO(L2)$ to $mathcalO(L)$ for a sequence of length $L$, significantly accelerating inference speed. Our comprehensive experiments demonstrate that FlashVideo achieves a $times9.17$ improvement over a traditional autoregressive-based transformer model, and its inference speed is of the same order of magnitude as that of BERT-based transformer models.
  arXiv  Detail & Related papers  (2023-12-30T00:06:28Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling [85.60543452539076]
  Existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. This study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. We propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models.
  arXiv  Detail & Related papers  (2023-10-23T17:59:58Z)
• ControlVideo: Training-free Controllable Text-to-Video Generation [117.06302461557044]
  ControlVideo is a framework to enable natural and efficient text-to-video generation. It generates both short and long videos within several minutes using one NVIDIA 2080Ti.
  arXiv  Detail & Related papers  (2023-05-22T14:48:53Z)
• Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos [69.22032459870242]
  We present a novel technique, Residual Radiance Field or ReRF, as a highly compact neural representation to achieve real-time free-view rendering on long-duration dynamic scenes. We show such a strategy can handle large motions without sacrificing quality. Based on ReRF, we design a special FVV that achieves three orders of magnitudes compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes.
  arXiv  Detail & Related papers  (2023-04-10T08:36:00Z)
• You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
  Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query. Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames. We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
  arXiv  Detail & Related papers  (2023-03-14T12:53:27Z)
• Learning Trajectory-Aware Transformer for Video Super-Resolution [50.49396123016185]
  Video super-resolution aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Existing approaches usually align and aggregate video frames from limited adjacent frames. We propose a novel Transformer for Video Super-Resolution (TTVSR)
  arXiv  Detail & Related papers  (2022-04-08T03:37:39Z)
• Across Scales & Across Dimensions: Temporal Super-Resolution using Deep Internal Learning [11.658606722158517]
  We train a video-specific CNN on examples extracted directly from the low-framerate input video. Our method exploits the strong recurrence of small space-time patches inside a single video sequence. The higher spatial resolution of video frames provides strong examples as to how to increase the temporal temporal resolution of that video.
  arXiv  Detail & Related papers  (2020-03-19T15:53:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.