OutDreamer: Video Outpainting with a Diffusion Transformer

• URL: http://arxiv.org/abs/2506.22298v1
• Date: Fri, 27 Jun 2025 15:08:54 GMT
• Title: OutDreamer: Video Outpainting with a Diffusion Transformer
• Authors: Linhao Zhong, Fan Li, Yi Huang, Jianzhuang Liu, Renjing Pei, Fenglong Song,
• Abstract summary: We introduce OutDreamer, a DiT-based video outpainting framework.<n>We propose a mask-driven self-attention layer that dynamically integrates the given mask information.<n>For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content.
• Score: 37.512451098188635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many state-of-the-art methods utilize latent diffusion models with U-Net backbones but still struggle to achieve high quality and adaptability in generated content. Diffusion transformers (DiTs) have emerged as a promising alternative because of their superior performance. We introduce OutDreamer, a DiT-based video outpainting framework comprising two main components: an efficient video control branch and a conditional outpainting branch. The efficient video control branch effectively extracts masked video information, while the conditional outpainting branch generates missing content based on these extracted conditions. Additionally, we propose a mask-driven self-attention layer that dynamically integrates the given mask information, further enhancing the model's adaptability to outpainting tasks. Furthermore, we introduce a latent alignment loss to maintain overall consistency both within and between frames. For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content, ensuring temporal consistency across video clips. Extensive evaluations demonstrate that our zero-shot OutDreamer outperforms state-of-the-art zero-shot methods on widely recognized benchmarks.

Related papers

• EraserDiT: Fast Video Inpainting with Diffusion Transformer Model [6.616553739135743]
  This paper introduces a novel video inpainting approach leveraging the Diffusion Transformer (DiT)<n>DiT synergistically combines the advantages of diffusion models and transformer architectures to maintain long-term temporal consistency.<n>It takes only 180 seconds to complete a video with a resolution of $1080 1920$ with 121 frames without any acceleration method.
  arXiv  Detail & Related papers  (2025-06-15T13:59:57Z)
• VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control [47.34885131252508]
  Video inpainting aims to restore corrupted video content.<n>We propose a novel dual-stream paradigm VideoPainter to process masked videos.<n>We also introduce a novel target region ID resampling technique that enables any-length video inpainting.
  arXiv  Detail & Related papers  (2025-03-07T17:59:46Z)
• FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors [64.54220123913154]
  We introduce FramePainter as an efficient instantiation of image-to-video generation problem.<n>It only uses a lightweight sparse control encoder to inject editing signals.<n>It domainantly outperforms previous state-of-the-art methods with far less training data.
  arXiv  Detail & Related papers  (2025-01-14T16:09:16Z)
• Replace Anyone in Videos [82.37852750357331]
  We present the ReplaceAnyone framework, which focuses on localized human replacement and insertion featuring intricate backgrounds.<n>We formulate this task as an image-conditioned video inpainting paradigm with pose guidance, utilizing a unified end-to-end video diffusion architecture.<n>The proposed ReplaceAnyone can be seamlessly applied not only to traditional 3D-UNet base models but also to DiT-based video models such as Wan2.1.
  arXiv  Detail & Related papers  (2024-09-30T03:27:33Z)
• Video Diffusion Models are Strong Video Inpainter [14.402778136825642]
  We propose a novel First Frame Filling Video Diffusion Inpainting model (FFF-VDI)<n>We propagate the noise latent information of future frames to fill the masked areas of the first frame's noise latent code.<n>Next, we fine-tune the pre-trained image-to-video diffusion model to generate the inpainted video.
  arXiv  Detail & Related papers  (2024-08-21T08:01:00Z)
• Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptation [44.92712228326116]
  Video outpainting is a challenging task, aiming at generating video content outside the viewport of the input video. We introduce MOTIA Mastering Video Outpainting Through Input-Specific Adaptation. MoTIA comprises two main phases: input-specific adaptation and pattern-aware outpainting.
  arXiv  Detail & Related papers  (2024-03-20T16:53:45Z)
• AVID: Any-Length Video Inpainting with Diffusion Model [30.860927136236374]
  We introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. Our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Our experiments show our model can robustly deal with various inpainting types at different video duration ranges, with high quality.
  arXiv  Detail & Related papers  (2023-12-06T18:56:14Z)
• Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation [93.18163456287164]
  This paper proposes a novel text-guided video-to-video translation framework to adapt image models to videos. Our framework achieves global style and local texture temporal consistency at a low cost.
  arXiv  Detail & Related papers  (2023-06-13T17:52:23Z)
• Occlusion-Aware Video Object Inpainting [72.38919601150175]
  This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. For more realistic results, VOIN is optimized using both T-PatchGAN and a newoc-temporal YouTube attention-based multi-class discriminator.
  arXiv  Detail & Related papers  (2021-08-15T15:46:57Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.