EraserDiT: Fast Video Inpainting with Diffusion Transformer Model

• URL: http://arxiv.org/abs/2506.12853v1
• Date: Sun, 15 Jun 2025 13:59:57 GMT
• Title: EraserDiT: Fast Video Inpainting with Diffusion Transformer Model
• Authors: Jie Liu, Zheng Hui,
• Abstract summary: This paper introduces a novel video inpainting approach leveraging the Diffusion Transformer (DiT)<n>DiT synergistically combines the advantages of diffusion models and transformer architectures to maintain long-term temporal consistency.<n>It takes only 180 seconds to complete a video with a resolution of $1080 1920$ with 121 frames without any acceleration method.
• Score: 6.616553739135743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video object removal and inpainting are critical tasks in the fields of computer vision and multimedia processing, aimed at restoring missing or corrupted regions in video sequences. Traditional methods predominantly rely on flow-based propagation and spatio-temporal Transformers, but these approaches face limitations in effectively leveraging long-term temporal features and ensuring temporal consistency in the completion results, particularly when dealing with large masks. Consequently, performance on extensive masked areas remains suboptimal. To address these challenges, this paper introduces a novel video inpainting approach leveraging the Diffusion Transformer (DiT). DiT synergistically combines the advantages of diffusion models and transformer architectures to maintain long-term temporal consistency while ensuring high-quality inpainting results. We propose a Circular Position-Shift strategy to further enhance long-term temporal consistency during the inference stage. Additionally, the proposed method automatically detects objects within videos, interactively removes specified objects, and generates corresponding prompts. In terms of processing speed, it takes only 180 seconds (testing on one NVIDIA A100 GPU) to complete a video with a resolution of $1080 \times 1920$ with 121 frames without any acceleration method. Experimental results indicate that the proposed method demonstrates superior performance in content fidelity, texture restoration, and temporal consistency. Project page: https://jieliu95.github.io/EraserDiT_demo.

Related papers

• OutDreamer: Video Outpainting with a Diffusion Transformer [37.512451098188635]
  We introduce OutDreamer, a DiT-based video outpainting framework.<n>We propose a mask-driven self-attention layer that dynamically integrates the given mask information.<n>For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content.
  arXiv  Detail & Related papers  (2025-06-27T15:08:54Z)
• DiTPainter: Efficient Video Inpainting with Diffusion Transformers [35.1896530415315]
  We present DiTPainter, an end-to-end video inpainting model based on Diffusion Transformer (DiT)<n>DiTPainter uses an efficient transformer network designed for video inpainting, which is trained from scratch instead of initializing from any large pretrained models.<n>Experiments show that DiTPainter outperforms existing video inpainting algorithms with higher quality and better spatial-temporal consistency.
  arXiv  Detail & Related papers  (2025-04-22T07:36:45Z)
• VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control [47.34885131252508]
  Video inpainting aims to restore corrupted video content.<n>We propose a novel dual-stream paradigm VideoPainter to process masked videos.<n>We also introduce a novel target region ID resampling technique that enables any-length video inpainting.
  arXiv  Detail & Related papers  (2025-03-07T17:59:46Z)
• SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models [10.66567645920237]
  Given an input video of a person and a new garment, the objective of this paper is to synthesize a new video where the person is wearing the garment while maintaining temporal consistency.<n>We reconceptualize video virtual try-on as a conditional video inpainting task, with garments serving as input conditions.<n>Specifically, our approach enhances image diffusion models by incorporating temporal attention layers to improve temporal coherence.
  arXiv  Detail & Related papers  (2024-12-13T14:50:26Z)
• ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
  Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation.<n>We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning.<n>Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
  arXiv  Detail & Related papers  (2024-10-08T03:01:54Z)
• TDViT: Temporal Dilated Video Transformer for Dense Video Tasks [35.16197118579414]
  Temporal Dilated Video Transformer (TDTTB) can efficiently extract video representations and effectively alleviate the negative effect of temporal redundancy. Experiments are conducted on two different dense video benchmarks, i.e., ImageNet VID for video object detection and YouTube VIS for video segmentation instance.
  arXiv  Detail & Related papers  (2024-02-14T15:41:07Z)
• VidToMe: Video Token Merging for Zero-Shot Video Editing [100.79999871424931]
  We propose a novel approach to enhance temporal consistency in generated videos by merging self-attention tokens across frames. Our method improves temporal coherence and reduces memory consumption in self-attention computations.
  arXiv  Detail & Related papers  (2023-12-17T09:05:56Z)
• Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation [115.09597127418452]
  Latent-Shift is an efficient text-to-video generation method based on a pretrained text-to-image generation model. We show that Latent-Shift achieves comparable or better results while being significantly more efficient.
  arXiv  Detail & Related papers  (2023-04-17T17:57:06Z)
• Occlusion-Aware Video Object Inpainting [72.38919601150175]
  This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. For more realistic results, VOIN is optimized using both T-PatchGAN and a newoc-temporal YouTube attention-based multi-class discriminator.
  arXiv  Detail & Related papers  (2021-08-15T15:46:57Z)
• Decoupled Spatial-Temporal Transformer for Video Inpainting [77.8621673355983]
  Video aims to fill the given holes with realistic appearance but is still a challenging task even with prosperous deep learning approaches. Recent works introduce the promising Transformer architecture into deep video inpainting and achieve better performance. We propose a Decoupled Spatial-Temporal Transformer (DSTT) for improving video inpainting with exceptional efficiency.
  arXiv  Detail & Related papers  (2021-04-14T05:47:46Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.