TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration

• URL: http://arxiv.org/abs/2501.02269v1
• Date: Sat, 04 Jan 2025 12:15:37 GMT
• Title: TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration
• Authors: Yizhou Li, Zihua Liu, Yusuke Monno, Masatoshi Okutomi,
• Abstract summary: Our method can restore various types of video degradation with a single unified model. Our method advances the video restoration task by providing a unified solution that enhances video quality across multiple applications.
• Score: 13.49297560533422
• License:
• Abstract: In this paper, we propose the first diffusion-based all-in-one video restoration method that utilizes the power of a pre-trained Stable Diffusion and a fine-tuned ControlNet. Our method can restore various types of video degradation with a single unified model, overcoming the limitation of standard methods that require specific models for each restoration task. Our contributions include an efficient training strategy with Task Prompt Guidance (TPG) for diverse restoration tasks, an inference strategy that combines Denoising Diffusion Implicit Models~(DDIM) inversion with a novel Sliding Window Cross-Frame Attention (SW-CFA) mechanism for enhanced content preservation and temporal consistency, and a scalable pipeline that makes our method all-in-one to adapt to different video restoration tasks. Through extensive experiments on five video restoration tasks, we demonstrate the superiority of our method in generalization capability to real-world videos and temporal consistency preservation over existing state-of-the-art methods. Our method advances the video restoration task by providing a unified solution that enhances video quality across multiple applications.

Related papers

• SVFR: A Unified Framework for Generalized Video Face Restoration [86.17060212058452]
  Face Restoration (FR) is a crucial area within image and video processing, focusing on reconstructing high-quality portraits from degraded inputs. We propose a novel approach for the Generalized Video Face Restoration task, which integrates video BFR, inpainting, and colorization tasks. This work advances the state-of-the-art in video FR and establishes a new paradigm for generalized video face restoration.
  arXiv  Detail & Related papers  (2025-01-02T12:51:20Z)
• VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models [58.464465016269614]
  In this paper, we propose a framework for solving high-definition video inverse problems using latent image diffusion models. Our approach leverages latent-space diffusion models to achieve enhanced video quality and resolution. Unlike previous methods, our approach supports multiple aspect ratios and delivers HD-resolution reconstructions in under 2.5 minutes on a single GPU.
  arXiv  Detail & Related papers  (2024-11-29T08:10:49Z)
• Learning Truncated Causal History Model for Video Restoration [14.381907888022615]
  TURTLE learns the truncated causal history model for efficient and high-performing video restoration. We report new state-of-the-art results on a multitude of video restoration benchmark tasks.
  arXiv  Detail & Related papers  (2024-10-04T21:31:02Z)
• UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation [50.27688690379488]
  Existing unified methods treat multi-degradation image restoration as a multi-task learning problem. We propose a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks.
  arXiv  Detail & Related papers  (2024-09-30T11:16:56Z)
• DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models [9.145545884814327]
  This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. We show that our method achieves top performance in zero-shot video restoration. Our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining.
  arXiv  Detail & Related papers  (2024-07-01T17:59:12Z)
• ConVRT: Consistent Video Restoration Through Turbulence with Test-time Optimization of Neural Video Representations [13.38405890753946]
  We introduce a self-supervised method, Consistent Video Restoration through Turbulence (ConVRT) ConVRT is a test-time optimization method featuring a neural video representation designed to enhance temporal consistency in restoration. A key innovation of ConVRT is the integration of a pretrained vision-language model (CLIP) for semantic-oriented supervision.
  arXiv  Detail & Related papers  (2023-12-07T20:19:48Z)
• PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance [65.5618804029422]
  Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. We propose PGDiff by introducing partial guidance, a fresh perspective that is more adaptable to real-world degradations. Our method not only outperforms existing diffusion-prior-based approaches but also competes favorably with task-specific models.
  arXiv  Detail & Related papers  (2023-09-19T17:51:33Z)
• Cross-Consistent Deep Unfolding Network for Adaptive All-In-One Video Restoration [78.14941737723501]
  We propose a Cross-consistent Deep Unfolding Network (CDUN) for All-In-One VR. By orchestrating two cascading procedures, CDUN achieves adaptive processing for diverse degradations. In addition, we introduce a window-based inter-frame fusion strategy to utilize information from more adjacent frames.
  arXiv  Detail & Related papers  (2023-09-04T14:18:00Z)
• An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement [132.60976158877608]
  We propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate-temporal-information across frames and reduces the need for high complexity networks.
  arXiv  Detail & Related papers  (2020-12-24T00:03:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.