TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation

• URL: http://arxiv.org/abs/2408.14227v1
• Date: Mon, 26 Aug 2024 12:43:48 GMT
• Title: TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation
• Authors: Anh-Dzung Doan, Vu Minh Hieu Phan, Surabhi Gupta, Markus Wagner, Tat-Jun Chin, Ian Reid,
• Abstract summary: This paper proposes a novel diffusion method, dubbed Temporally Consistent Patch Diffusion Models (TC-DPM) Our method faithfully preserves the semantic structure of generated visible images. Experiment shows that TC-PDM outperforms state-of-the-art methods by 35.3% in FVD for infrared-to-visible video translation and by 6.1% in AP50 for day-to-night object detection.
• Score: 25.542902579879367
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Infrared imaging offers resilience against changing lighting conditions by capturing object temperatures. Yet, in few scenarios, its lack of visual details compared to daytime visible images, poses a significant challenge for human and machine interpretation. This paper proposes a novel diffusion method, dubbed Temporally Consistent Patch Diffusion Models (TC-DPM), for infrared-to-visible video translation. Our method, extending the Patch Diffusion Model, consists of two key components. Firstly, we propose a semantic-guided denoising, leveraging the strong representations of foundational models. As such, our method faithfully preserves the semantic structure of generated visible images. Secondly, we propose a novel temporal blending module to guide the denoising trajectory, ensuring the temporal consistency between consecutive frames. Experiment shows that TC-PDM outperforms state-of-the-art methods by 35.3% in FVD for infrared-to-visible video translation and by 6.1% in AP50 for day-to-night object detection. Our code is publicly available at https://github.com/dzungdoan6/tc-pdm

Related papers

• Let Video Teaches You More: Video-to-Image Knowledge Distillation using DEtection TRansformer for Medical Video Lesion Detection [91.97935118185]
  We propose Video-to-Image knowledge distillation for the task of medical video lesion detection. By distilling multi-frame contexts into a single frame, the proposed V2I-DETR combines the advantages of utilizing temporal contexts from video-based models and the inference speed of image-based models. V2I-DETR outperforms previous state-of-the-art methods by a large margin while achieving the real-time inference speed (30 FPS) as the image-based model.
  arXiv  Detail & Related papers  (2024-08-26T07:17:05Z)
• PID: Physics-Informed Diffusion Model for Infrared Image Generation [11.416759828137701]
  Infrared imaging technology has gained significant attention for its reliable sensing ability in low visibility conditions. Most existing image translation methods treat infrared images as a stylistic variation, neglecting the underlying physical laws. We propose a Physics-Informed Diffusion (PID) model for translating RGB images to infrared images that adhere to physical laws.
  arXiv  Detail & Related papers  (2024-07-12T14:32:30Z)
• Blind Image Restoration via Fast Diffusion Inversion [17.139433082780037]
  Blind Image Restoration via fast Diffusion (BIRD) is a blind IR method that jointly optimize for the degradation model parameters and the restored image. A key idea in our method is not to modify the reverse sampling, i.e., not to alter all the intermediate latents, once an initial noise is sampled. We experimentally validate BIRD on several image restoration tasks and show that it achieves state of the art performance on all of them.
  arXiv  Detail & Related papers  (2024-05-29T23:38:12Z)
• One-Step Image Translation with Text-to-Image Models [35.0987002313882]
  We introduce a general method for adapting a single-step diffusion model to new tasks and domains through adversarial learning objectives. We consolidate various modules of the vanilla latent diffusion model into a single end-to-end generator network with small trainable weights. Our model CycleGAN-Turbo outperforms existing GAN-based and diffusion-based methods for various scene translation tasks.
  arXiv  Detail & Related papers  (2024-03-18T17:59:40Z)
• Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing [58.48890547818074]
  We present a powerful modification of Contrastive Denoising Score (CUT) for latent diffusion models (LDM) Our approach enables zero-shot imageto-image translation and neural field (NeRF) editing, achieving structural correspondence between the input and output.
  arXiv  Detail & Related papers  (2023-11-30T15:06:10Z)
• Prompt-tuning latent diffusion models for inverse problems [72.13952857287794]
  We propose a new method for solving imaging inverse problems using text-to-image latent diffusion models as general priors. Our method, called P2L, outperforms both image- and latent-diffusion model-based inverse problem solvers on a variety of tasks, such as super-resolution, deblurring, and inpainting.
  arXiv  Detail & Related papers  (2023-10-02T11:31:48Z)
• Denoising Diffusion Models for Plug-and-Play Image Restoration [135.6359475784627]
  This paper proposes DiffPIR, which integrates the traditional plug-and-play method into the diffusion sampling framework. Compared to plug-and-play IR methods that rely on discriminative Gaussian denoisers, DiffPIR is expected to inherit the generative ability of diffusion models.
  arXiv  Detail & Related papers  (2023-05-15T20:24:38Z)
• ADIR: Adaptive Diffusion for Image Reconstruction [46.838084286784195]
  We propose a conditional sampling scheme that exploits the prior learned by diffusion models. We then combine it with a novel approach for adapting pretrained diffusion denoising networks to their input. We show that our proposed adaptive diffusion for image reconstruction' approach achieves a significant improvement in the super-resolution, deblurring, and text-based editing tasks.
  arXiv  Detail & Related papers  (2022-12-06T18:39:58Z)
• T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models [71.94264837503135]
  We propose a Denoising Diffusion Probabilistic Model (DDPM) based solution for Thermal-to-Visible (T2V) image translation. During training, the model learns the conditional distribution of visible facial images given their corresponding thermal image. We achieve the state-of-the-art results on multiple datasets.
  arXiv  Detail & Related papers  (2022-09-19T07:59:32Z)
• Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario [87.72258480670627]
  Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images. This paper proposes a Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) to achieve a more comprehensive spatial-temporal feature representation.
  arXiv  Detail & Related papers  (2022-07-05T09:27:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.