PromptRR: Diffusion Models as Prompt Generators for Single Image Reflection Removal

• URL: http://arxiv.org/abs/2402.02374v1
• Date: Sun, 4 Feb 2024 07:11:10 GMT
• Title: PromptRR: Diffusion Models as Prompt Generators for Single Image Reflection Removal
• Authors: Tao Wang, Wanglong Lu, Kaihao Zhang, Wenhan Luo, Tae-Kyun Kim, Tong Lu, Hongdong Li, Ming-Hsuan Yang
• Abstract summary: Existing single image reflection removal (SIRR) methods tend to miss key low-frequency (LF) and high-frequency (HF) differences in images. This paper proposes a novel prompt-guided reflection removal framework that uses frequency information as new visual prompts for better reflection performance.
• Score: 138.38229287266915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing single image reflection removal (SIRR) methods using deep learning tend to miss key low-frequency (LF) and high-frequency (HF) differences in images, affecting their effectiveness in removing reflections. To address this problem, this paper proposes a novel prompt-guided reflection removal (PromptRR) framework that uses frequency information as new visual prompts for better reflection performance. Specifically, the proposed framework decouples the reflection removal process into the prompt generation and subsequent prompt-guided restoration. For the prompt generation, we first propose a prompt pre-training strategy to train a frequency prompt encoder that encodes the ground-truth image into LF and HF prompts. Then, we adopt diffusion models (DMs) as prompt generators to generate the LF and HF prompts estimated by the pre-trained frequency prompt encoder. For the prompt-guided restoration, we integrate specially generated prompts into the PromptFormer network, employing a novel Transformer-based prompt block to effectively steer the model toward enhanced reflection removal. The results on commonly used benchmarks show that our method outperforms state-of-the-art approaches. The codes and models are available at https://github.com/TaoWangzj/PromptRR.

Related papers

• InstantIR: Blind Image Restoration with Instant Generative Reference [10.703499573064537]
  We introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method. We first extract a compact representation of the input via a pre-trained vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate it in the generative prior. The degraded image is then encoded with this reference, providing robust generation condition.
  arXiv  Detail & Related papers  (2024-10-09T05:15:29Z)
• Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration [44.35766203309201]
  We develop a Frequency Prompting image restoration method, dubbed FPro, which can effectively provide prompt components from a frequency perspective to guild the restoration model. Experimental results on commonly used benchmarks have demonstrated the favorable performance of our pipeline against SOTA methods on 5 image restoration tasks.
  arXiv  Detail & Related papers  (2024-03-30T08:42:34Z)
• Single Image Reflection Removal with Reflection Intensity Prior Knowledge [14.335849624907611]
  We propose a general reflection intensity prior that captures the intensity of the reflection phenomenon. By segmenting images into regional patches, RPEN learns non-uniform reflection prior in an image. We propose Prior-based Reflection Removal Network (PRRN) using a simple transformer U-Net architecture.
  arXiv  Detail & Related papers  (2023-12-06T14:52:11Z)
• Reverse Stable Diffusion: What prompt was used to generate this image? [73.10116197883303]
  We study the task of predicting the prompt embedding given an image generated by a generative diffusion model. We propose a novel learning framework comprising a joint prompt regression and multi-label vocabulary classification objective. We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion.
  arXiv  Detail & Related papers  (2023-08-02T23:39:29Z)
• Robust Real-World Image Super-Resolution against Adversarial Attacks [115.04009271192211]
  adversarial image samples with quasi-imperceptible noises could threaten deep learning SR models. We propose a robust deep learning framework for real-world SR that randomly erases potential adversarial noises. Our proposed method is more insensitive to adversarial attacks and presents more stable SR results than existing models and defenses.
  arXiv  Detail & Related papers  (2022-07-31T13:26:33Z)
• ReflectNet -- A Generative Adversarial Method for Single Image Reflection Suppression [0.6980076213134382]
  We propose a single image reflection removal method based on context understanding modules and adversarial training. Our proposed reflection removal method outperforms state-of-the-art methods in terms of PSNR and SSIM on the SIR benchmark dataset.
  arXiv  Detail & Related papers  (2021-05-11T17:33:40Z)
• Location-aware Single Image Reflection Removal [54.93808224890273]
  This paper proposes a novel location-aware deep learning-based single image reflection removal method. We use a reflection confidence map as the cues for the network to learn how to encode the reflection information adaptively. The integration of location information into the network significantly improves the quality of reflection removal results.
  arXiv  Detail & Related papers  (2020-12-13T19:34:35Z)
• Two-Stage Single Image Reflection Removal with Reflection-Aware Guidance [78.34235841168031]
  We present a novel two-stage network with reflection-aware guidance (RAGNet) for single image reflection removal (SIRR) RAG can be used (i) to mitigate the effect of reflection from the observation, and (ii) to generate mask in partial convolution for mitigating the effect of deviating from linear combination hypothesis. Experiments on five commonly used datasets demonstrate the quantitative and qualitative superiority of our RAGNet in comparison to the state-of-the-art SIRR methods.
  arXiv  Detail & Related papers  (2020-12-02T03:14:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.