LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network

• URL: http://arxiv.org/abs/2307.09815v3
• Date: Tue, 21 Nov 2023 07:19:03 GMT
• Title: LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network
• Authors: Hao Yang, Liyuan Pan, Yan Yang, Richard Hartley, Miaomiao Liu
• Abstract summary: We introduce the contrastive language-image pre-training framework (CLIP) to accurately estimate the blur map from a DP pair unsupervisedly. Our method achieves state-of-the-art performance in extensive experiments.
• Score: 20.42001628066274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering sharp images from dual-pixel (DP) pairs with disparity-dependent blur is a challenging task.~Existing blur map-based deblurring methods have demonstrated promising results. In this paper, we propose, to the best of our knowledge, the first framework that introduces the contrastive language-image pre-training framework (CLIP) to accurately estimate the blur map from a DP pair unsupervisedly. To achieve this, we first carefully design text prompts to enable CLIP to understand blur-related geometric prior knowledge from the DP pair. Then, we propose a format to input a stereo DP pair to CLIP without any fine-tuning, despite the fact that CLIP is pre-trained on monocular images. Given the estimated blur map, we introduce a blur-prior attention block, a blur-weighting loss, and a blur-aware loss to recover the all-in-focus image. Our method achieves state-of-the-art performance in extensive experiments (see Fig.~\ref{fig:teaser}).

Related papers

• Iterative Prompt Learning for Unsupervised Backlit Image Enhancement [86.90993077000789]
  We propose a novel unsupervised backlit image enhancement method, abbreviated as CLIP-LIT. We show that the open-world CLIP prior aids in distinguishing between backlit and well-lit images. Our method alternates between updating the prompt learning framework and enhancement network until visually pleasing results are achieved.
  arXiv  Detail & Related papers  (2023-03-30T17:37:14Z)
• Unsupervised Light Field Depth Estimation via Multi-view Feature Matching with Occlusion Prediction [15.421219881815956]
  It is costly to obtain sufficient depth labels for supervised training. In this paper, we propose an unsupervised framework to estimate depth from LF images.
  arXiv  Detail & Related papers  (2023-01-20T06:11:17Z)
• Learning Single Image Defocus Deblurring with Misaligned Training Pairs [80.13320797431487]
  We propose a joint deblurring and reblurring learning framework for single image defocus deblurring. Our framework can be applied to boost defocus deblurring networks in terms of both quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2022-11-26T07:36:33Z)
• SVBR-NET: A Non-Blind Spatially Varying Defocus Blur Removal Network [2.4975981795360847]
  We propose a non-blind approach for image deblurring that can deal with spatially-varying kernels. We introduce two encoder-decoder sub-networks that are fed with the blurry image and the estimated blur map. The network is trained with synthetically blur kernels that are augmented to emulate blur maps produced by existing blur estimation methods.
  arXiv  Detail & Related papers  (2022-06-26T17:21:12Z)
• CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation [16.082155440640964]
  We propose a pixel-wise contrastive learning method called CP2 (Copy-Paste Contrastive Pretraining) In detail, we copy-paste a random crop from an image (the foreground) onto different background images and pretrain a semantic segmentation model. Experiments show the strong performance of CP2 in downstream semantic segmentation.
  arXiv  Detail & Related papers  (2022-03-22T13:21:49Z)
• Image Inpainting with Edge-guided Learnable Bidirectional Attention Maps [85.67745220834718]
  We present an edge-guided learnable bidirectional attention map (Edge-LBAM) for improving image inpainting of irregular holes. Our Edge-LBAM method contains dual procedures,including structure-aware mask-updating guided by predict edges. Extensive experiments show that our Edge-LBAM is effective in generating coherent image structures and preventing color discrepancy and blurriness.
  arXiv  Detail & Related papers  (2021-04-25T07:25:16Z)
• Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration [77.1056200937214]
  We study the formation of the DP pair which links the blur and the depth information. We propose an end-to-end DDDNet (DP-based Depth and De Network) to jointly estimate the depth and restore the image.
  arXiv  Detail & Related papers  (2020-12-01T06:53:57Z)
• Deblurring by Realistic Blurring [110.54173799114785]
  We propose a new method which combines two GAN models, i.e., a learning-to-blurr GAN (BGAN) and learning-to-DeBlur GAN (DBGAN) The first model, BGAN, learns how to blur sharp images with unpaired sharp and blurry image sets, and then guides the second model, DBGAN, to learn how to correctly deblur such images. As an additional contribution, this paper also introduces a Real-World Blurred Image (RWBI) dataset including diverse blurry images.
  arXiv  Detail & Related papers  (2020-04-04T05:25:15Z)
• Self-Supervised Linear Motion Deblurring [112.75317069916579]
  Deep convolutional neural networks are state-of-the-art for image deblurring. We present a differentiable reblur model for self-supervised motion deblurring. Our experiments demonstrate that self-supervised single image deblurring is really feasible.
  arXiv  Detail & Related papers  (2020-02-10T20:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.