Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera

• URL: http://arxiv.org/abs/2304.06019v1
• Date: Wed, 12 Apr 2023 17:56:42 GMT
• Title: Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera
• Authors: Ruicheng Feng, Chongyi Li, Huaijin Chen, Shuai Li, Jinwei Gu, Chen Change Loy
• Abstract summary: We revisit the classic stereo setup for training data collection -- capturing two images of the same scene with one UDC and one standard camera. The key idea is to "copy" details from a high-quality reference image and "paste" them on the UDC image. A novel Transformer-based framework generates well-aligned yet high-quality target data for the corresponding UDC input.
• Score: 84.41316720913785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to the difficulty in collecting large-scale and perfectly aligned paired training data for Under-Display Camera (UDC) image restoration, previous methods resort to monitor-based image systems or simulation-based methods, sacrificing the realness of the data and introducing domain gaps. In this work, we revisit the classic stereo setup for training data collection -- capturing two images of the same scene with one UDC and one standard camera. The key idea is to "copy" details from a high-quality reference image and "paste" them on the UDC image. While being able to generate real training pairs, this setting is susceptible to spatial misalignment due to perspective and depth of field changes. The problem is further compounded by the large domain discrepancy between the UDC and normal images, which is unique to UDC restoration. In this paper, we mitigate the non-trivial domain discrepancy and spatial misalignment through a novel Transformer-based framework that generates well-aligned yet high-quality target data for the corresponding UDC input. This is made possible through two carefully designed components, namely, the Domain Alignment Module (DAM) and Geometric Alignment Module (GAM), which encourage robust and accurate discovery of correspondence between the UDC and normal views. Extensive experiments show that high-quality and well-aligned pseudo UDC training pairs are beneficial for training a robust restoration network. Code and the dataset are available at https://github.com/jnjaby/AlignFormer.

Related papers

• Deep Video Restoration for Under-Display Camera [98.17505013737446]
  We propose a GAN-based generation pipeline to simulate the realistic UDC degradation process. We build the first large-scale UDC video restoration dataset called PexelsUDC. We propose a novel transformer-based baseline method that adaptively enhances degraded videos.
  arXiv  Detail & Related papers  (2023-09-09T10:48:06Z)
• Blind Face Restoration for Under-Display Camera via Dictionary Guided Transformer [32.06570655576273]
  Under-Display Camera (UDC) provides users with a full-screen experience by hiding the front-facing camera below the display panel. UDC images suffer from significant quality degradation due to the characteristics of the display. We propose a two-stage network UDC Degradation Model Network named UDC-DMNet to synthesize UDC images by modeling the processes of UDC imaging.
  arXiv  Detail & Related papers  (2023-08-20T08:02:23Z)
• Learning Single Image Defocus Deblurring with Misaligned Training Pairs [80.13320797431487]
  We propose a joint deblurring and reblurring learning framework for single image defocus deblurring. Our framework can be applied to boost defocus deblurring networks in terms of both quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2022-11-26T07:36:33Z)
• Modular Degradation Simulation and Restoration for Under-Display Camera [21.048590332029995]
  Under-display camera (UDC) provides an elegant solution for full-screen smartphones. UDC captured images suffer from severe degradation since sensors lie under the display. We propose a modular network dubbed MPGNet trained using the generative adversarial network (GAN) framework for simulating UDC imaging.
  arXiv  Detail & Related papers  (2022-09-23T07:36:07Z)
• UDC-UNet: Under-Display Camera Image Restoration via U-Shape Dynamic Network [13.406025621307132]
  Under-Display Camera (UDC) has been widely exploited to help smartphones realize full screen display. As the screen could inevitably affect the light propagation process, the images captured by the UDC system usually contain flare, haze, blur, and noise. In this paper, we propose a new deep model, namely UDC-UNet, to address the UDC image restoration problem with the known Point Spread Function (PSF) in HDR scenes.
  arXiv  Detail & Related papers  (2022-09-05T07:41:44Z)
• Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World [83.36195426897768]
  We propose VRVO, a novel framework for retrieving the absolute scale from virtual data. We first train a scale-aware disparity network using both monocular real images and stereo virtual data. The resulting scale-consistent disparities are then integrated with a direct VO system.
  arXiv  Detail & Related papers  (2022-03-11T01:51:54Z)
• Self-supervised Correlation Mining Network for Person Image Generation [9.505343361614928]
  Person image generation aims to perform non-rigid deformation on source images. We propose a Self-supervised Correlation Mining Network (SCM-Net) to rearrange the source images in the feature space. For improving the fidelity of cross-scale pose transformation, we propose a graph based Body Structure Retaining Loss.
  arXiv  Detail & Related papers  (2021-11-26T03:57:46Z)
• DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation [97.74059510314554]
  Unsupervised domain adaptation (UDA) for semantic segmentation aims to adapt a segmentation model trained on the labeled source domain to the unlabeled target domain. Existing methods try to learn domain invariant features while suffering from large domain gaps. We propose a novel Dual Soft-Paste (DSP) method in this paper.
  arXiv  Detail & Related papers  (2021-07-20T16:22:40Z)
• iFAN: Image-Instance Full Alignment Networks for Adaptive Object Detection [48.83883375118966]
  iFAN aims to precisely align feature distributions on both image and instance levels. It outperforms state-of-the-art methods with a boost of 10%+ AP over the source-only baseline.
  arXiv  Detail & Related papers  (2020-03-09T13:27:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.