UDC-VIT: A Real-World Video Dataset for Under-Display Cameras

• URL: http://arxiv.org/abs/2501.18545v1
• Date: Thu, 30 Jan 2025 18:13:29 GMT
• Title: UDC-VIT: A Real-World Video Dataset for Under-Display Cameras
• Authors: Kyusu Ahn, JiSoo Kim, Sangik Lee, HyunGyu Lee, Byeonghyun Ko, Chanwoo Park, Jaejin Lee,
• Abstract summary: Under Display Camera (UDC) is an advanced imaging system that places a digital camera lens underneath a display panel, effectively concealing the camera. UDCs significantly degrade captured images or videos, introducing low transmittance, blur, noise, and flare issues. Despite extensive research on UDC images and their restoration models, studies on videos have yet to be significantly explored.
• Score: 6.669223432319789
• License:
• Abstract: Under Display Camera (UDC) is an advanced imaging system that places a digital camera lens underneath a display panel, effectively concealing the camera. However, the display panel significantly degrades captured images or videos, introducing low transmittance, blur, noise, and flare issues. Tackling such issues is challenging because of the complex degradation of UDCs, including diverse flare patterns. Despite extensive research on UDC images and their restoration models, studies on videos have yet to be significantly explored. While two UDC video datasets exist, they primarily focus on unrealistic or synthetic UDC degradation rather than real-world UDC degradation. In this paper, we propose a real-world UDC video dataset called UDC-VIT. Unlike existing datasets, only UDC-VIT exclusively includes human motions that target facial recognition. We propose a video-capturing system to simultaneously acquire non-degraded and UDC-degraded videos of the same scene. Then, we align a pair of captured videos frame by frame, using discrete Fourier transform (DFT). We compare UDC-VIT with six representative UDC still image datasets and two existing UDC video datasets. Using six deep-learning models, we compare UDC-VIT and an existing synthetic UDC video dataset. The results indicate the ineffectiveness of models trained on earlier synthetic UDC video datasets, as they do not reflect the actual characteristics of UDC-degraded videos. We also demonstrate the importance of effective UDC restoration by evaluating face recognition accuracy concerning PSNR, SSIM, and LPIPS scores. UDC-VIT enables further exploration in the UDC video restoration and offers better insights into the challenge. UDC-VIT is available at our project site.

Related papers

• AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction [88.70116693750452]
  Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. We introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions.
  arXiv  Detail & Related papers  (2024-06-10T17:02:08Z)
• DVANet: Disentangling View and Action Features for Multi-View Action Recognition [56.283944756315066]
  We present a novel approach to multi-view action recognition where we guide learned action representations to be separated from view-relevant information in a video. Our model and method of training significantly outperforms all other uni-modal models on four multi-view action recognition datasets.
  arXiv  Detail & Related papers  (2023-12-10T01:19:48Z)
• Deep Video Restoration for Under-Display Camera [98.17505013737446]
  We propose a GAN-based generation pipeline to simulate the realistic UDC degradation process. We build the first large-scale UDC video restoration dataset called PexelsUDC. We propose a novel transformer-based baseline method that adaptively enhances degraded videos.
  arXiv  Detail & Related papers  (2023-09-09T10:48:06Z)
• Blind Face Restoration for Under-Display Camera via Dictionary Guided Transformer [32.06570655576273]
  Under-Display Camera (UDC) provides users with a full-screen experience by hiding the front-facing camera below the display panel. UDC images suffer from significant quality degradation due to the characteristics of the display. We propose a two-stage network UDC Degradation Model Network named UDC-DMNet to synthesize UDC images by modeling the processes of UDC imaging.
  arXiv  Detail & Related papers  (2023-08-20T08:02:23Z)
• Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera [84.41316720913785]
  We revisit the classic stereo setup for training data collection -- capturing two images of the same scene with one UDC and one standard camera. The key idea is to "copy" details from a high-quality reference image and "paste" them on the UDC image. A novel Transformer-based framework generates well-aligned yet high-quality target data for the corresponding UDC input.
  arXiv  Detail & Related papers  (2023-04-12T17:56:42Z)
• Video Instance Shadow Detection Under the Sun and Sky [81.95848151121739]
  ViShadow is a semi-supervised video instance shadow detection framework. It identifies shadow and object instances through contrastive learning for cross-frame pairing. A retrieval mechanism is introduced to manage temporary disappearances.
  arXiv  Detail & Related papers  (2022-11-23T10:20:19Z)
• UDC-UNet: Under-Display Camera Image Restoration via U-Shape Dynamic Network [13.406025621307132]
  Under-Display Camera (UDC) has been widely exploited to help smartphones realize full screen display. As the screen could inevitably affect the light propagation process, the images captured by the UDC system usually contain flare, haze, blur, and noise. In this paper, we propose a new deep model, namely UDC-UNet, to address the UDC image restoration problem with the known Point Spread Function (PSF) in HDR scenes.
  arXiv  Detail & Related papers  (2022-09-05T07:41:44Z)
• Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network [80.67717076541956]
  Under-Display Camera (UDC) systems provide a true bezel-less and notch-free viewing experience on smartphones. In a typical UDC system, the pixel array attenuates and diffracts the incident light on the camera, resulting in significant image quality degradation. In this work, we aim to analyze and tackle the aforementioned degradation problems.
  arXiv  Detail & Related papers  (2021-04-19T18:41:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.