Low Latency Video Denoising for Online Conferencing Using CNN Architectures

• URL: http://arxiv.org/abs/2302.08638v1
• Date: Fri, 17 Feb 2023 00:55:54 GMT
• Title: Low Latency Video Denoising for Online Conferencing Using CNN Architectures
• Authors: Altanai Bisht, Ana Carolina de Souza Mendes, Justin David Thoreson II, Shadrokh Samavi
• Abstract summary: We propose a pipeline for real-time video denoising with low runtime cost and high perceptual quality. A custom noise detector analyzer provides real-time feedback to adapt the weights and improve the models' output.
• Score: 4.7805617044617446
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we propose a pipeline for real-time video denoising with low runtime cost and high perceptual quality. The vast majority of denoising studies focus on image denoising. However, a minority of research works focusing on video denoising do so with higher performance costs to obtain higher quality while maintaining temporal coherence. The approach we introduce in this paper leverages the advantages of both image and video-denoising architectures. Our pipeline first denoises the keyframes or one-fifth of the frames using HI-GAN blind image denoising architecture. Then, the remaining four-fifths of the noisy frames and the denoised keyframe data are fed into the FastDVDnet video denoising model. The final output is rendered in the user's display in real-time. The combination of these low-latency neural network architectures produces real-time denoising with high perceptual quality with applications in video conferencing and other real-time media streaming systems. A custom noise detector analyzer provides real-time feedback to adapt the weights and improve the models' output.

Related papers

• Temporal As a Plugin: Unsupervised Video Denoising with Pre-Trained Image Denoisers [30.965705043127144]
  In this paper, we propose a novel unsupervised video denoising framework, named Temporal As aTAP' (TAP) By incorporating temporal modules, our method can harness temporal information across noisy frames, complementing its power of spatial denoising. Compared to other unsupervised video denoising methods, our framework demonstrates superior performance on both sRGB and raw video denoising datasets.
  arXiv  Detail & Related papers  (2024-09-17T15:05:33Z)
• Heterogeneous window transformer for image denoising [59.953076646860985]
  We propose a heterogeneous window transformer (HWformer) for image denoising. The proposed HWformer only takes 30% of popular Restormer in terms of denoising time.
  arXiv  Detail & Related papers  (2024-07-08T08:10:16Z)
• Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements [83.5820690348833]
  We present a framework for low-level vision tasks that does not require any external training data corpus. Our approach learns neural modules by optimizing over a corrupted sequence, leveraging the weights of the coherence-temporal test and statistics internal statistics.
  arXiv  Detail & Related papers  (2023-12-13T01:57:11Z)
• RViDeformer: Efficient Raw Video Denoising Transformer with a Larger Benchmark Dataset [16.131438855407175]
  There is no large dataset with realistic motions for supervised raw video denoising. We construct a video denoising dataset (named as ReCRVD) with 120 groups of noisy-clean videos. We propose an efficient raw video denoising transformer network (RViDeformer) that explores both short and long-distance correlations.
  arXiv  Detail & Related papers  (2023-05-01T11:06:58Z)
• Real-time Streaming Video Denoising with Bidirectional Buffers [48.57108807146537]
  Real-time denoising algorithms are typically adopted on the user device to remove the noise involved during the shooting and transmission of video streams. Recent multi-output inference works propagate the bidirectional temporal feature with a parallel or recurrent framework. We propose a Bidirectional Streaming Video Denoising framework, to achieve high-fidelity real-time denoising for streaming videos with both past and future temporal receptive fields.
  arXiv  Detail & Related papers  (2022-07-14T14:01:03Z)
• Efficient Burst Raw Denoising with Variance Stabilization and Multi-frequency Denoising Network [39.829676908118806]
  Smartphones have small apertures and small sensor cells, which lead to noisy images in low light environment. We propose an efficient burst denoising system based on noise prior integration, multi-frame alignment and multi-frame denoising. Our three-stage design is efficient and shows strong performance on burst denoising.
  arXiv  Detail & Related papers  (2022-05-10T07:46:45Z)
• Learning Spatial and Spatio-Temporal Pixel Aggregations for Image and Video Denoising [104.59305271099967]
  We present a pixel aggregation network and learn the pixel sampling and averaging strategies for image denoising. We develop a pixel aggregation network for video denoising to sample pixels across the spatial-temporal space. Our method is able to solve the misalignment issues caused by large motion in dynamic scenes.
  arXiv  Detail & Related papers  (2021-01-26T13:00:46Z)
• Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images [98.82804259905478]
  We present Neighbor2Neighbor to train an effective image denoising model with only noisy images. In detail, input and target used to train a network are images sub-sampled from the same noisy image. A denoising network is trained on sub-sampled training pairs generated in the first stage, with a proposed regularizer as additional loss for better performance.
  arXiv  Detail & Related papers  (2021-01-08T02:03:25Z)
• Learning Model-Blind Temporal Denoisers without Ground Truths [46.778450578529814]
  Denoisers trained with synthetic data often fail to cope with the diversity of unknown noises. Previous image-based method leads to noise overfitting if directly applied to video denoisers. We propose a general framework for video denoising networks that successfully addresses these challenges.
  arXiv  Detail & Related papers  (2020-07-07T07:19:48Z)
• First image then video: A two-stage network for spatiotemporal video denoising [19.842488445174524]
  Video denoising is to remove noise from noise-corrupted data, thus recovering true motion signals. Existing approaches for video denoising tend to suffer from blur artifacts, that is the boundary of a moving object tends to appear blurry. This paper introduces a first-image-then-video two-stage denoising neural network, consisting of an image denoising module and a regular intratemporal video denoising module. It yields state-of-the-art performances on the video denoising Vimeo90K dataset in terms of both denoising quality and computation.
  arXiv  Detail & Related papers  (2020-01-02T07:21:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.