Self-Supervised training for blind multi-frame video denoising
- URL: http://arxiv.org/abs/2004.06957v4
- Date: Tue, 20 Apr 2021 17:18:55 GMT
- Title: Self-Supervised training for blind multi-frame video denoising
- Authors: Val\'ery Dewil, J\'er\'emy Anger, Axel Davy, Thibaud Ehret, Pablo
Arias, Gabriele Facciolo
- Abstract summary: We propose a self-supervised approach for training multi-frame video denoising networks.
Our approach benefits from the video temporal consistency by penalizing a loss between the predicted frame t and a neighboring target frame.
- Score: 15.078027648304115
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a self-supervised approach for training multi-frame video
denoising networks. These networks predict frame t from a window of frames
around t. Our self-supervised approach benefits from the video temporal
consistency by penalizing a loss between the predicted frame t and a
neighboring target frame, which are aligned using an optical flow. We use the
proposed strategy for online internal learning, where a pre-trained network is
fine-tuned to denoise a new unknown noise type from a single video. After a few
frames, the proposed fine-tuning reaches and sometimes surpasses the
performance of a state-of-the-art network trained with supervision. In
addition, for a wide range of noise types, it can be applied blindly without
knowing the noise distribution. We demonstrate this by showing results on blind
denoising of different synthetic and realistic noises.
Related papers
- Temporal As a Plugin: Unsupervised Video Denoising with Pre-Trained Image Denoisers [30.965705043127144]
In this paper, we propose a novel unsupervised video denoising framework, named Temporal As aTAP' (TAP)
By incorporating temporal modules, our method can harness temporal information across noisy frames, complementing its power of spatial denoising.
Compared to other unsupervised video denoising methods, our framework demonstrates superior performance on both sRGB and raw video denoising datasets.
arXiv Detail & Related papers (2024-09-17T15:05:33Z) - TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models [94.24861019513462]
TRIP is a new recipe of image-to-video diffusion paradigm.
It pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning.
Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate TRIP's effectiveness.
arXiv Detail & Related papers (2024-03-25T17:59:40Z) - Video Dynamics Prior: An Internal Learning Approach for Robust Video
Enhancements [83.5820690348833]
We present a framework for low-level vision tasks that does not require any external training data corpus.
Our approach learns neural modules by optimizing over a corrupted sequence, leveraging the weights of the coherence-temporal test and statistics internal statistics.
arXiv Detail & Related papers (2023-12-13T01:57:11Z) - Real-time Streaming Video Denoising with Bidirectional Buffers [48.57108807146537]
Real-time denoising algorithms are typically adopted on the user device to remove the noise involved during the shooting and transmission of video streams.
Recent multi-output inference works propagate the bidirectional temporal feature with a parallel or recurrent framework.
We propose a Bidirectional Streaming Video Denoising framework, to achieve high-fidelity real-time denoising for streaming videos with both past and future temporal receptive fields.
arXiv Detail & Related papers (2022-07-14T14:01:03Z) - Dynamic Slimmable Denoising Network [64.77565006158895]
Dynamic slimmable denoising network (DDSNet) is a general method to achieve good denoising quality with less computational complexity.
OurNet is empowered with the ability of dynamic inference by a dynamic gate.
Our experiments demonstrate our-Net consistently outperforms the state-of-the-art individually trained static denoising networks.
arXiv Detail & Related papers (2021-10-17T22:45:33Z) - Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask [14.265454188161819]
We propose a learning-based approach for denoising raw videos captured under low lighting conditions.
We first explicitly align the neighboring frames to the current frame using a convolutional neural network (CNN)
We then fuse the registered frames using another CNN to obtain the final denoised frame.
arXiv Detail & Related papers (2021-03-04T06:57:48Z) - Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images [98.82804259905478]
We present Neighbor2Neighbor to train an effective image denoising model with only noisy images.
In detail, input and target used to train a network are images sub-sampled from the same noisy image.
A denoising network is trained on sub-sampled training pairs generated in the first stage, with a proposed regularizer as additional loss for better performance.
arXiv Detail & Related papers (2021-01-08T02:03:25Z) - Learning Model-Blind Temporal Denoisers without Ground Truths [46.778450578529814]
Denoisers trained with synthetic data often fail to cope with the diversity of unknown noises.
Previous image-based method leads to noise overfitting if directly applied to video denoisers.
We propose a general framework for video denoising networks that successfully addresses these challenges.
arXiv Detail & Related papers (2020-07-07T07:19:48Z) - Restore from Restored: Video Restoration with Pseudo Clean Video [28.057705167363327]
We propose a self-supervised video denoising method called "restore-from-restored"
This method fine-tunes a pre-trained network by using a pseudo clean video during the test phase.
We analyze the restoration performance of the fine-tuned video denoising networks with the proposed self-supervision-based learning algorithm.
arXiv Detail & Related papers (2020-03-09T17:37:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.