EGVD: Event-Guided Video Deraining
- URL: http://arxiv.org/abs/2309.17239v1
- Date: Fri, 29 Sep 2023 13:47:53 GMT
- Title: EGVD: Event-Guided Video Deraining
- Authors: Yueyi Zhang, Jin Wang, Wenming Weng, Xiaoyan Sun, Zhiwei Xiong
- Abstract summary: We propose an end-to-end learning-based network to unlock the potential of the event camera for video deraining.
We build a real-world dataset consisting of rainy videos and temporally synchronized event streams.
- Score: 57.59935209162314
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: With the rapid development of deep learning, video deraining has experienced
significant progress. However, existing video deraining pipelines cannot
achieve satisfying performance for scenes with rain layers of complex
spatio-temporal distribution. In this paper, we approach video deraining by
employing an event camera. As a neuromorphic sensor, the event camera suits
scenes of non-uniform motion and dynamic light conditions. We propose an
end-to-end learning-based network to unlock the potential of the event camera
for video deraining. First, we devise an event-aware motion detection module to
adaptively aggregate multi-frame motion contexts using event-aware masks.
Second, we design a pyramidal adaptive selection module for reliably separating
the background and rain layers by incorporating multi-modal contextualized
priors. In addition, we build a real-world dataset consisting of rainy videos
and temporally synchronized event streams. We compare our method with extensive
state-of-the-art methods on synthetic and self-collected real-world datasets,
demonstrating the clear superiority of our method. The code and dataset are
available at \url{https://github.com/booker-max/EGVD}.
Related papers
- Temporal Event Stereo via Joint Learning with Stereoscopic Flow [44.479946706395694]
Event cameras are dynamic vision sensors inspired by the biological retina.
We propose a novel temporal event stereo framework.
We have achieved state-of-the-art performance on the MVSEC and the DSEC datasets.
arXiv Detail & Related papers (2024-07-15T15:43:08Z) - Temporal-Mapping Photography for Event Cameras [5.838762448259289]
Event cameras capture brightness changes as a continuous stream of events'' rather than traditional intensity frames.
We realize events to dense intensity image conversion using a stationary event camera in static scenes.
arXiv Detail & Related papers (2024-03-11T05:29:46Z) - Event-aware Video Corpus Moment Retrieval [79.48249428428802]
Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos.
Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos.
We propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval.
arXiv Detail & Related papers (2024-02-21T06:55:20Z) - An Event-Oriented Diffusion-Refinement Method for Sparse Events
Completion [36.64856578682197]
Event cameras or dynamic vision sensors (DVS) record asynchronous response to brightness changes instead of conventional intensity frames.
We propose an inventive event completion sequence approach conforming to unique characteristics of event data in both the processing stage and the output form.
Specifically, we treat event streams as 3D event clouds in thetemporal domain, develop a diffusion-based generative model to generate dense clouds in a coarse-to-fine manner, and recover exact timestamps to maintain the temporal resolution of raw data successfully.
arXiv Detail & Related papers (2024-01-06T08:09:54Z) - Event-based Continuous Color Video Decompression from Single Frames [38.59798259847563]
We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera.
Our approach combines continuous long-range motion modeling with a feature-plane-based neural integration model, enabling frame prediction at arbitrary times within the events.
arXiv Detail & Related papers (2023-11-30T18:59:23Z) - DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking
Tasks [76.24996889649744]
Masked autoencoder (MAE) pretraining on videos for matching-based downstream tasks, including visual object tracking (VOT) and video object segmentation (VOS)
We propose DropMAE, which adaptively performs spatial-attention dropout in the frame reconstruction to facilitate temporal correspondence learning in videos.
Our model sets new state-of-the-art performance on 8 out of 9 highly competitive video tracking and segmentation datasets.
arXiv Detail & Related papers (2023-04-02T16:40:42Z) - Feature-Aligned Video Raindrop Removal with Temporal Constraints [68.49161092870224]
Raindrop removal is challenging for both single image and video.
Unlike rain streaks, adherent raindrops tend to cover the same area in several frames.
Our method employs a two-stage video-based raindrop removal method.
arXiv Detail & Related papers (2022-05-29T05:42:14Z) - Event-guided Deblurring of Unknown Exposure Time Videos [31.992673443516235]
Event cameras can capture apparent motion with a high temporal resolution.
We propose a novel Exposure Time-based Event Selection module to selectively use event features.
Our method achieves state-of-the-art performance.
arXiv Detail & Related papers (2021-12-13T19:46:17Z) - VisEvent: Reliable Object Tracking via Collaboration of Frame and Event
Flows [93.54888104118822]
We propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task.
Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios.
Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods.
arXiv Detail & Related papers (2021-08-11T03:55:12Z) - Learning Monocular Dense Depth from Events [53.078665310545745]
Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames.
Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction.
We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
arXiv Detail & Related papers (2020-10-16T12:36:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.