Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence
- URL: http://arxiv.org/abs/2404.13605v1
- Date: Sun, 21 Apr 2024 10:28:34 GMT
- Title: Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence
- Authors: Ripon Kumar Saha, Dehao Qin, Nianyi Li, Jinwei Ye, Suren Jayasuriya,
- Abstract summary: This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment.
We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration.
Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos.
- Score: 10.8380383565446
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Tackling image degradation due to atmospheric turbulence, particularly in dynamic environment, remains a challenge for long-range imaging systems. Existing techniques have been primarily designed for static scenes or scenes with small motion. This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment. We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration. After camera shake compensation and segmentation, we introduce foreground/background enhancement leveraging the statistics of turbulence strength and a transformer model trained on a novel noise-based procedural turbulence generator for fast dataset augmentation. Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos. We make our code, simulator, and data publicly available to advance the field of video restoration from turbulence: riponcs.github.io/TurbSegRes
Related papers
- Unsupervised Motion Segmentation for Neuromorphic Aerial Surveillance [42.04157319642197]
Event cameras have exceptional temporal resolution, superior dynamic range, and minimal power requirements.
Previous methods for event-based motion segmentation encountered limitations.
Our proposed method leverages features from self-supervised transformers on both event data and optical flow information.
arXiv Detail & Related papers (2024-05-24T04:36:13Z) - ConVRT: Consistent Video Restoration Through Turbulence with Test-time
Optimization of Neural Video Representations [13.38405890753946]
We introduce a self-supervised method, Consistent Video Restoration through Turbulence (ConVRT)
ConVRT is a test-time optimization method featuring a neural video representation designed to enhance temporal consistency in restoration.
A key innovation of ConVRT is the integration of a pretrained vision-language model (CLIP) for semantic-oriented supervision.
arXiv Detail & Related papers (2023-12-07T20:19:48Z) - GenDeF: Learning Generative Deformation Field for Video Generation [89.49567113452396]
We propose to render a video by warping one static image with a generative deformation field (GenDeF)
Such a pipeline enjoys three appealing advantages.
arXiv Detail & Related papers (2023-12-07T18:59:41Z) - Event-based Continuous Color Video Decompression from Single Frames [38.59798259847563]
We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera.
Our approach combines continuous long-range motion modeling with a feature-plane-based neural integration model, enabling frame prediction at arbitrary times within the events.
arXiv Detail & Related papers (2023-11-30T18:59:23Z) - Unsupervised Region-Growing Network for Object Segmentation in
Atmospheric Turbulence [11.62754560134596]
We present a two-stage unsupervised object segmentation network tailored for dynamic scenes affected by atmospheric turbulence.
In the first stage, we utilize averaged optical flow from turbulence-distorted image sequences to craft preliminary masks for each moving object.
We release the first moving object segmentation dataset of turbulence-affected videos, complete with manually annotated ground truth masks.
arXiv Detail & Related papers (2023-11-06T22:17:18Z) - DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene.
We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views.
We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
arXiv Detail & Related papers (2022-11-20T20:57:02Z) - Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A
New Physics-Inspired Transformer Model [82.23276183684001]
We propose a physics-inspired transformer model for imaging through atmospheric turbulence.
The proposed network utilizes the power of transformer blocks to jointly extract a dynamical turbulence distortion map.
We present two new real-world turbulence datasets that allow for evaluation with both classical objective metrics and a new task-driven metric using text recognition accuracy.
arXiv Detail & Related papers (2022-07-20T17:09:16Z) - NeuralDiff: Segmenting 3D objects that move in egocentric videos [92.95176458079047]
We study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground.
This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion.
In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them.
arXiv Detail & Related papers (2021-10-19T12:51:35Z) - Restoration of Video Frames from a Single Blurred Image with Motion
Understanding [69.90724075337194]
We propose a novel framework to generate clean video frames from a single motion-red image.
We formulate video restoration from a single blurred image as an inverse problem by setting clean image sequence and their respective motion as latent factors.
Our framework is based on anblur-decoder structure with spatial transformer network modules.
arXiv Detail & Related papers (2021-04-19T08:32:57Z) - Non-Rigid Neural Radiance Fields: Reconstruction and Novel View
Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes.
We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
arXiv Detail & Related papers (2020-12-22T18:46:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.