DeTurb: Atmospheric Turbulence Mitigation with Deformable 3D Convolutions and 3D Swin Transformers
- URL: http://arxiv.org/abs/2407.20855v2
- Date: Mon, 30 Sep 2024 23:01:51 GMT
- Title: DeTurb: Atmospheric Turbulence Mitigation with Deformable 3D Convolutions and 3D Swin Transformers
- Authors: Zhicheng Zou, Nantheera Anantrasirichai,
- Abstract summary: Atmospheric turbulence in long-range imaging significantly degrades the quality and fidelity captured scenes due to random variations in both spatial and temporal dimensions.
This paper proposes a new framework that combines geometric restoration with an enhancement module.
The proposed framework demonstrates superior performance over the state of the art for both synthetic and real atmospheric turbulence effects, with reasonable speed and model size.
- Score: 2.9695823613761316
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Atmospheric turbulence in long-range imaging significantly degrades the quality and fidelity of captured scenes due to random variations in both spatial and temporal dimensions. These distortions present a formidable challenge across various applications, from surveillance to astronomy, necessitating robust mitigation strategies. While model-based approaches achieve good results, they are very slow. Deep learning approaches show promise in image and video restoration but have struggled to address these spatiotemporal variant distortions effectively. This paper proposes a new framework that combines geometric restoration with an enhancement module. Random perturbations and geometric distortion are removed using a pyramid architecture with deformable 3D convolutions, resulting in aligned frames. These frames are then used to reconstruct a sharp, clear image via a multi-scale architecture of 3D Swin Transformers. The proposed framework demonstrates superior performance over the state of the art for both synthetic and real atmospheric turbulence effects, with reasonable speed and model size.
Related papers
- Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image [28.759158325097093]
Unique3D is a novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images.
Our framework features state-of-the-art generation fidelity and strong generalizability.
arXiv Detail & Related papers (2024-05-30T17:59:54Z) - VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting.
Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
arXiv Detail & Related papers (2024-02-27T11:40:50Z) - TriHuman : A Real-time and Controllable Tri-plane Representation for
Detailed Human Geometry and Appearance Synthesis [76.73338151115253]
TriHuman is a novel human-tailored, deformable, and efficient tri-plane representation.
We non-rigidly warp global ray samples into our undeformed tri-plane texture space.
We show how such a tri-plane feature representation can be conditioned on the skeletal motion to account for dynamic appearance and geometry changes.
arXiv Detail & Related papers (2023-12-08T16:40:38Z) - AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using
Denoising Diffusion Probabilistic Models [64.24948495708337]
Atmospheric turbulence causes significant degradation to image quality by introducing blur and geometric distortion.
Various deep learning-based single image atmospheric turbulence mitigation methods, including CNN-based and GAN inversion-based, have been proposed.
Denoising Diffusion Probabilistic Models (DDPMs) have recently gained some traction because of their stable training process and their ability to generate high quality images.
arXiv Detail & Related papers (2022-08-24T03:13:04Z) - Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A
New Physics-Inspired Transformer Model [82.23276183684001]
We propose a physics-inspired transformer model for imaging through atmospheric turbulence.
The proposed network utilizes the power of transformer blocks to jointly extract a dynamical turbulence distortion map.
We present two new real-world turbulence datasets that allow for evaluation with both classical objective metrics and a new task-driven metric using text recognition accuracy.
arXiv Detail & Related papers (2022-07-20T17:09:16Z) - Atmospheric Turbulence Removal with Complex-Valued Convolutional Neural
Network [2.657505380055164]
Atmospheric turbulence distorts visual imagery and is always problematic for information interpretation by both human and machine.
Deep learning-based approaches have gained more attention but currently work efficiently only on static scenes.
This paper presents a novel learning-based framework offering short temporal spanning to support dynamic scenes.
arXiv Detail & Related papers (2022-04-14T14:29:32Z) - Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori.
Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry.
This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
arXiv Detail & Related papers (2022-04-09T03:46:18Z) - Image Reconstruction of Static and Dynamic Scenes through Anisoplanatic
Turbulence [1.6114012813668934]
We present a unified method for atmospheric turbulence mitigation in both static and dynamic sequences.
We are able to achieve better results compared to existing methods by utilizing a novel space-time non-local averaging method.
arXiv Detail & Related papers (2020-08-31T19:20:46Z) - Learning to Restore a Single Face Image Degraded by Atmospheric
Turbulence using CNNs [93.72048616001064]
Images captured under such condition suffer from a combination of geometric deformation and space varying blur.
We present a deep learning-based solution to the problem of restoring a turbulence-degraded face image.
arXiv Detail & Related papers (2020-07-16T15:25:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.