Physical Perception Network and an All-weather Multi-modality Benchmark for Adverse Weather Image Fusion

• URL: http://arxiv.org/abs/2402.02090v1
• Date: Sat, 3 Feb 2024 09:02:46 GMT
• Title: Physical Perception Network and an All-weather Multi-modality Benchmark for Adverse Weather Image Fusion
• Authors: Xilai Li, Wuyang Liu, Xiaosong Li, Haishu Tan
• Abstract summary: Multi-modality image fusion (MMIF) integrates the complementary information from different modal images to provide comprehensive and objective interpretation of a scenes. Existing MMIF methods lack the ability to resist different weather interferences in real-life scenarios. We have established a benchmark for MMIF research under extreme weather conditions.
• Score: 4.3773535988950725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modality image fusion (MMIF) integrates the complementary information from different modal images to provide comprehensive and objective interpretation of a scenes. However, existing MMIF methods lack the ability to resist different weather interferences in real-life scenarios, preventing them from being useful in practical applications such as autonomous driving. To bridge this research gap, we proposed an all-weather MMIF model. Regarding deep learning architectures, their network designs are often viewed as a black box, which limits their multitasking capabilities. For deweathering module, we propose a physically-aware clear feature prediction module based on an atmospheric scattering model that can deduce variations in light transmittance from both scene illumination and depth. For fusion module, We utilize a learnable low-rank representation model to decompose images into low-rank and sparse components. This highly interpretable feature separation allows us to better observe and understand images. Furthermore, we have established a benchmark for MMIF research under extreme weather conditions. It encompasses multiple scenes under three types of weather: rain, haze, and snow, with each weather condition further subdivided into various impact levels. Extensive fusion experiments under adverse weather demonstrate that the proposed algorithm has excellent detail recovery and multi-modality feature extraction capabilities.

Related papers

• Multiple weather images restoration using the task transformer and adaptive mixup strategy [14.986500375481546]
  We introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.
  arXiv  Detail & Related papers  (2024-09-05T04:55:40Z)
• Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion [5.417493475406649]
  Multi-modal image fusion (MMIF) integrates valuable information from different modality images into a fused one. This paper proposes a MMIF framework for joint focused integration and modalities information extraction. The proposed algorithm can surpass the state-of-the-art methods in visual perception and quantitative evaluation.
  arXiv  Detail & Related papers  (2023-11-03T12:58:39Z)
• Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal [97.53040662243768]
  We propose a CLIP embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters.
  arXiv  Detail & Related papers  (2023-06-15T10:06:13Z)
• Counting Crowds in Bad Weather [68.50690406143173]
  We propose a method for robust crowd counting in adverse weather scenarios. Our model learns effective features and adaptive queries to account for large appearance variations. Experimental results show that the proposed algorithm is effective in counting crowds under different weather types on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-02T00:00:09Z)
• Contrastive Learning Based Recursive Dynamic Multi-Scale Network for Image Deraining [47.764883957379745]
  Rain streaks significantly decrease the visibility of captured images. Existing deep learning-based image deraining methods employ manually crafted networks and learn a straightforward projection from rainy images to clear images. We propose a contrastive learning-based image deraining method that investigates the correlation between rainy and clear images.
  arXiv  Detail & Related papers  (2023-05-29T13:51:41Z)
• WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal [38.257012295118145]
  Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. We propose Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset.
  arXiv  Detail & Related papers  (2023-03-24T01:46:25Z)
• Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion [59.19469551774703]
  Infrared and visible image fusion aims to integrate comprehensive information from multiple sources to achieve superior performances on various practical tasks. We propose a dynamic image fusion framework with a multi-modal gated mixture of local-to-global experts. Our model consists of a Mixture of Local Experts (MoLE) and a Mixture of Global Experts (MoGE) guided by a multi-modal gate.
  arXiv  Detail & Related papers  (2023-02-02T20:06:58Z)
• Competitive Simplicity for Multi-Task Learning for Real-Time Foggy Scene Understanding via Domain Adaptation [17.530091734327296]
  We propose a multi-task learning approach capable of performing in real-time semantic scene understanding and monocular depth estimation under foggy weather conditions. Our model incorporates RGB colour, depth, and luminance images via distinct encoders with dense connectivity and features fusing.
  arXiv  Detail & Related papers  (2020-12-09T20:38:34Z)
• Crowdsampling the Plenoptic Function [56.10020793913216]
  We present a new approach to novel view synthesis under time-varying illumination from such data. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time.
  arXiv  Detail & Related papers  (2020-07-30T02:52:10Z)
• Multi-Scale Progressive Fusion Network for Single Image Deraining [84.0466298828417]
  Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features.
  arXiv  Detail & Related papers  (2020-03-24T17:22:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.