SpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation

• URL: http://arxiv.org/abs/2303.08692v2
• Date: Wed, 27 Sep 2023 10:03:09 GMT
• Title: SpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation
• Authors: Siqi Fan, Zhe Wang, Yan Wang, Jingjing Liu
• Abstract summary: We propose a Spatial-aware Demand-guided Recursive Meshing (SpiderMesh) framework for practical RGB-T (thermal) segmentation. SpiderMesh proactively compensates inadequate contextual semantics in optically-impaired regions. Experiments on MFNet and PST900 datasets demonstrate that SpiderMesh achieves state-of-the-art performance on standard RGB-T segmentation benchmarks.
• Score: 13.125707028339292
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology in challenging lighting conditions. Thermal signal is an informative additional channel that can bring to light the contour and fine-grained texture of blurred regions in low-quality RGB image. Aiming at practical RGB-T (thermal) segmentation, we systematically propose a Spatial-aware Demand-guided Recursive Meshing (SpiderMesh) framework that: 1) proactively compensates inadequate contextual semantics in optically-impaired regions via a demand-guided target masking algorithm; 2) refines multimodal semantic features with recursive meshing to improve pixel-level semantic analysis performance. We further introduce an asymmetric data augmentation technique M-CutOut, and enable semi-supervised learning to fully utilize RGB-T labels only sparsely available in practical use. Extensive experiments on MFNet and PST900 datasets demonstrate that SpiderMesh achieves state-of-the-art performance on standard RGB-T segmentation benchmarks.

Related papers

• Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer [10.982521876026281]
  We introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. We demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements.
  arXiv  Detail & Related papers  (2024-09-23T15:23:01Z)
• GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding [101.32590239809113]
  Generalized Perception NeRF (GP-NeRF) is a novel pipeline that makes the widely used segmentation model and NeRF work compatibly under a unified framework. We propose two self-distillation mechanisms, i.e., the Semantic Distill Loss and the Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality of the semantic field.
  arXiv  Detail & Related papers  (2023-11-20T15:59:41Z)
• Channel and Spatial Relation-Propagation Network for RGB-Thermal Semantic Segmentation [10.344060599932185]
  RGB-Thermal (RGB-T) semantic segmentation has shown great potential in handling low-light conditions. The key to RGB-T semantic segmentation is to effectively leverage the complementarity nature of RGB and thermal images.
  arXiv  Detail & Related papers  (2023-08-24T03:43:47Z)
• Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation [19.41334573257174]
  Traditional methods mostly use RGB images which are heavily affected by lighting conditions, eg, darkness. Recent studies show thermal images are robust to the night scenario as a compensating modality for segmentation. This work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-17T14:28:08Z)
• Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution [52.582632746409665]
  We propose a novel Symmetric Uncertainty-aware Feature Transmission (SUFT) for color-guided DSR. Our method achieves superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-01T06:35:59Z)
• Complementary Random Masking for RGB-Thermal Semantic Segmentation [63.93784265195356]
  RGB-thermal semantic segmentation is a potential solution to achieve reliable semantic scene understanding in adverse weather and lighting conditions. This paper proposes 1) a complementary random masking strategy of RGB-T images and 2) self-distillation loss between clean and masked input modalities. We achieve state-of-the-art performance over three RGB-T semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-03-30T13:57:21Z)
• Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution [123.04455334124188]
  Guided depth map super-resolution (GDSR) aims to upsample low-resolution (LR) depth maps with additional information involved in high-resolution (HR) RGB images from the same scene. In this paper, we propose the Spherical Space feature Decomposition Network (SSDNet) to solve the above issues. Our method can achieve state-of-the-art results on four test datasets, as well as successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2023-03-15T21:22:21Z)
• Mirror Complementary Transformer Network for RGB-thermal Salient Object Detection [16.64781797503128]
  RGB-thermal object detection (RGB-T SOD) aims to locate the common prominent objects of an aligned visible and thermal infrared image pair. In this paper, we propose a novel mirror complementary Transformer network (MCNet) for RGB-T SOD. Experiments on benchmark and VT723 datasets show that the proposed method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-07-07T20:26:09Z)
• TBNet:Two-Stream Boundary-aware Network for Generic Image Manipulation Localization [49.521622399483846]
  We propose a novel end-to-end two-stream boundary-aware network (abbreviated as TBNet) for generic image manipulation localization. The proposed TBNet can significantly outperform state-of-the-art generic image manipulation localization methods in terms of both MCC and F1.
  arXiv  Detail & Related papers  (2021-08-10T08:22:05Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.