AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation

• URL: http://arxiv.org/abs/2309.14065v7
• Date: Wed, 17 Apr 2024 10:42:06 GMT
• Title: AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation
• Authors: Siqi Du, Weixi Wang, Renzhong Guo, Ruisheng Wang, Yibin Tian, Shengjun Tang,
• Abstract summary: AsymFormer is a novel network that improves real-time semantic segmentation accuracy using RGB-D multi-modal information. AsymFormer uses an asymmetrical backbone for multimodal feature extraction, reducing redundant parameters. AsymFormer can strike a balance between high accuracy and efficiency.
• Score: 4.54656749337919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding indoor scenes is crucial for urban studies. Considering the dynamic nature of indoor environments, effective semantic segmentation requires both real-time operation and high accuracy.To address this, we propose AsymFormer, a novel network that improves real-time semantic segmentation accuracy using RGB-D multi-modal information without substantially increasing network complexity. AsymFormer uses an asymmetrical backbone for multimodal feature extraction, reducing redundant parameters by optimizing computational resource distribution. To fuse asymmetric multimodal features, a Local Attention-Guided Feature Selection (LAFS) module is used to selectively fuse features from different modalities by leveraging their dependencies. Subsequently, a Cross-Modal Attention-Guided Feature Correlation Embedding (CMA) module is introduced to further extract cross-modal representations. The AsymFormer demonstrates competitive results with 54.1% mIoU on NYUv2 and 49.1% mIoU on SUNRGBD. Notably, AsymFormer achieves an inference speed of 65 FPS (79 FPS after implementing mixed precision quantization) on RTX3090, demonstrating that AsymFormer can strike a balance between high accuracy and efficiency.

Related papers

• Two-stream Beats One-stream: Asymmetric Siamese Network for Efficient Visual Tracking [54.124445709376154]
  We propose a novel asymmetric Siamese tracker named textbfAsymTrack for efficient tracking. Building on this architecture, we devise an efficient template modulation mechanism to inject crucial cues into the search features. Experiments demonstrate that AsymTrack offers superior speed-precision trade-offs across different platforms.
  arXiv  Detail & Related papers  (2025-03-01T14:44:54Z)
• A feature refinement module for light-weight semantic segmentation network [11.285793559719702]
  This paper proposes a novel semantic segmentation method to improve the capacity of obtaining semantic information for the light-weight network. On Cityscapes and Bdd100K datasets, the experimental results demonstrate that the proposed method achieves a promising trade-off between accuracy and computational cost.
  arXiv  Detail & Related papers  (2024-12-11T03:31:20Z)
• HAFormer: Unleashing the Power of Hierarchy-Aware Features for Lightweight Semantic Segmentation [11.334990474402915]
  We introduce HAFormer, a model that combines the hierarchical features extraction ability of CNNs with the global dependency modeling capability of Transformers. HAFormer achieves high performance with minimal computational overhead and compact model size.
  arXiv  Detail & Related papers  (2024-07-10T07:53:24Z)
• CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes [0.0]
  multimodal semantic segmentation methods suffer from high computational complexity and low inference speed. We propose the Cosine Similarity Fusion Network (CSFNet) as a real-time RGB-X semantic segmentation model. CSFNet has competitive accuracy with state-of-the-art methods while being state-of-the-art in terms of speed.
  arXiv  Detail & Related papers  (2024-07-01T14:34:32Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Mutual-Guided Dynamic Network for Image Fusion [51.615598671899335]
  We propose a novel mutual-guided dynamic network (MGDN) for image fusion, which allows for effective information utilization across different locations and inputs. Experimental results on five benchmark datasets demonstrate that our proposed method outperforms existing methods on four image fusion tasks.
  arXiv  Detail & Related papers  (2023-08-24T03:50:37Z)
• WCCNet: Wavelet-integrated CNN with Crossmodal Rearranging Fusion for Fast Multispectral Pedestrian Detection [16.43119521684829]
  We propose a novel framework named WCCNet that is able to differentially extract rich features of different spectra with lower computational complexity. Based on the well extracted features, we elaborately design the crossmodal rearranging fusion module (CMRF) We conduct comprehensive evaluations on KAIST and FLIR benchmarks, in which WCCNet outperforms state-of-the-art methods with considerable computational efficiency and competitive accuracy.
  arXiv  Detail & Related papers  (2023-08-02T09:35:21Z)
• Cross-CBAM: A Lightweight network for Scene Segmentation [2.064612766965483]
  We present the Cross-CBAM network, a novel lightweight network for real-time semantic segmentation. In experiments on the Cityscapes dataset and Camvid dataset, we achieve 73.4% mIoU with a speed of 240.9FPS and 77.2% mIoU with a speed of 88.6FPS on NVIDIA GTX 1080Ti.
  arXiv  Detail & Related papers  (2023-06-04T09:03:05Z)
• Complementary Random Masking for RGB-Thermal Semantic Segmentation [63.93784265195356]
  RGB-thermal semantic segmentation is a potential solution to achieve reliable semantic scene understanding in adverse weather and lighting conditions. This paper proposes 1) a complementary random masking strategy of RGB-T images and 2) self-distillation loss between clean and masked input modalities. We achieve state-of-the-art performance over three RGB-T semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-03-30T13:57:21Z)
• Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution [90.16462805389943]
  We develop a spatially-adaptive feature modulation (SAFM) mechanism upon a vision transformer (ViT)-like block. Proposed method is $3times$ smaller than state-of-the-art efficient SR methods.
  arXiv  Detail & Related papers  (2023-02-27T14:19:31Z)
• MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation [27.232578592161673]
  We devise a novel lightweight network using a multi-scale context fusion scheme (MSCFNet) The proposed MSCFNet contains only 1.15M parameters, achieves 71.9% Mean IoU and can run at over 50 FPS on a single Titan XP GPU configuration.
  arXiv  Detail & Related papers  (2021-03-24T08:28:26Z)
• GhostSR: Learning Ghost Features for Efficient Image Super-Resolution [49.393251361038025]
  Single image super-resolution (SISR) system based on convolutional neural networks (CNNs) achieves fancy performance while requires huge computational costs. We propose to use shift operation to generate the redundant features (i.e., Ghost features) of SISR models. We show that both the non-compact and lightweight SISR models embedded in our proposed module can achieve comparable performance to that of their baselines.
  arXiv  Detail & Related papers  (2021-01-21T10:09:47Z)
• Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection [91.43066633305662]
  The main purpose of RGB-D salient object detection (SOD) is how to better integrate and utilize cross-modal fusion information. In this paper, we explore these issues from a new perspective. We implement a kind of more flexible and efficient multi-scale cross-modal feature processing.
  arXiv  Detail & Related papers  (2020-07-13T07:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.