MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders

• URL: http://arxiv.org/abs/2407.02228v2
• Date: Sun, 14 Jul 2024 07:50:04 GMT
• Title: MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders
• Authors: Baijiong Lin, Weisen Jiang, Pengguang Chen, Yu Zhang, Shu Liu, Ying-Cong Chen,
• Abstract summary: We propose MTMamba, a novel Mamba-based architecture for multi-task scene understanding. Experiments on NYUDv2 and PASCAL-Context datasets demonstrate the superior performance of MTMamba over Transformer-based and CNN-based methods.
• Score: 27.487314321249627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task dense scene understanding, which learns a model for multiple dense prediction tasks, has a wide range of application scenarios. Modeling long-range dependency and enhancing cross-task interactions are crucial to multi-task dense prediction. In this paper, we propose MTMamba, a novel Mamba-based architecture for multi-task scene understanding. It contains two types of core blocks: self-task Mamba (STM) block and cross-task Mamba (CTM) block. STM handles long-range dependency by leveraging Mamba, while CTM explicitly models task interactions to facilitate information exchange across tasks. Experiments on NYUDv2 and PASCAL-Context datasets demonstrate the superior performance of MTMamba over Transformer-based and CNN-based methods. Notably, on the PASCAL-Context dataset, MTMamba achieves improvements of +2.08, +5.01, and +4.90 over the previous best methods in the tasks of semantic segmentation, human parsing, and object boundary detection, respectively. The code is available at https://github.com/EnVision-Research/MTMamba.

Related papers

• ms-Mamba: Multi-scale Mamba for Time-Series Forecasting [9.008102132057733]
  ms-Mamba incorporates multiple temporal scales by using multiple Mamba blocks with different sampling rates. Our experiments on many benchmarks demonstrate that ms-Mamba outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-04-10T11:06:57Z)
• LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models [1.249658136570244]
  State space models (SSMs) have emerged as an efficient alternative to transformers for long-context sequence modeling. SSMs lack the interpretability tools that have been crucial for understanding and improving attention-based architectures. We introduce LaTIM, a novel token-level decomposition method for both Mamba-1 and Mamba-2 that enables fine-grained interpretability.
  arXiv  Detail & Related papers  (2025-02-21T17:33:59Z)
• TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba [88.31117598044725]
  We explore cross-architecture training to transfer the ready knowledge in existing Transformer models to alternative architecture Mamba, termed TransMamba. Our approach employs a two-stage strategy to expedite training new Mamba models, ensuring effectiveness in across uni-modal and cross-modal tasks. For cross-modal learning, we propose a cross-Mamba module that integrates language awareness into Mamba's visual features, enhancing the cross-modal interaction capabilities of Mamba architecture.
  arXiv  Detail & Related papers  (2025-02-21T01:22:01Z)
• Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement [54.427965535613886]
  Mamba, as a novel state-space model (SSM), has gained widespread application in natural language processing and computer vision. In this work, we introduce Mamba-SEUNet, an innovative architecture that integrates Mamba with U-Net for SE tasks.
  arXiv  Detail & Related papers  (2024-12-21T13:43:51Z)
• MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt [60.10555128510744]
  Multi-modal object Re-IDentification (ReID) aims to retrieve specific objects by utilizing complementary image information from different modalities. Recently, large-scale pre-trained models like CLIP have demonstrated impressive performance in traditional single-modal object ReID tasks. We introduce a novel framework called MambaPro for multi-modal object ReID.
  arXiv  Detail & Related papers  (2024-12-14T06:33:53Z)
• V2M: Visual 2-Dimensional Mamba for Image Representation Learning [68.51380287151927]
  Mamba has garnered widespread attention due to its flexible design and efficient hardware performance to process 1D sequences. Recent studies have attempted to apply Mamba to the visual domain by flattening 2D images into patches and then regarding them as a 1D sequence. We propose a Visual 2-Dimensional Mamba model as a complete solution, which directly processes image tokens in the 2D space.
  arXiv  Detail & Related papers  (2024-10-14T11:11:06Z)
• MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders [26.236118242986805]
  We propose MTMamba++, a novel architecture for multi-task scene understanding featuring with a Mamba-based decoder. It contains two types of core blocks: self-task Mamba block and cross-task Mamba block. Experiments on NYUDv2, PASCAL-Context, and Cityscapes datasets demonstrate the superior performance of MTMamba++ over CNN-based and Transformer-based methods.
  arXiv  Detail & Related papers  (2024-08-27T14:36:46Z)
• MambaMIM: Pre-training Mamba with State Space Token-interpolation [14.343466340528687]
  We introduce a generative self-supervised learning method for Mamba (MambaMIM) based on Selective Structure State Space Sequence Token-interpolation (S6T) MambaMIM can be used on any single or hybrid Mamba architectures to enhance the Mamba long-range representation capability.
  arXiv  Detail & Related papers  (2024-08-15T10:35:26Z)
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone [54.965143338206644]
  We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision, which is specifically tailored for vision applications. Our core contribution includes redesigning the Mamba formulation to enhance its capability for efficient modeling of visual features. We conduct a comprehensive ablation study on the feasibility of integrating Vision Transformers (ViT) with Mamba.
  arXiv  Detail & Related papers  (2024-07-10T23:02:45Z)
• MambaOut: Do We Really Need Mamba for Vision? [70.60495392198686]
  Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism. This paper conceptually concludes that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. We construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM.
  arXiv  Detail & Related papers  (2024-05-13T17:59:56Z)
• Visual Mamba: A Survey and New Outlooks [33.90213491829634]
  Mamba, a recent selective structured state space model, excels in long sequence modeling. Since January 2024, Mamba has been actively applied to diverse computer vision tasks. This paper reviews visual Mamba approaches, analyzing over 200 papers.
  arXiv  Detail & Related papers  (2024-04-29T16:51:30Z)
• ReMamber: Referring Image Segmentation with Mamba Twister [51.291487576255435]
  ReMamber is a novel RIS architecture that integrates the power of Mamba with a multi-modal Mamba Twister block. The Mamba Twister explicitly models image-text interaction, and fuses textual and visual features through its unique channel and spatial twisting mechanism.
  arXiv  Detail & Related papers  (2024-03-26T16:27:37Z)
• MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection [72.46396769642787]
  We develop a nested structure, Mamba-in-Mamba (MiM-ISTD), for efficient infrared small target detection. MiM-ISTD is $8 times$ faster than the SOTA method and reduces GPU memory usage by 62.2$%$ when testing on $2048 times 2048$ images.
  arXiv  Detail & Related papers  (2024-03-04T15:57:29Z)
• SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation [16.476244833079182]
  We introduce SegMamba, a novel 3D medical image textbfSegmentation textbfMamba model. SegMamba excels in whole volume feature modeling from a state space model standpoint. Experiments on the BraTS2023 dataset demonstrate the effectiveness and efficiency of our SegMamba.
  arXiv  Detail & Related papers  (2024-01-24T16:17:23Z)
• MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning [82.62433731378455]
  We show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales. We propose a novel architecture, namely MTI-Net, that builds upon this finding.
  arXiv  Detail & Related papers  (2020-01-19T21:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.