Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Prediction

• URL: http://arxiv.org/abs/2508.20376v1
• Date: Thu, 28 Aug 2025 02:50:19 GMT
• Title: Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Prediction
• Authors: Mang Cao, Sanping Zhou, Yizhe Li, Ye Deng, Wenli Huang, Le Wang,
• Abstract summary: Cross-task interaction is crucial for success in multi-task dense prediction.<n>Existing methods face the trade-off between interaction completeness and computational efficiency.<n>This work proposes a Bidirectional Interaction Mamba, which incorporates novel scanning mechanisms.
• Score: 37.625609555296364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sufficient cross-task interaction is crucial for success in multi-task dense prediction. However, sufficient interaction often results in high computational complexity, forcing existing methods to face the trade-off between interaction completeness and computational efficiency. To address this limitation, this work proposes a Bidirectional Interaction Mamba (BIM), which incorporates novel scanning mechanisms to adapt the Mamba modeling approach for multi-task dense prediction. On the one hand, we introduce a novel Bidirectional Interaction Scan (BI-Scan) mechanism, which constructs task-specific representations as bidirectional sequences during interaction. By integrating task-first and position-first scanning modes within a unified linear complexity architecture, BI-Scan efficiently preserves critical cross-task information. On the other hand, we employ a Multi-Scale Scan~(MS-Scan) mechanism to achieve multi-granularity scene modeling. This design not only meets the diverse granularity requirements of various tasks but also enhances nuanced cross-task feature interactions. Extensive experiments on two challenging benchmarks, \emph{i.e.}, NYUD-V2 and PASCAL-Context, show the superiority of our BIM vs its state-of-the-art competitors.

Related papers

• Parameter Aware Mamba Model for Multi-task Dense Prediction [69.94454603308196]
  We introduce a novel decoder-based framework, Aware Mamba Model (PAMM), specifically designed for dense prediction in multi-task learning setting.<n>It features dual state space parameter experts that integrate and set task-specific parameter priors, capturing the intrinsic properties of each task.<n>We employ the Multi-Directional Hilbert Scanning method to construct multi-angle feature sequences, thereby enhancing the sequence model's perceptual capabilities for 2D data.
  arXiv  Detail & Related papers  (2025-11-18T13:48:00Z)
• MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization [52.149337961205624]
  We propose a framework that empowers both inter- and intra-task optimization for surgical triplet recognition.<n>For inter-task optimization, we introduce the Shared-Specific-Disentangled (S$2$D) learning scheme that decomposes representations into task-shared and task-specific components.<n>For intra-task optimization conflicts, we develop a Coordinated Gradient Learning (CGL) strategy, which dissects and rebalances the positive-negative ambiguities.
  arXiv  Detail & Related papers  (2025-09-16T09:48:52Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout.<n>DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder.<n>Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Task Indicating Transformer for Task-conditional Dense Predictions [16.92067246179703]
  We introduce a novel task-conditional framework called Task Indicating Transformer (TIT) to tackle this challenge. Our approach designs a Mix Task Adapter module within the transformer block, which incorporates a Task Indicating Matrix through matrix decomposition. We also propose a Task Gate Decoder module that harnesses a Task Indicating Vector and gating mechanism to facilitate adaptive multi-scale feature refinement.
  arXiv  Detail & Related papers  (2024-03-01T07:06:57Z)
• BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-task Dense Predictions [29.049866510120093]
  Multi-task dense prediction aims at handling multiple pixel-wise prediction tasks within a unified network simultaneously for visual scene understanding. To tackle these under-explored issues, we propose a novel BridgeNet framework, which extracts comprehensive and discriminative intermediate Bridge Features. To the best of our knowledge, this is the first work considering the completeness and quality of feature participants in cross-task interactions.
  arXiv  Detail & Related papers  (2023-12-21T01:30:44Z)
• MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts [92.76662894585809]
  We introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE) MMoE is able to be applied to various types of models to gain improvement.
  arXiv  Detail & Related papers  (2023-11-16T05:31:21Z)
• Contrastive Multi-Task Dense Prediction [11.227696986100447]
  A core objective in design is how to effectively model cross-task interactions to achieve a comprehensive improvement on different tasks. We introduce feature-wise contrastive consistency into modeling the cross-task interactions for multi-task dense prediction. We propose a novel multi-task contrastive regularization method based on the consistency to effectively boost the representation learning of the different sub-tasks.
  arXiv  Detail & Related papers  (2023-07-16T03:54:01Z)
• InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding [11.608682595506354]
  Multi-task scene understanding aims to design models that can simultaneously predict several scene understanding tasks with one versatile model. Previous studies typically process multi-task features in a more local way, and thus cannot effectively learn spatially global and cross-task interactions. We propose an Inverted Pyramid multi-task Transformer, capable of modeling cross-task interaction among spatial features of different tasks in a global context.
  arXiv  Detail & Related papers  (2023-06-08T00:28:22Z)
• DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learning [18.745373058797714]
  We propose a novel multi-task learning architecture that leverages pairwise cross-task exchange through correlation-guided attention and self-attention. We conduct extensive experiments across three multi-task setups, showing the advantages of our approach compared to competitive baselines in both synthetic and real-world benchmarks.
  arXiv  Detail & Related papers  (2022-06-17T17:59:45Z)
• A Co-Interactive Transformer for Joint Slot Filling and Intent Detection [61.109486326954205]
  Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. Previous studies either model the two tasks separately or only consider the single information flow from intent to slot. We propose a Co-Interactive Transformer to consider the cross-impact between the two tasks simultaneously.
  arXiv  Detail & Related papers  (2020-10-08T10:16:52Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.