InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model

• URL: http://arxiv.org/abs/2603.18031v1
• Date: Sun, 08 Mar 2026 18:08:38 GMT
• Title: InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
• Abstract summary: We present a consistency boundary analysis that characterizes when diagonal short-memory SSMs can approximate causal attention.<n>Motivated by this analysis, we propose InfoMamba, an attention-free hybrid architecture.
• Score: 5.889846622892644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Balancing fine-grained local modeling with long-range dependency capture under computational constraints remains a central challenge in sequence modeling. While Transformers provide strong token mixing, they suffer from quadratic complexity, whereas Mamba-style selective state-space models (SSMs) scale linearly but often struggle to capture high-rank and synchronous global interactions. We present a consistency boundary analysis that characterizes when diagonal short-memory SSMs can approximate causal attention and identifies structural gaps that remain. Motivated by this analysis, we propose InfoMamba, an attention-free hybrid architecture. InfoMamba replaces token-level self-attention with a concept bottleneck linear filtering layer that serves as a minimal-bandwidth global interface and integrates it with a selective recurrent stream through information-maximizing fusion (IMF). IMF dynamically injects global context into the SSM dynamics and encourages complementary information usage through a mutual-information-inspired objective. Extensive experiments on classification, dense prediction, and non-vision tasks show that InfoMamba consistently outperforms strong Transformer and SSM baselines, achieving competitive accuracy-efficiency trade-offs while maintaining near-linear scaling.

Related papers

• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• MHMamba: Multi-Head Mamba for 3D Brain Tumor Segmentation [6.1337977581640075]
  This paper proposes Multi-Head Mamba (MHMamba) to address the limitations of CNNs in modeling long-range dependencies.<n>Method combines a U-shaped architecture with a multi-head state-space model (Mamba), splitting the channel dimension into parallel SSM heads and aggregating them with residuals.<n>Results: MHMamba achieved stable and significant improvements in overall accuracy, boundary smoothness, and sensitivity to tumor core and small-volume enhancement areas.
  arXiv  Detail & Related papers  (2026-05-15T10:05:30Z)
• AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba [56.52470564147458]
  We propose textbfAlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis.<n>Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy.<n>More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts.
  arXiv  Detail & Related papers  (2026-03-19T03:47:21Z)
• TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition [59.99922360648663]
  TSkel-Mamba is a hybrid Transformer-Mamba framework that effectively captures both spatial and temporal dynamics.<n>The MTI module employs multi-scale Cycle operators to capture cross-channel temporal interactions, a critical factor in action recognition.
  arXiv  Detail & Related papers  (2025-12-12T11:55:16Z)
• FAIM: Frequency-Aware Interactive Mamba for Time Series Classification [87.84511960413715]
  Time series classification (TSC) is crucial in numerous real-world applications, such as environmental monitoring, medical diagnosis, and posture recognition.<n>We propose FAIM, a lightweight Frequency-Aware Interactive Mamba model.<n>We show that FAIM consistently outperforms existing state-of-the-art (SOTA) methods, achieving a superior trade-off between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-26T08:36:33Z)
• HyM-UNet: Synergizing Local Texture and Global Context via Hybrid CNN-Mamba Architecture for Medical Image Segmentation [3.976000861085382]
  HyM-UNet is designed to synergize the local feature extraction capabilities of CNNs with the efficient global modeling capabilities of Mamba.<n>To bridge the semantic gap between the encoder and the decoder, we propose a Mamba-Guided Fusion Skip Connection.<n>The results demonstrate that HyM-UNet significantly outperforms existing state-of-the-art methods in terms of Dice coefficient and IoU.
  arXiv  Detail & Related papers  (2025-11-22T09:02:06Z)
• MVNet: Hyperspectral Remote Sensing Image Classification Based on Hybrid Mamba-Transformer Vision Backbone Architecture [12.168520751389622]
  Hyperspectral image (HSI) classification faces challenges such as high-dimensional data, limited training samples, and spectral redundancy.<n>This paper proposes a novel MVNet network architecture that integrates 3D-CNN's local feature extraction, Transformer's global modeling, and Mamba's linear sequence modeling capabilities.<n>On IN, UP, and KSC datasets, MVNet outperforms mainstream hyperspectral image classification methods in both classification accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-07-06T14:52:26Z)
• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection [88.47928738482719]
  Linear State Space Models (SSMs) offer remarkable performance gains in sequence modeling.<n>Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations.<n>We introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts.
  arXiv  Detail & Related papers  (2025-06-22T19:26:55Z)
• VRS-UIE: Value-Driven Reordering Scanning for Underwater Image Enhancement [104.78586859995333]
  State Space Models (SSMs) have emerged as a promising backbone for vision tasks due to their linear complexity and global receptive field.<n>The predominance of large-portion, homogeneous but useless oceanic backgrounds can dilute the feature representation responses of sparse yet valuable targets.<n>We propose a novel Value-Driven Reordering Scanning framework for Underwater Image Enhancement (UIE)<n>Our framework sets a new state-of-the-art, delivering superior enhancement performance (surpassing WMamba by 0.89 dB on average) by effectively suppressing water bias and preserving structural and color fidelity.
  arXiv  Detail & Related papers  (2025-05-02T12:21:44Z)
• DynSTG-Mamba: Dynamic Spatio-Temporal Graph Mamba with Cross-Graph Knowledge Distillation for Gait Disorders Recognition [0.3517488417793045]
  DynTG-Mamba is a novel framework that combines DF-STGNN and STG-Mamba to enhance motion modeling.<n> DF-STGNN incorporates a dynamic spatial filter that adaptively adjusts between skeletal joints and temporal interactions.<n> STG-Mamba, an extension of Mamba, ensures a continuous propagation of states while reducing computational costs.
  arXiv  Detail & Related papers  (2025-03-17T13:26:47Z)
• SIGMA: Selective Gated Mamba for Sequential Recommendation [56.85338055215429]
  Mamba, a recent advancement, has exhibited exceptional performance in time series prediction.<n>We introduce a new framework named Selective Gated Mamba ( SIGMA) for Sequential Recommendation.<n>Our results indicate that SIGMA outperforms current models on five real-world datasets.
  arXiv  Detail & Related papers  (2024-08-21T09:12:59Z)
• FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba [19.761723108363796]
  FusionMamba aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks.<n>The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms.<n>Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments.
  arXiv  Detail & Related papers  (2024-04-15T06:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.