ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning

• URL: http://arxiv.org/abs/2604.08050v1
• Date: Thu, 09 Apr 2026 09:58:56 GMT
• Title: ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
• Abstract summary: We focus on video captioning by fully open multimodal large language models (MLLMs)<n>ABMamba is a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences.<n>On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs.
• Score: 11.388109553982089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core attention mechanisms of existing Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences. ABMamba extends Deep State Space Models as its language backbone, replacing the costly quadratic attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes videos across multiple temporal resolutions. On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.

Related papers

• Kwai Keye-VL-2.0 Technical Report [53.82434681649277]
  Keye-VL-2.0 is a multimodal foundation model designed to advance long-video understanding and agentic intelligence.<n>It is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures.<n>Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL overcomes the algorithmic dilemma of catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-06-09T09:58:08Z)
• Empowering Video Translation using Multimodal Large Language Models [33.07174302639983]
  multimodal large language models (MLLMs) are playing an increasingly important supporting role in video translation.<n>With strong multimodal understanding, reasoning, and generation capabilities, MLLMs-based video translation systems are overcoming the limitations of traditional cascaded pipelines.<n>We provide the first comprehensive overview of MLLMs-based video translation, organized around a three-role taxonomy.
  arXiv  Detail & Related papers  (2026-04-13T10:42:31Z)
• Mamba-VMR: Multimodal Query Augmentation via Generated Videos for Precise Temporal Grounding [19.92734717848329]
  Text-driven video moment retrieval (VMR) remains challenging due to limited capture of hidden temporal dynamics in untrimmed videos.<n>Existing approaches fail to integrate subtitle contexts and generated temporal priors effectively.<n>We propose a novel two-stage framework for enhanced temporal grounding.
  arXiv  Detail & Related papers  (2026-03-23T15:44:48Z)
• ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding [12.236081012244533]
  This study focuses on video understanding by large language models (MLLMs)<n>We propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations.<n>We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2026-02-18T12:37:35Z)
• LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs [52.24096832965001]
  We present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method.<n>The PVC method can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding.<n>Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x.
  arXiv  Detail & Related papers  (2025-11-26T08:11:10Z)
• MLVTG: Mamba-Based Feature Alignment and LLM-Driven Purification for Multi-Modal Video Temporal Grounding [13.025856914576673]
  Video Temporal Grounding aims to localize video clips corresponding to natural language queries.<n>Existing Transformer-based methods often suffer from redundant attention and suboptimal multi-modal alignment.<n>We propose MLVTG, a novel framework that integrates two key modules: MambaAligner and LLMRefiner.
  arXiv  Detail & Related papers  (2025-06-10T07:20:12Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• BIMBA: Selective-Scan Compression for Long-Range Video Question Answering [46.199493246921435]
  Video Question Answering (VQA) in long videos poses the key challenge of extracting relevant information.<n>We introduce BIMBA, an efficient state-space model to handle long-form videos.
  arXiv  Detail & Related papers  (2025-03-12T17:57:32Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing [52.050036778325094]
  Video-Ma$2$mba is a novel architecture that incorporates State Space Models (SSMs) within the Mamba-2 framework.<n>Our approach significantly reduces the memory footprint compared to standard gradient checkpointing.<n>By maintaining a detailed capture of temporal dynamics, our model improves the accuracy and relevance of responses in long video understanding tasks.
  arXiv  Detail & Related papers  (2024-11-29T04:12:13Z)
• Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities [67.89368528234394]
  One of the main challenges of multimodal learning is the need to combine heterogeneous modalities. Video and audio are obtained at much higher rates than text and are roughly aligned in time. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models.
  arXiv  Detail & Related papers  (2023-11-09T19:15:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.