M$^4$-SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection

• URL: http://arxiv.org/abs/2605.11760v1
• Date: Tue, 12 May 2026 08:33:08 GMT
• Title: M$^4$-SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection
• Abstract summary: We present Multi-Modal Mixture-of-Experts with Memory-Augmented SAM (M$4$-SAM), which equips SAM2 with modality-related PEFT and hierarchical feature fusion.<n>M$4$-SAM achieves the state-of-the-art performance across all evaluation metrics on three public RGB-D VSOD datasets.
• Score: 43.183940069110186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model 2 (SAM2) has emerged as a foundation model for universal segmentation. Owing to its generalizable visual representations, SAM2 has been successfully applied to various downstream tasks. However, extending SAM2 to the RGB-D video salient object detection (RGB-D VSOD) task encounters three challenges including limited spatial modeling of linear LoRA, insufficient employment of SAM's multi-scale features, and dependence of initialization on explicit prompts. To address the issues, we present Multi-Modal Mixture-of-Experts with Memory-Augmented SAM (M$^4$-SAM), which equips SAM2 with modality-related PEFT, hierarchical feature fusion, and prompt-free memory initialization. Firstly, we inject Modality-Aware MoE-LORA, which employs convolutional experts to encode local spatial priors and introduces a modality dispatcher for efficient multi-modal fine-tuning, into SAM2's encoder. Secondly, we deploy Gated Multi-Level Feature Fusion, which hierarchically aggregates multi-scale encoder features with an adaptive gating mechanism, to balance spatial details and semantic context. Finally, to conduct zero-shot VSOD without manual prompts, we utilize a Pseudo-Guided Initialization, where a coarse mask is regarded as a pseudo prior and used to bootstrap the memory bank. Extensive experiments demonstrate that M$^4$-SAM achieves the state-of-the-art performance across all evaluation metrics on three public RGB-D VSOD datasets.

Related papers

• SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting [13.388638483362628]
  textbfSAM+D, a parameter-efficient framework, lifts SAM-family models by one spatial dimension.<n>textbfSAM+D introduces two lightweight, model-agnostic modules into frozen transformer blocks.<n>textbfSAM+D generalizes across SAM-family architectures, target dimensional (3D, 4D), and domains spanning medical imaging and bio-scene understanding.
  arXiv  Detail & Related papers  (2026-07-31T05:11:25Z)
• TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model [9.471131843884953]
  TinySAM 2 is a lightweight video segmentation model that balances performance and efficiency.<n>Experiments on challenging datasets such as DAVIS and SA-V demonstrate that TinySAM 2 achieves 90% of the performance of SAM 2.1.
  arXiv  Detail & Related papers  (2026-05-18T08:05:59Z)
• Evaluating SAM2 for Video Semantic Segmentation [60.157605818225186]
  The Anything Model 2 (SAM2) has proven to be a powerful foundation model for promptable visual object segmentation in both images and videos.<n>This paper explores the extension of SAM2 to dense Video Semantic (VSS)<n>Our experiments suggest that leveraging SAM2 enhances overall performance in VSS, primarily due to its precise predictions of object boundaries.
  arXiv  Detail & Related papers  (2025-12-01T15:15:16Z)
• SAM-DAQ: Segment Anything Model with Depth-guided Adaptive Queries for RGB-D Video Salient Object Detection [44.480885765890925]
  We propose a novel method, namely Segment Anything Model with Depth-guided Adaptive Queries (SAM-DAQ)<n>SAM-DAQ adapts SAM2 to pop-out salient objects from videos by seamlessly integrating depth and temporal cues within a unified framework.<n>Experiments are conducted on three RGB-D VSOD datasets, and the results show that the proposed SAM-DAQ consistently outperforms state-of-the-art methods in terms of all evaluation metrics.
  arXiv  Detail & Related papers  (2025-11-13T02:04:03Z)
• SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks [50.97089872043121]
  We propose SAM2-UNeXT, an advanced framework that builds upon the core principles of SAM2-UNet.<n>We extend the representational capacity of SAM2 through the integration of an auxiliary DINOv2 encoder.<n>Our approach enables more accurate segmentation with a simple architecture, relaxing the need for complex decoder designs.
  arXiv  Detail & Related papers  (2025-08-05T15:36:13Z)
• MGD-SAM2: Multi-view Guided Detail-enhanced Segment Anything Model 2 for High-Resolution Class-agnostic Segmentation [6.976534642198541]
  We propose MGD-SAM2, which integrates SAM2 with multi-view feature interaction between a global image and local patches to achieve precise segmentation.<n>We first introduce MPAdapter to adapt the SAM2 encoder for enhanced extraction of local details and global semantics in HRCS images.<n>Then, MCEM and HMIM are proposed to further exploit local texture and global context by aggregating multi-view features within and across multi-scales.
  arXiv  Detail & Related papers  (2025-03-31T07:02:32Z)
• Segment Anything with Multiple Modalities [61.74214237816402]
  We develop MM-SAM, which supports cross-modal and multi-modal processing for robust and enhanced segmentation with different sensor suites. MM-SAM features two key designs, namely, unsupervised cross-modal transfer and weakly-supervised multi-modal fusion. It addresses three main challenges: 1) adaptation toward diverse non-RGB sensors for single-modal processing, 2) synergistic processing of multi-modal data via sensor fusion, and 3) mask-free training for different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-17T03:45:40Z)
• Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection [58.241593208031816]
  Segment Anything Model (SAM) has been proposed as a visual fundamental model, which gives strong segmentation and generalization capabilities. We propose a Multi-scale and Detail-enhanced SAM (MDSAM) for Salient Object Detection (SOD) Experimental results demonstrate the superior performance of our model on multiple SOD datasets.
  arXiv  Detail & Related papers  (2024-08-08T09:09:37Z)
• Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM [62.85895749882285]
  Marine Animal (MAS) involves segmenting animals within marine environments. We propose a novel feature learning framework, named Dual-SAM for high-performance MAS. Our proposed method achieves state-of-the-art performances on five widely-used MAS datasets.
  arXiv  Detail & Related papers  (2024-04-07T15:34:40Z)
• WSI-SAM: Multi-resolution Segment Anything Model (SAM) for histopathology whole-slide images [8.179859593451285]
  We present WSI-SAM, enhancing Segment Anything Model (SAM) with precise object segmentation capabilities for histopathology images. To fully exploit pretrained knowledge while minimizing training overhead, we keep SAM frozen, introducing only minimal extra parameters. Our model outperforms SAM by 4.1 and 2.5 percent points on a ductal carcinoma in situ (DCIS) segmentation tasks and breast cancer metastasis segmentation task.
  arXiv  Detail & Related papers  (2024-03-14T10:30:43Z)
• A SAM-guided Two-stream Lightweight Model for Anomaly Detection [44.73985145110819]
  We propose a SAM-guided Two-stream Lightweight Model for unsupervised anomaly detection (STLM) Our experiments conducted on MVTec AD benchmark show that STLM, with about 16M parameters and achieving an inference time in 20ms, competes effectively with state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-29T13:29:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.