SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost

• URL: http://arxiv.org/abs/2506.01304v1
• Date: Mon, 02 Jun 2025 04:30:14 GMT
• Title: SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost
• Abstract summary: We introduce SAM-I2V, an effective image-tovideo upgradation method for cultivating a promptable video segmentation (PVS) model.<n>Our approach strategically upgrades the pre-trained SAM to support PVS, significantly reducing training complexity and resource requirements.
• Score: 23.51101492249404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models like the Segment Anything Model (SAM) have significantly advanced promptable image segmentation in computer vision. However, extending these capabilities to videos presents substantial challenges, particularly in ensuring precise and temporally consistent mask propagation in dynamic scenes. SAM 2 attempts to address this by training a model on massive image and video data from scratch to learn complex spatiotemporal associations, resulting in huge training costs that hinder research and practical deployment. In this paper, we introduce SAM-I2V, an effective image-to-video upgradation method for cultivating a promptable video segmentation (PVS) model. Our approach strategically upgrades the pre-trained SAM to support PVS, significantly reducing training complexity and resource requirements. To achieve this, we introduce three key innovations: (i) an image-to-video feature extraction upgrader built upon SAM's static image encoder to enable spatiotemporal video perception, (ii) a memory filtering strategy that selects the most relevant past frames for more effective utilization of historical information, and (iii) a memory-as-prompt mechanism leveraging object memory to ensure temporally consistent mask propagation in dynamic scenes. Comprehensive experiments demonstrate that our method achieves over 90% of SAM 2's performance while using only 0.2% of its training cost. Our work presents a resource-efficient pathway to PVS, lowering barriers for further research in PVS model design and enabling broader applications and advancements in the field. Code and model are available at: https://github.com/showlab/SAM-I2V.

Related papers

• MobileSAM2: Lightweight Segment Anything for Spatial Intelligence [104.24442074291396]
  Large video foundation model, SAM2, enables segment anything in both images and videos.<n>We aim to make SAM2 more mobile-friendly by distilling the heavyweight SAM2 into a lightweight model.<n>We present MobileSAM2, a new family of lightweight SAM2 that balances efficiency and effectiveness.
  arXiv  Detail & Related papers  (2026-07-14T03:10:11Z)
• TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model [9.471131843884953]
  TinySAM 2 is a lightweight video segmentation model that balances performance and efficiency.<n>Experiments on challenging datasets such as DAVIS and SA-V demonstrate that TinySAM 2 achieves 90% of the performance of SAM 2.1.
  arXiv  Detail & Related papers  (2026-05-18T08:05:59Z)
• EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3 [1.0243211938789691]
  We present EfficientSAM3, a family of efficient models built on Progressive Hierarchical Distillation (PHD)<n>PHD aligns image features via prompt-in-the-loop training on SA-1B; (2) Temporal Memory Distillation replaces dense memory with a compact Perceiver-based module trained on SA-V to retrieve compress and retrieve features efficiently.<n>We benchmark on popular VOS datasets, and compare with varies of releated work, achieing strong performance-efficiency trade-offs.
  arXiv  Detail & Related papers  (2025-11-19T19:42:22Z)
• TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object Segmentation and Tracking in Surgical Scenarios [1.0596160761674702]
  We propose TSMS-SAM2, a novel framework that enhances promptable VOST in surgical videos by addressing challenges of rapid object motion and memory redundancy.<n>TSMS-SAM2 introduces two key strategies: multi-temporal-scale video sampling augmentation to improve robustness against motion variability, and a memory splitting and pruning mechanism that organizes and filters past frame features.
  arXiv  Detail & Related papers  (2025-08-07T20:11:15Z)
• Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation [18.71772979219666]
  We introduce Memory Augmented (MA)-SAM2, a training-free video object segmentation strategy.<n>MA-SAM2 exhibits strong robustness against occlusions and interactions arising from complex instrument movements.<n>Without introducing any additional parameters or requiring further training, MA-SAM2 achieved performance improvements of 4.36% and 6.1% over SAM2 on the EndoVis 2017 and EndoVis 2018 datasets.
  arXiv  Detail & Related papers  (2025-07-13T11:05:25Z)
• SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2 [2.659882635924329]
  Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks.<n>Recent methods augment SAM 2 with hand-crafted update rules to better handle distractors and object motion.<n>We propose a fundamentally different approach using reinforcement learning for optimizing memory updates in SAM 2.
  arXiv  Detail & Related papers  (2025-07-11T12:53:19Z)
• SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes [30.870903750545004]
  We introduce a novel framework, termed SAM2-LOVE, which integrates textual, audio, and visual representations into a learnable token.<n>Technically, our approach includes a multimodal fusion module aimed at improving multimodal understanding of SAM2.<n>We conducted extensive experiments to demonstrate that SAM2-LOVE outperforms the SOTA by 8.5% in $calmathJ&F$ on the Ref-AVS benchmark.
  arXiv  Detail & Related papers  (2025-06-02T11:36:25Z)
• Rethinking Few-Shot Medical Image Segmentation by SAM2: A Training-Free Framework with Augmentative Prompting and Dynamic Matching [4.1253497486581026]
  We conceptualize 3D medical image volumes as video sequences, departing from the traditional slice-by-slice paradigm.<n>We perform extensive data augmentation on a single labeled support image and, for each frame in the query volume, algorithmically select the most analogous augmented support image.<n>We demonstrate state-of-the-art performance on benchmark few-shot medical image segmentation datasets, achieving significant improvements in accuracy and annotation efficiency.
  arXiv  Detail & Related papers  (2025-03-05T06:12:13Z)
• Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos [110.3379755761583]
  Sa2VA is a unified model for grounded understanding of both images and videos.<n>It supports a wide range of image and video tasks, including referring segmentation and conversation.<n>We show that Sa2VA achieves state-of-the-art across multiple tasks, particularly in referring video object segmentation.
  arXiv  Detail & Related papers  (2025-01-07T18:58:54Z)
• From SAM to SAM 2: Exploring Improvements in Meta's Segment Anything Model [0.5639904484784127]
  The Segment Anything Model (SAM) was introduced to the computer vision community by Meta in April 2023. SAM excels in zero-shot performance, segmenting unseen objects without additional training, stimulated by a large dataset of over one billion image masks. SAM 2 expands this functionality to video, leveraging memory from preceding and subsequent frames to generate accurate segmentation across entire videos.
  arXiv  Detail & Related papers  (2024-08-12T17:17:35Z)
• Segment Anything for Videos: A Systematic Survey [52.28931543292431]
  The recent wave of foundation models has witnessed tremendous success in computer vision (CV) and beyond. The segment anything model (SAM) has sparked a passion for exploring task-agnostic visual foundation models. This work conducts a systematic review on SAM for videos in the era of foundation models.
  arXiv  Detail & Related papers  (2024-07-31T02:24:53Z)
• Generalizable Visual Reinforcement Learning with Segment Anything Model [28.172477166023697]
  We introduce Segment Anything Model for Generalizable visual RL (SAM-G) SAM-G is a novel framework that leverages the promptable segmentation ability of Segment Anything Model (SAM) to enhance the generalization capabilities of visual RL agents. evaluated across 8 DMControl tasks and 3 Adroit tasks, SAM-G significantly improves the visual generalization ability without altering the RL agents' architecture but merely their observations.
  arXiv  Detail & Related papers  (2023-12-28T16:53:23Z)
• EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything [36.553867358541154]
  Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. We propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning.
  arXiv  Detail & Related papers  (2023-12-01T18:31:00Z)
• Selective Volume Mixup for Video Action Recognition [89.67716672312358]
  We propose a novel video augmentation strategy named Selective Volume Mixup (SV-Mix) to improve the generalization ability of deep models with limited training videos. SV-Mix devises a learnable selective module to choose the most informative volumes from two videos and mixes the volumes up to achieve a new training video. We empirically demonstrate the merits of the SV-Mix augmentation on a wide range of video action recognition benchmarks.
  arXiv  Detail & Related papers  (2023-09-18T07:26:58Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring [82.84513669453744]
  Image-text pretrained models, e.g., CLIP, have shown impressive general multi-modal knowledge learned from large-scale image-text data pairs. We revisit temporal modeling in the context of image-to-video knowledge transferring. We present a simple and effective temporal modeling mechanism extending CLIP model to diverse video tasks.
  arXiv  Detail & Related papers  (2023-01-26T14:12:02Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.