EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3

• URL: http://arxiv.org/abs/2511.15833v1
• Date: Wed, 19 Nov 2025 19:42:22 GMT
• Title: EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3
• Abstract summary: We present EfficientSAM3, a family of efficient models built on Progressive Hierarchical Distillation (PHD)<n>PHD aligns image features via prompt-in-the-loop training on SA-1B; (2) Temporal Memory Distillation replaces dense memory with a compact Perceiver-based module trained on SA-V to retrieve compress and retrieve features efficiently.<n>We benchmark on popular VOS datasets, and compare with varies of releated work, achieing strong performance-efficiency trade-offs.
• Score: 1.0243211938789691
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Segment Anything Model 3 (SAM3) advances visual understanding with Promptable Concept Segmentation (PCS) across images and videos, but its unified architecture (shared vision backbone, DETR-style detector, dense-memory tracker) remains prohibitive for on-device use. We present EfficientSAM3, a family of efficient models built on Progressive Hierarchical Distillation (PHD) that transfers capability from SAM3 to lightweight students in three stages: (1) Encoder Distillation aligns image features via prompt-in-the-loop training on SA-1B; (2) Temporal Memory Distillation replaces dense memory with a compact Perceiver-based module trained on SA-V to compress and retrieve spatiotemporal features efficiently; and (3) End-to-End Fine-Tuning refines the full pipeline on the official SAM3 PCS data to preserve concept-level performance. PHD yields a spectrum of student variants using RepViT, TinyViT, and EfficientViT backbones, enabling on-device concept segmentation and tracking while maintaining high fidelity to teacher behavior. We benchmark on popular VOS datasets, and compare with varies of releated work, achieing strong performance-efficiency trade-offs.

Related papers

• TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model [9.471131843884953]
  TinySAM 2 is a lightweight video segmentation model that balances performance and efficiency.<n>Experiments on challenging datasets such as DAVIS and SA-V demonstrate that TinySAM 2 achieves 90% of the performance of SAM 2.1.
  arXiv  Detail & Related papers  (2026-05-18T08:05:59Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• SAM 3: Segment Anything with Concepts [93.97262932669081]
  We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts.<n>PCS takes such prompts and returns segmentation masks and identities for all matching object instances.<n>Our model consists of an image-level detector and a memory-based video tracker that share a single backbone.
  arXiv  Detail & Related papers  (2025-11-20T18:59:56Z)
• Enriched Feature Representation and Motion Prediction Module for MOSEv2 Track of 7th LSVOS Challenge: 3rd Place Solution [8.540105031750434]
  We propose a framework that integrates the strengths of Cutie and SAM2.<n>We achieve 3rd place in the MOSEv2 track of the 7th LSVOS Challenge.<n>This demonstrates the effectiveness of enriched feature representation and motion prediction for robust video object segmentation.
  arXiv  Detail & Related papers  (2025-09-19T09:11:01Z)
• SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2 [2.659882635924329]
  Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks.<n>Recent methods augment SAM 2 with hand-crafted update rules to better handle distractors and object motion.<n>We propose a fundamentally different approach using reinforcement learning for optimizing memory updates in SAM 2.
  arXiv  Detail & Related papers  (2025-07-11T12:53:19Z)
• METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding [55.38256656122857]
  We propose METok, a training-free, Multi-stage Event-based Token compression framework.<n>We show METok achieves an optimal trade-off between efficiency and accuracy by dynamically selecting informative visual tokens.<n>For instance, equipping LongVA-7B with METok realizes an 80.6% FLOPs reduction and 93.5% KV Cache memory savings.
  arXiv  Detail & Related papers  (2025-06-03T13:19:41Z)
• SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost [23.51101492249404]
  We introduce SAM-I2V, an effective image-tovideo upgradation method for cultivating a promptable video segmentation (PVS) model.<n>Our approach strategically upgrades the pre-trained SAM to support PVS, significantly reducing training complexity and resource requirements.
  arXiv  Detail & Related papers  (2025-06-02T04:30:14Z)
• One RL to See Them All: Visual Triple Unified Reinforcement Learning [92.90120580989839]
  We propose V-Triune, a Visual Triple Unified Reinforcement Learning system that enables visual reasoning and perception tasks within a single training pipeline.<n>V-Triune comprises triple complementary components: Sample-Level Datashelf (to unify diverse task inputs), Verifier-Level Reward (to deliver custom rewards via specialized verifiers).<n>We introduce a novel Dynamic IoU reward, which provides adaptive, progressive, and definite feedback for perception tasks handled by V-Triune.
  arXiv  Detail & Related papers  (2025-05-23T17:41:14Z)
• EdgeTAM: On-Device Track Anything Model [65.10032957471824]
  Segment Anything Model (SAM) 2 further extends its capability from image to video inputs through a memory bank mechanism.<n>We aim at making SAM 2 much more efficient so that it even runs on mobile devices while maintaining a comparable performance.<n>We propose EdgeTAM, which leverages a novel 2D Spatial Perceiver to reduce the computational cost.
  arXiv  Detail & Related papers  (2025-01-13T12:11:07Z)
• FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images [7.2993352400518035]
  We present FastSAM3D, which accelerates SAM inference to 8 milliseconds per 128*128*128 3D volumetric image on an NVIDIA A100 GPU. FastSAM3D achieves a remarkable speedup of 527.38x compared to 2D SAMs and 8.75x compared to 3D SAMs on the same volumes without significant performance decline.
  arXiv  Detail & Related papers  (2024-03-14T19:29:44Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.