EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss

• URL: http://arxiv.org/abs/2402.05008v2
• Date: Thu, 16 May 2024 20:51:52 GMT
• Title: EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss
• Authors: Zhuoyang Zhang, Han Cai, Song Han,
• Abstract summary: We present EfficientViT-SAM, a new family of accelerated segment anything models. For the training, we begin with the knowledge distillation from the SAM-ViT-H image encoder to EfficientViT. Benefiting from EfficientViT's efficiency and capacity, EfficientViT-SAM delivers 48.9x measuredRT speedup on A100 GPU over SAM-ViT-H.
• Score: 23.428671076019207
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present EfficientViT-SAM, a new family of accelerated segment anything models. We retain SAM's lightweight prompt encoder and mask decoder while replacing the heavy image encoder with EfficientViT. For the training, we begin with the knowledge distillation from the SAM-ViT-H image encoder to EfficientViT. Subsequently, we conduct end-to-end training on the SA-1B dataset. Benefiting from EfficientViT's efficiency and capacity, EfficientViT-SAM delivers 48.9x measured TensorRT speedup on A100 GPU over SAM-ViT-H without sacrificing performance. Our code and pre-trained models are released at https://github.com/mit-han-lab/efficientvit.

Related papers

• TinySAM: Pushing the Envelope for Efficient Segment Anything Model [76.21007576954035]
  We propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance. We first propose a full-stage knowledge distillation method with hard prompt sampling and hard mask weighting strategy to distill a lightweight student model. We also adapt the post-training quantization to the promptable segmentation task and further reduce the computational cost.
  arXiv  Detail & Related papers  (2023-12-21T12:26:11Z)
• EdgeSAM: Prompt-In-the-Loop Distillation for On-Device Deployment of SAM [71.868623296582]
  EdgeSAM is an accelerated variant of the Segment Anything Model (SAM) Our approach involves distilling the original ViT-based SAM image encoder into a purely CNN-based architecture. It is the first SAM variant that can run at over 30 FPS on an iPhone 14.
  arXiv  Detail & Related papers  (2023-12-11T18:59:52Z)
• RepViT-SAM: Towards Real-Time Segmenting Anything [71.94042743317937]
  Segment Anything Model (SAM) has shown impressive zero-shot transfer performance for various computer vision tasks. MobileSAM proposes to replace the heavyweight image encoder in SAM with TinyViT by employing distillation. RepViT-SAM can enjoy significantly better zero-shot transfer capability than MobileSAM, along with nearly $10times$ faster inference speed.
  arXiv  Detail & Related papers  (2023-12-10T04:42:56Z)
• EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything [36.553867358541154]
  Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. We propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning.
  arXiv  Detail & Related papers  (2023-12-01T18:31:00Z)
• Faster Segment Anything: Towards Lightweight SAM for Mobile Applications [47.177751899636164]
  This work aims to make Segment Anything Model (SAM) mobile-friendly by replacing the heavyweight image encoder with a lightweight one. We distill the knowledge from the heavy image encoder to a lightweight image encoder, which can be automatically compatible with the mask decoder in the original SAM. The resulting lightweight SAM is termed MobileSAM which is more than 60 times smaller yet performs on par with the original SAM.
  arXiv  Detail & Related papers  (2023-06-25T16:37:25Z)
• Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach [132.37966970098645]
  One of the popular solutions is Sharpness-Aware Minimization (SAM), which minimizes the change of weight loss when adding a perturbation. In this paper, we propose an efficient effective training scheme coined as Sparse SAM (SSAM), which achieves double overhead of common perturbations. In addition, we theoretically prove that S can converge at the same SAM, i.e., $O(log T/sqrtTTTTTTTTTTTTTTTTT
  arXiv  Detail & Related papers  (2022-10-11T06:30:10Z)
• Towards Efficient and Scalable Sharpness-Aware Minimization [81.22779501753695]
  We propose a novel algorithm LookSAM that only periodically calculates the inner gradient ascent. LookSAM achieves similar accuracy gains to SAM while being tremendously faster. We are the first to successfully scale up the batch size when training Vision Transformers (ViTs)
  arXiv  Detail & Related papers  (2022-03-05T11:53:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.