SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory

• URL: http://arxiv.org/abs/2411.11922v1
• Date: Mon, 18 Nov 2024 05:59:03 GMT
• Title: SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory
• Authors: Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, Jenq-Neng Hwang,
• Abstract summary: This paper introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for visual object tracking. By incorporating temporal motion cues with the proposed motion-aware memory selection mechanism, SAMURAI effectively predicts object motion and refines mask selection, achieving robust, accurate tracking without the need for retraining or fine-tuning. In evaluations, SAMURAI achieves significant improvements in success rate and precision over existing trackers, with a 7.1% AUC gain on LaSOT$_ext$ and a 3.5% AO gain on GOT-10k.
• Score: 23.547018300192065
• License:
• Abstract: The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks but faces challenges in visual object tracking, particularly when managing crowded scenes with fast-moving or self-occluding objects. Furthermore, the fixed-window memory approach in the original model does not consider the quality of memories selected to condition the image features for the next frame, leading to error propagation in videos. This paper introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for visual object tracking. By incorporating temporal motion cues with the proposed motion-aware memory selection mechanism, SAMURAI effectively predicts object motion and refines mask selection, achieving robust, accurate tracking without the need for retraining or fine-tuning. SAMURAI operates in real-time and demonstrates strong zero-shot performance across diverse benchmark datasets, showcasing its ability to generalize without fine-tuning. In evaluations, SAMURAI achieves significant improvements in success rate and precision over existing trackers, with a 7.1% AUC gain on LaSOT$_{\text{ext}}$ and a 3.5% AO gain on GOT-10k. Moreover, it achieves competitive results compared to fully supervised methods on LaSOT, underscoring its robustness in complex tracking scenarios and its potential for real-world applications in dynamic environments. Code and results are available at https://github.com/yangchris11/samurai.

Related papers

• A Distractor-Aware Memory for Visual Object Tracking with SAM2 [11.864619292028278]
  Memory-based trackers are video object segmentation methods that form the target model by concatenating recently tracked frames into a memory buffer and localize the target by attending the current image to the buffered frames. SAM2.1++ outperforms SAM2.1 and related SAM memory extensions on seven benchmarks and sets a solid new state-of-the-art on six of them.
  arXiv  Detail & Related papers  (2024-11-26T16:41:09Z)
• From SAM to SAM 2: Exploring Improvements in Meta's Segment Anything Model [0.5639904484784127]
  The Segment Anything Model (SAM) was introduced to the computer vision community by Meta in April 2023. SAM excels in zero-shot performance, segmenting unseen objects without additional training, stimulated by a large dataset of over one billion image masks. SAM 2 expands this functionality to video, leveraging memory from preceding and subsequent frames to generate accurate segmentation across entire videos.
  arXiv  Detail & Related papers  (2024-08-12T17:17:35Z)
• SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation [13.609341065893739]
  This study explores the zero-shot segmentation performance of SAM 2 in robot-assisted surgery based on prompts. We employ two forms of prompts: 1-point and bounding box, while for video sequences, the 1-point prompt is applied to the initial frame. The results with point prompts also exhibit a substantial enhancement over SAM's capabilities, nearing or even surpassing existing unprompted SOTA methods.
  arXiv  Detail & Related papers  (2024-08-08T17:08:57Z)
• RobustSAM: Segment Anything Robustly on Degraded Images [19.767828436963317]
  Segment Anything Model (SAM) has emerged as a transformative approach in image segmentation. We propose the Robust Segment Anything Model (RobustSAM), which enhances SAM's performance on low-quality images. Our method has been shown to effectively improve the performance of SAM-based downstream tasks such as single image dehazing and deblurring.
  arXiv  Detail & Related papers  (2024-06-13T23:33:59Z)
• MAS-SAM: Segment Any Marine Animal with Aggregated Features [55.91291540810978]
  We propose a novel feature learning framework named MAS-SAM for marine animal segmentation. Our method enables to extract richer marine information from global contextual cues to fine-grained local details.
  arXiv  Detail & Related papers  (2024-04-24T07:38:14Z)
• TinySAM: Pushing the Envelope for Efficient Segment Anything Model [76.21007576954035]
  We propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance. We first propose a full-stage knowledge distillation method with hard prompt sampling and hard mask weighting strategy to distill a lightweight student model. We also adapt the post-training quantization to the promptable segmentation task and further reduce the computational cost.
  arXiv  Detail & Related papers  (2023-12-21T12:26:11Z)
• EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything [36.553867358541154]
  Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. We propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning.
  arXiv  Detail & Related papers  (2023-12-01T18:31:00Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• Personalize Segment Anything Model with One Shot [52.54453744941516]
  We propose a training-free Personalization approach for Segment Anything Model (SAM) Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior. PerSAM segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement.
  arXiv  Detail & Related papers  (2023-05-04T17:59:36Z)
• An Effective Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds [50.19288542498838]
  3D single object tracking in LiDAR point clouds (LiDAR SOT) plays a crucial role in autonomous driving. Current approaches all follow the Siamese paradigm based on appearance matching. We introduce a motion-centric paradigm to handle LiDAR SOT from a new perspective.
  arXiv  Detail & Related papers  (2023-03-21T17:28:44Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.