Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking

• URL: http://arxiv.org/abs/2505.18111v1
• Date: Fri, 23 May 2025 17:04:56 GMT
• Title: Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking
• Abstract summary: We present an effective approach for adapting the Segment Anything Model 2 (SAM2) to the Visual Object Tracking (VOT) task.<n>Our method leverages the powerful pre-trained capabilities of SAM2 and incorporates several key techniques to enhance its performance in VOT applications.
• Score: 18.75180282940254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an effective approach for adapting the Segment Anything Model 2 (SAM2) to the Visual Object Tracking (VOT) task. Our method leverages the powerful pre-trained capabilities of SAM2 and incorporates several key techniques to enhance its performance in VOT applications. By combining SAM2 with our proposed optimizations, we achieved a first place AUC score of 89.4 on the 2024 ICPR Multi-modal Object Tracking challenge, demonstrating the effectiveness of our approach. This paper details our methodology, the specific enhancements made to SAM2, and a comprehensive analysis of our results in the context of VOT solutions along with the multi-modality aspect of the dataset.

Related papers

• Advancing Complex Video Object Segmentation via Tracking-Enhanced Prompt: The 1st Winner for 5th PVUW MOSE Challenge [65.24213788883016]
  In the Complex Video Object task, researchers are required to track and segment specific targets within cluttered environments.<n>The root cause of this limitation lies in SAM3's insufficient comprehension of these specific target types.<n>We propose TEP: Advancing Complex Video Object via Tracking-Enhanced Prompts.
  arXiv  Detail & Related papers  (2026-04-01T02:23:23Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 [48.71856814549096]
  We propose UAP-SAM2, the first cross-prompt universal adversarial attack against SAM2 driven by dual semantic deviation.<n>We show that UAP-SAM2 significantly outperforms state-of-the-art (SOTA) attacks by a large margin.
  arXiv  Detail & Related papers  (2025-10-28T08:59:11Z)
• Distractor-Aware Memory-Based Visual Object Tracking [17.945503249662675]
  We propose a distractor-aware drop-in memory module and introspection-based management method for SAM2.<n>Our design effectively reduces the tracking drift toward distractors and improves redetection capability after object occlusion.<n>We show DAM4SAM outperforms SAM2.1 on thirteen benchmarks and sets new state-of-the-art results on ten.
  arXiv  Detail & Related papers  (2025-09-17T09:54:27Z)
• SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation [11.1906749425206]
  Segment Anything 2 (SAM2) enables robust single-object tracking using segmentation.<n>We propose SAM2MOT, a novel Tracking by paradigm for multi-object tracking.<n> SAM2MOT directly generates tracking boxes from segmentation masks, reducing reliance on detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T15:32:08Z)
• Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond [52.486290612938895]
  We propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability.<n> Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM.<n>Our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency.
  arXiv  Detail & Related papers  (2025-03-03T06:16:31Z)
• Action Recognition Using Temporal Shift Module and Ensemble Learning [0.0]
  The paper presents the first-rank solution for the Multi-Modal Action Recognition Challenge, part of the Multi-Modal Visual Pattern Recognition Workshop at the aclICPR 2024.<n>The competition aimed to recognize human actions using a diverse dataset of 20 action classes, collected from multi-modal sources.<n>Our solution achieved a perfect top-1 accuracy on the test set, demonstrating the effectiveness of the proposed approach in recognizing human actions across 20 classes.
  arXiv  Detail & Related papers  (2025-01-29T10:36:55Z)
• Continual Learning for Segment Anything Model Adaptation [14.00191851894315]
  We propose a novel Continual SAM adaptation (CoSAM) benchmark with 8 different task domains.<n>We then propose a novel simple-yet-effective Mixture of Domain Adapters (MoDA) algorithm to help the SAM encoder extract well-separated features for different task domains.<n>Our MoDA maintains highly competitive results in the natural image domain, approaching the zero-shot performance of the original SAM.
  arXiv  Detail & Related papers  (2024-12-09T11:51:28Z)
• Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts [17.6980007370549]
  We make the first attempt to adapt Segment Anything Model (SAM) for multi-modal semantic segmentation.<n>By training only the MoE-LoRA layers while keeping SAM's weights frozen, SAM's strong generalization and segmentation capabilities can be preserved for downstream tasks.<n>Specifically, to address cross-modal inconsistencies, we propose a novel MoE routing strategy that adaptively generates weighted features across modalities.
  arXiv  Detail & Related papers  (2024-12-05T14:54:31Z)
• On Efficient Variants of Segment Anything Model: A Survey [63.127753705046]
  The Segment Anything Model (SAM) is a foundational model for image segmentation tasks, known for its strong generalization across diverse applications. To address this, a variety of SAM variants have been proposed to enhance efficiency while keeping accuracy. This survey provides the first comprehensive review of these efficient SAM variants.
  arXiv  Detail & Related papers  (2024-10-07T11:59:54Z)
• Underwater Camouflaged Object Tracking Meets Vision-Language SAM2 [60.47622353256502]
  We propose the first large-scale multi-modal underwater camouflaged object tracking dataset, namely UW-COT220.<n>Based on the proposed dataset, this work first evaluates current advanced visual object tracking methods, including SAM- and SAM2-based trackers, in challenging underwater environments.<n>Our findings highlight the improvements of SAM2 over SAM, demonstrating its enhanced ability to handle the complexities of underwater camouflaged objects.
  arXiv  Detail & Related papers  (2024-09-25T13:10:03Z)
• Segment and Track Anything [57.20918630166862]
  This report presents a framework called Segment And Track Anything (SAMTrack) SAM-Track allows users to precisely and effectively segment and track any object in a video. It can be used across an array of fields, ranging from drone technology, autonomous driving, medical imaging, augmented reality, to biological analysis.
  arXiv  Detail & Related papers  (2023-05-11T04:33:08Z)
• Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration [56.01625477187448]
  We propose a MultiModality PAnoramic multi-object Tracking framework (MMPAT) It takes both 2D panorama images and 3D point clouds as input and then infers target trajectories using the multimodality data. We evaluate the proposed method on the JRDB dataset, where the MMPAT achieves the top performance in both the detection and tracking tasks.
  arXiv  Detail & Related papers  (2021-05-31T03:16:38Z)
• Multi-Task Learning for Dense Prediction Tasks: A Survey [87.66280582034838]
  Multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint. We provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision.
  arXiv  Detail & Related papers  (2020-04-28T09:15:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.