Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation for Zero-shot Generalization

• URL: http://arxiv.org/abs/2509.11772v1
• Date: Mon, 15 Sep 2025 10:52:27 GMT
• Title: Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation for Zero-shot Generalization
• Authors: Diogo Mendonça, Tiago Barros, Cristiano Premebida, Urbano J. Nunes,
• Abstract summary: Seg2Track-SAM2 is a framework that integrates pre-trained object detectors with SAM2 and a novel Seg2Track module.<n>Seg2Track-SAM2 achieves state-of-the-art (SOTA) performance, ranking fourth overall in both car and pedestrian classes on KITTI MOTS.<n>Results confirm that Seg2Track-SAM2 advances MOTS by combining robust zero-shot tracking, enhanced identity preservation, and efficient memory utilization.
• Score: 3.108551551357326
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous systems require robust Multi-Object Tracking (MOT) capabilities to operate reliably in dynamic environments. MOT ensures consistent object identity assignment and precise spatial delineation. Recent advances in foundation models, such as SAM2, have demonstrated strong zero-shot generalization for video segmentation, but their direct application to MOTS (MOT+Segmentation) remains limited by insufficient identity management and memory efficiency. This work introduces Seg2Track-SAM2, a framework that integrates pre-trained object detectors with SAM2 and a novel Seg2Track module to address track initialization, track management, and reinforcement. The proposed approach requires no fine-tuning and remains detector-agnostic. Experimental results on KITTI MOT and KITTI MOTS benchmarks show that Seg2Track-SAM2 achieves state-of-the-art (SOTA) performance, ranking fourth overall in both car and pedestrian classes on KITTI MOTS, while establishing a new benchmark in association accuracy (AssA). Furthermore, a sliding-window memory strategy reduces memory usage by up to 75% with negligible performance degradation, supporting deployment under resource constraints. These results confirm that Seg2Track-SAM2 advances MOTS by combining robust zero-shot tracking, enhanced identity preservation, and efficient memory utilization. The code is available at https://github.com/hcmr-lab/Seg2Track-SAM2

Related papers

• Distractor-Aware Memory-Based Visual Object Tracking [17.945503249662675]
  We propose a distractor-aware drop-in memory module and introspection-based management method for SAM2.<n>Our design effectively reduces the tracking drift toward distractors and improves redetection capability after object occlusion.<n>We show DAM4SAM outperforms SAM2.1 on thirteen benchmarks and sets new state-of-the-art results on ten.
  arXiv  Detail & Related papers  (2025-09-17T09:54:27Z)
• TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object Segmentation and Tracking in Surgical Scenarios [1.0596160761674702]
  We propose TSMS-SAM2, a novel framework that enhances promptable VOST in surgical videos by addressing challenges of rapid object motion and memory redundancy.<n>TSMS-SAM2 introduces two key strategies: multi-temporal-scale video sampling augmentation to improve robustness against motion variability, and a memory splitting and pruning mechanism that organizes and filters past frame features.
  arXiv  Detail & Related papers  (2025-08-07T20:11:15Z)
• SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks [50.97089872043121]
  We propose SAM2-UNeXT, an advanced framework that builds upon the core principles of SAM2-UNet.<n>We extend the representational capacity of SAM2 through the integration of an auxiliary DINOv2 encoder.<n>Our approach enables more accurate segmentation with a simple architecture, relaxing the need for complex decoder designs.
  arXiv  Detail & Related papers  (2025-08-05T15:36:13Z)
• SAMITE: Position Prompted SAM2 with Calibrated Memory for Visual Object Tracking [58.35852822355312]
  Visual Object Tracking (VOT) is widely used in applications like autonomous driving to continuously track targets in videos.<n>To address these issues, some methods propose to adapt the video foundation model SAM2 for VOT, where the tracking results of each frame would be encoded as memory for conditioning the rest of frames in an autoregressive manner.<n>We present a SAMITE model, built upon SAM2 with additional modules, to tackle these challenges.
  arXiv  Detail & Related papers  (2025-07-29T12:11:56Z)
• SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation [11.1906749425206]
  Segment Anything 2 (SAM2) enables robust single-object tracking using segmentation.<n>We propose SAM2MOT, a novel Tracking by paradigm for multi-object tracking.<n> SAM2MOT directly generates tracking boxes from segmentation masks, reducing reliance on detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T15:32:08Z)
• A Distractor-Aware Memory for Visual Object Tracking with SAM2 [11.864619292028278]
  Memory-based trackers are video object segmentation methods that form the target model by concatenating recently tracked frames into a memory buffer and localize the target by attending the current image to the buffered frames.<n> SAM2.1++ outperforms SAM2.1 and related SAM memory extensions on seven benchmarks and sets a solid new state-of-the-art on six of them.
  arXiv  Detail & Related papers  (2024-11-26T16:41:09Z)
• TinySAM: Pushing the Envelope for Efficient Segment Anything Model [73.06322749886483]
  We propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance.<n>With all these proposed methods, our TinySAM leads to orders of magnitude computational reduction and pushes the envelope for efficient segment anything task.
  arXiv  Detail & Related papers  (2023-12-21T12:26:11Z)
• Segment Anything Meets Point Tracking [116.44931239508578]
  This paper presents a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions.
  arXiv  Detail & Related papers  (2023-07-03T17:58:01Z)
• SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking [20.286114226299237]
  This paper introduces SMILEtrack, an innovative object tracker with a Siamese network-based Similarity Learning Module (SLM) The SLM calculates the appearance similarity between two objects, overcoming the limitations of feature descriptors in Separate Detection and Embedding models. Second, we develop a Similarity Matching Cascade (SMC) module with a novel GATE function for robust object matching across consecutive video frames.
  arXiv  Detail & Related papers  (2022-11-16T10:49:48Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)
• Distractor-Aware Fast Tracking via Dynamic Convolutions and MOT Philosophy [63.91005999481061]
  A practical long-term tracker typically contains three key properties, i.e. an efficient model design, an effective global re-detection strategy and a robust distractor awareness mechanism. We propose a two-task tracking frame work (named DMTrack) to achieve distractor-aware fast tracking via Dynamic convolutions (d-convs) and Multiple object tracking (MOT) philosophy. Our tracker achieves state-of-the-art performance on the LaSOT, OxUvA, TLP, VOT2018LT and VOT 2019LT benchmarks and runs in real-time (3x faster
  arXiv  Detail & Related papers  (2021-04-25T00:59:53Z)
• Online Multiple Object Tracking with Cross-Task Synergy [120.70085565030628]
  We propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model.
  arXiv  Detail & Related papers  (2021-04-01T10:19:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.