Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking

• URL: http://arxiv.org/abs/2410.01806v1
• Date: Wed, 2 Oct 2024 17:59:57 GMT
• Title: Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking
• Authors: Mattia Segu, Luigi Piccinelli, Siyuan Li, Yung-Hsu Yang, Bernt Schiele, Luc Van Gool,
• Abstract summary: Multiple object tracking in complex scenarios, such as coordinated dance performances, team sports, or dynamic animal groups, presents unique challenges. We introduce Samba, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations. We introduce an effective technique for dealing with uncertain observations (MaskObs) and an efficient training recipe to scale SambaMOTR to longer sequences.
• Score: 97.25156823720211
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multiple object tracking in complex scenarios - such as coordinated dance performances, team sports, or dynamic animal groups - presents unique challenges. In these settings, objects frequently move in coordinated patterns, occlude each other, and exhibit long-term dependencies in their trajectories. However, it remains a key open research question on how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. To this end, we introduce Samba, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations across tracklets. By integrating Samba into a tracking-by-propagation framework, we propose SambaMOTR, the first tracker effectively addressing the aforementioned issues, including long-range dependencies, tracklet interdependencies, and temporal occlusions. Additionally, we introduce an effective technique for dealing with uncertain observations (MaskObs) and an efficient training recipe to scale SambaMOTR to longer sequences. By modeling long-range dependencies and interactions among tracked objects, SambaMOTR implicitly learns to track objects accurately through occlusions without any hand-crafted heuristics. Our approach significantly surpasses prior state-of-the-art on the DanceTrack, BFT, and SportsMOT datasets.

Related papers

• SMTrack: State-Aware Mamba for Efficient Temporal Modeling in Visual Tracking [39.1131712751769]
  We propose a novel temporal modeling paradigm for visual tracking, termed State-aware Mamba Tracker (SMTrack)<n>SMTrack provides a neat pipeline for training and tracking without needing customized modules or substantial computational costs to build long-range temporal dependencies.<n>Extensive experimental results demonstrate that SMTrack achieves promising performance with low computational costs.
  arXiv  Detail & Related papers  (2026-02-02T05:44:59Z)
• CAMELTrack: Context-Aware Multi-cue ExpLoitation for Online Multi-Object Tracking [68.24998698508344]
  We introduce CAMEL, a novel association module for Context-Aware Multi-Cue ExpLoitation.<n>Unlike end-to-end detection-by-tracking approaches, our method remains lightweight and fast to train while being able to leverage external off-the-shelf models.<n>Our proposed online tracking pipeline, CAMELTrack, achieves state-of-the-art performance on multiple tracking benchmarks.
  arXiv  Detail & Related papers  (2025-05-02T13:26:23Z)
• Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking [53.33637391723555]
  We propose a unified multimodal spatial-temporal tracking approach named STTrack. In contrast to previous paradigms, we introduced a temporal state generator (TSG) that continuously generates a sequence of tokens containing multimodal temporal information. These temporal information tokens are used to guide the localization of the target in the next time state, establish long-range contextual relationships between video frames, and capture the temporal trajectory of the target.
  arXiv  Detail & Related papers  (2024-12-20T09:10:17Z)
• Deciphering Movement: Unified Trajectory Generation Model for Multi-Agent [53.637837706712794]
  We propose a Unified Trajectory Generation model, UniTraj, that processes arbitrary trajectories as masked inputs. Specifically, we introduce a Ghost Spatial Masking (GSM) module embedded within a Transformer encoder for spatial feature extraction. We benchmark three practical sports game datasets, Basketball-U, Football-U, and Soccer-U, for evaluation.
  arXiv  Detail & Related papers  (2024-05-27T22:15:23Z)
• MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking [56.92165669843006]
  We propose MotionTrack, which learns robust short-term and long-term motions in a unified framework to associate trajectories from a short to long range. For dense crowds, we design a novel Interaction Module to learn interaction-aware motions from short-term trajectories, which can estimate the complex movement of each target. For extreme occlusions, we build a novel Refind Module to learn reliable long-term motions from the target's history trajectory, which can link the interrupted trajectory with its corresponding detection.
  arXiv  Detail & Related papers  (2023-03-18T12:38:33Z)
• Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking [30.357116118917368]
  We propose an end-to-end multi-camera 3D multi-object tracking framework. We name it "Past-and-Future reasoning for Tracking" (PFTrack)
  arXiv  Detail & Related papers  (2023-02-07T23:46:34Z)
• DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion [56.1428110894411]
  We propose a large-scale dataset for multi-human tracking, where humans have similar appearance, diverse motion and extreme articulation. As the dataset contains mostly group dancing videos, we name it "DanceTrack" We benchmark several state-of-the-art trackers on our dataset and observe a significant performance drop on DanceTrack when compared against existing benchmarks.
  arXiv  Detail & Related papers  (2021-11-29T16:49:06Z)
• Two is a crowd: tracking relations in videos [2.1485350418225244]
  We propose a plug-in Relation Module (REM) to extend current state-of-the-art trackers. REM encodes relations between tracked objects by running a message passing over a corresponding graph-temporal embeddings for the tracked objects. REM allows tracking severely or even fully occluded objects by utilizing relational cues.
  arXiv  Detail & Related papers  (2021-08-11T17:19:34Z)
• MOTR: End-to-End Multiple-Object Tracking with TRansformer [31.78906135775541]
  We present MOTR, the first fully end-to-end multiple object tracking framework. It learns to model the long-range temporal variation of the objects. Results show that MOTR achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-05-07T13:27:01Z)
• Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers [20.806348407522083]
  MO3TR is an end-to-end online multi-object tracking framework. It encodes object interactions into long-term temporal embeddings. It tracks initiation and termination without the need for an explicit data association module.
  arXiv  Detail & Related papers  (2021-03-27T07:23:38Z)
• Learning to Track with Object Permanence [61.36492084090744]
  We introduce an end-to-end trainable approach for joint object detection and tracking. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI, and MOT17 datasets.
  arXiv  Detail & Related papers  (2021-03-26T04:43:04Z)
• Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking [20.66906781151]
  In multi-object tracking, the tracker maintains in its memory the appearance and motion information for each object in the scene. Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory. We propose a training strategy adapted to multi-track pooling which generates hard tracking episodes online.
  arXiv  Detail & Related papers  (2021-01-28T18:12:39Z)
• SoDA: Multi-Object Tracking with Soft Data Association [75.39833486073597]
  Multi-object tracking (MOT) is a prerequisite for a safe deployment of self-driving cars. We propose a novel approach to MOT that uses attention to compute track embeddings that encode dependencies between observed objects.
  arXiv  Detail & Related papers  (2020-08-18T03:40:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.