Track-On: Transformer-based Online Point Tracking with Memory

• URL: http://arxiv.org/abs/2501.18487v1
• Date: Thu, 30 Jan 2025 17:04:11 GMT
• Title: Track-On: Transformer-based Online Point Tracking with Memory
• Authors: Görkay Aydemir, Xiongyi Cai, Weidi Xie, Fatma Güney,
• Abstract summary: We introduce Track-On, a simple transformer-based model designed for online long-term point tracking. Unlike prior methods that depend on full temporal modeling, our model processes video frames causally without access to future frames. At inference time, it employs patch classification and refinement to identify correspondences and track points with high accuracy.
• Score: 34.744546679670734
• License:
• Abstract: In this paper, we consider the problem of long-term point tracking, which requires consistent identification of points across multiple frames in a video, despite changes in appearance, lighting, perspective, and occlusions. We target online tracking on a frame-by-frame basis, making it suitable for real-world, streaming scenarios. Specifically, we introduce Track-On, a simple transformer-based model designed for online long-term point tracking. Unlike prior methods that depend on full temporal modeling, our model processes video frames causally without access to future frames, leveraging two memory modules -- spatial memory and context memory -- to capture temporal information and maintain reliable point tracking over long time horizons. At inference time, it employs patch classification and refinement to identify correspondences and track points with high accuracy. Through extensive experiments, we demonstrate that Track-On sets a new state-of-the-art for online models and delivers superior or competitive results compared to offline approaches on seven datasets, including the TAP-Vid benchmark. Our method offers a robust and scalable solution for real-time tracking in diverse applications. Project page: https://kuis-ai.github.io/track_on

Related papers

• Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking [53.33637391723555]
  We propose a unified multimodal spatial-temporal tracking approach named STTrack. In contrast to previous paradigms, we introduced a temporal state generator (TSG) that continuously generates a sequence of tokens containing multimodal temporal information. These temporal information tokens are used to guide the localization of the target in the next time state, establish long-range contextual relationships between video frames, and capture the temporal trajectory of the target.
  arXiv  Detail & Related papers  (2024-12-20T09:10:17Z)
• ODTrack: Online Dense Temporal Token Learning for Visual Tracking [22.628561792412686]
  ODTrack is a video-level tracking pipeline that densely associates contextual relationships of video frames in an online token propagation manner. It achieves a new itSOTA performance on seven benchmarks, while running at real-time speed.
  arXiv  Detail & Related papers  (2024-01-03T11:44:09Z)
• PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking [90.29143475328506]
  We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. We animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos.
  arXiv  Detail & Related papers  (2023-07-27T17:58:11Z)
• STTracker: Spatio-Temporal Tracker for 3D Single Object Tracking [11.901758708579642]
  3D single object tracking with point clouds is a critical task in 3D computer vision. Previous methods usually input the last two frames and use the template point cloud in previous frame and the search area point cloud in the current frame respectively.
  arXiv  Detail & Related papers  (2023-06-30T07:25:11Z)
• TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement [64.11385310305612]
  We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS.
  arXiv  Detail & Related papers  (2023-06-14T17:07:51Z)
• Tracking by Associating Clips [110.08925274049409]
  In this paper, we investigate an alternative by treating object association as clip-wise matching. Our new perspective views a single long video sequence as multiple short clips, and then the tracking is performed both within and between the clips. The benefits of this new approach are two folds. First, our method is robust to tracking error accumulation or propagation, as the video chunking allows bypassing the interrupted frames. Second, the multiple frame information is aggregated during the clip-wise matching, resulting in a more accurate long-range track association than the current frame-wise matching.
  arXiv  Detail & Related papers  (2022-12-20T10:33:17Z)
• Context-aware Visual Tracking with Joint Meta-updating [11.226947525556813]
  We propose a context-aware tracking model to optimize the tracker over the representation space, which jointly meta-update both branches by exploiting information along the whole sequence. The proposed tracking method achieves an EAO score of 0.514 on VOT2018 with the speed of 40FPS, demonstrating its capability of improving the accuracy and robustness of the underlying tracker with little speed drop.
  arXiv  Detail & Related papers  (2022-04-04T14:16:00Z)
• Learning to Track with Object Permanence [61.36492084090744]
  We introduce an end-to-end trainable approach for joint object detection and tracking. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI, and MOT17 datasets.
  arXiv  Detail & Related papers  (2021-03-26T04:43:04Z)
• TrackFormer: Multi-Object Tracking with Transformers [92.25832593088421]
  TrackFormer is an end-to-end multi-object tracking and segmentation model based on an encoder-decoder Transformer architecture. New track queries are spawned by the DETR object detector and embed the position of their corresponding object over time. TrackFormer achieves a seamless data association between frames in a new tracking-by-attention paradigm.
  arXiv  Detail & Related papers  (2021-01-07T18:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.