Exploring Temporally-Aware Features for Point Tracking

• URL: http://arxiv.org/abs/2501.12218v1
• Date: Tue, 21 Jan 2025 15:39:40 GMT
• Title: Exploring Temporally-Aware Features for Point Tracking
• Authors: Inès Hyeonsu Kim, Seokju Cho, Jiahui Huang, Jung Yi, Joon-Young Lee, Seungryong Kim,
• Abstract summary: Chrono is a feature backbone specifically designed for point tracking with built-in temporal awareness. Chrono achieves state-of-the-art performance in a refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets.
• Score: 58.63091479730935
• License:
• Abstract: Point tracking in videos is a fundamental task with applications in robotics, video editing, and more. While many vision tasks benefit from pre-trained feature backbones to improve generalizability, point tracking has primarily relied on simpler backbones trained from scratch on synthetic data, which may limit robustness in real-world scenarios. Additionally, point tracking requires temporal awareness to ensure coherence across frames, but using temporally-aware features is still underexplored. Most current methods often employ a two-stage process: an initial coarse prediction followed by a refinement stage to inject temporal information and correct errors from the coarse stage. These approach, however, is computationally expensive and potentially redundant if the feature backbone itself captures sufficient temporal information. In this work, we introduce Chrono, a feature backbone specifically designed for point tracking with built-in temporal awareness. Leveraging pre-trained representations from self-supervised learner DINOv2 and enhanced with a temporal adapter, Chrono effectively captures long-term temporal context, enabling precise prediction even without the refinement stage. Experimental results demonstrate that Chrono achieves state-of-the-art performance in a refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets, among common feature backbones used in point tracking as well as DINOv2, with exceptional efficiency. Project page: https://cvlab-kaist.github.io/Chrono/

Related papers

• TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video [30.16638127979361]
  We present TAPTRv3, which is built upon TAPTRv2, to improve its point tracking robustness in long videos. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the challenging datasets and obtains state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-11-27T17:37:22Z)
• ACTrack: Adding Spatio-Temporal Condition for Visual Object Tracking [0.5371337604556311]
  Efficiently modeling-temporal relations of objects is a key challenge in visual object tracking (VOT) Existing methods track by appearance-based similarity or long-term relation modeling, resulting in rich temporal contexts between consecutive frames being easily overlooked. In this paper we present ACTrack, a new framework with additive pre-temporal tracking framework with large memory conditions. It preserves the quality and capabilities of the pre-trained backbone by freezing its parameters, and makes a trainable lightweight additive net to model temporal relations in tracking. We design an additive siamese convolutional network to ensure the integrity of spatial features and temporal sequence
  arXiv  Detail & Related papers  (2024-02-27T07:34:08Z)
• LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry [52.131996528655094]
  We present the Long-term Effective Any Point Tracking (LEAP) module. LEAP innovatively combines visual, inter-track, and temporal cues with mindfully selected anchors for dynamic track estimation. Based on these traits, we develop LEAP-VO, a robust visual odometry system adept at handling occlusions and dynamic scenes.
  arXiv  Detail & Related papers  (2024-01-03T18:57:27Z)
• Dense Optical Tracking: Connecting the Dots [82.79642869586587]
  DOT is a novel, simple and efficient method for solving the problem of point tracking in a video. We show that DOT is significantly more accurate than current optical flow techniques, outperforms sophisticated "universal trackers" like OmniMotion, and is on par with, or better than, the best point tracking algorithms like CoTracker.
  arXiv  Detail & Related papers  (2023-12-01T18:59:59Z)
• Towards Real-World Visual Tracking with Temporal Contexts [64.7981374129495]
  We propose a two-level framework (TCTrack) that can exploit temporal contexts efficiently. Based on it, we propose a stronger version for real-world visual tracking, i.e., TCTrack++. For feature extraction, we propose an attention-based temporally adaptive convolution to enhance the spatial features. For similarity map refinement, we introduce an adaptive temporal transformer to encode the temporal knowledge efficiently.
  arXiv  Detail & Related papers  (2023-08-20T17:59:40Z)
• 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation [0.0]
  3D-FCT is a Siamese network architecture that utilizes temporal information to simultaneously perform the related tasks of 3D object detection and tracking. Our proposed method is evaluated on the KITTI tracking dataset where it is shown to provide an improvement of 5.57% mAP over a state-of-the-art approach.
  arXiv  Detail & Related papers  (2021-10-06T06:36:29Z)
• Interpretable Deep Feature Propagation for Early Action Recognition [39.966828592322315]
  In this study, we address action prediction by investigating how action patterns evolve over time in a spatial feature space. We work with intermediate-layer ConvNet features, which allow for abstraction from raw data, while retaining spatial layout. We employ a Kalman filter to combat error build-up and unify across prediction start times.
  arXiv  Detail & Related papers  (2021-07-11T19:40:19Z)
• Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos [78.45050529204701]
  We propose a novel framework to pursue discriminative and robust representation by modeling cross-scale spatial-temporal correlation. CTL utilizes a CNN backbone and a key-points estimator to extract semantic local features from human body. It explores a context-reinforced topology to construct multi-scale graphs by considering both global contextual information and physical connections of human body.
  arXiv  Detail & Related papers  (2021-04-15T14:32:12Z)
• A Prospective Study on Sequence-Driven Temporal Sampling and Ego-Motion Compensation for Action Recognition in the EPIC-Kitchens Dataset [68.8204255655161]
  Action recognition is one of the top-challenging research fields in computer vision. ego-motion recorded sequences have become of important relevance. The proposed method aims to cope with it by estimating this ego-motion or camera motion.
  arXiv  Detail & Related papers  (2020-08-26T14:44:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.