TAPVid-3D: A Benchmark for Tracking Any Point in 3D

• URL: http://arxiv.org/abs/2407.05921v2
• Date: Tue, 27 Aug 2024 17:14:16 GMT
• Title: TAPVid-3D: A Benchmark for Tracking Any Point in 3D
• Authors: Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, João Carreira, Andrew Zisserman, Gabriel Brostow, Carl Doersch,
• Abstract summary: We introduce a new benchmark, TAPVid-3D, for evaluating the task of Tracking Any Point in 3D. This benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video.
• Score: 63.060421798990845
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io

Related papers

• PillarTrack: Redesigning Pillar-based Transformer Network for Single Object Tracking on Point Clouds [5.524413892353708]
  LiDAR-based 3D single object tracking (3D SOT) is a critical issue in robotics and autonomous driving. We propose PillarTrack, a pillar-based 3D single object tracking framework. PillarTrack achieves state-of-the-art performance on the KITTI and nuScenes dataset and enables real-time tracking speed.
  arXiv  Detail & Related papers  (2024-04-11T06:06:56Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• Tracking by 3D Model Estimation of Unknown Objects in Videos [122.56499878291916]
  We argue that this representation is limited and instead propose to guide and improve 2D tracking with an explicit object representation. Our representation tackles a complex long-term dense correspondence problem between all 3D points on the object for all video frames. The proposed optimization minimizes a novel loss function to estimate the best 3D shape, texture, and 6DoF pose.
  arXiv  Detail & Related papers  (2023-04-13T11:32:36Z)
• TAP-Vid: A Benchmark for Tracking Any Point in a Video [84.94877216665793]
  We formalize the problem of tracking arbitrary physical points on surfaces over longer video clips, naming it tracking any point (TAP) We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. We propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.
  arXiv  Detail & Related papers  (2022-11-07T17:57:02Z)
• Time3D: End-to-End Joint Monocular 3D Object Detection and Tracking for Autonomous Driving [3.8073142980733]
  We propose jointly training 3D detection and 3D tracking from only monocular videos in an end-to-end manner. Time3D achieves 21.4% AMOTA, 13.6% AMOTP on the nuScenes 3D tracking benchmark, surpassing all published competitors.
  arXiv  Detail & Related papers  (2022-05-30T06:41:10Z)
• 3D Visual Tracking Framework with Deep Learning for Asteroid Exploration [22.808962211830675]
  In this paper, we focus on the studied accurate and real-time method for 3D tracking. A new large-scale 3D asteroid tracking dataset is presented, including binocular video sequences, depth maps, and point clouds of diverse asteroids. We propose a deep-learning based 3D tracking framework, named as Track3D, which involves 2D monocular tracker and a novel light-weight amodal axis-aligned bounding-box network, A3BoxNet.
  arXiv  Detail & Related papers  (2021-11-21T04:14:45Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)
• Tracking Emerges by Looking Around Static Scenes, with Neural 3D Mapping [23.456046776979903]
  We propose to leverage multiview data of textitstatic points in arbitrary scenes (static or dynamic) to learn a neural 3D mapping module. The neural 3D mapper consumes RGB-D data as input, and produces a 3D voxel grid of deep features as output. We show that our unsupervised 3D object trackers outperform prior unsupervised 2D and 2.5D trackers, and approach the accuracy of supervised trackers.
  arXiv  Detail & Related papers  (2020-08-04T02:59:23Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.