Tracking Everything Everywhere All at Once

• URL: http://arxiv.org/abs/2306.05422v2
• Date: Tue, 12 Sep 2023 16:32:52 GMT
• Title: Tracking Everything Everywhere All at Once
• Authors: Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely
• Abstract summary: We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. We propose a complete and globally consistent motion representation, dubbed OmniMotion. Our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.
• Score: 111.00807055441028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively. See our project page for more results: http://omnimotion.github.io/

Related papers

• Decomposition Betters Tracking Everything Everywhere [8.199205242808592]
  We propose a new test-time optimization method, named DecoMotion, for estimating per-pixel and long-range motion. Our method boosts the point-tracking accuracy by a large margin and performs on par with some state-of-the-art dedicated point-tracking solutions.
  arXiv  Detail & Related papers  (2024-07-09T04:01:23Z)
• Dense Optical Tracking: Connecting the Dots [82.79642869586587]
  DOT is a novel, simple and efficient method for solving the problem of point tracking in a video. We show that DOT is significantly more accurate than current optical flow techniques, outperforms sophisticated "universal trackers" like OmniMotion, and is on par with, or better than, the best point tracking algorithms like CoTracker.
  arXiv  Detail & Related papers  (2023-12-01T18:59:59Z)
• PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking [90.29143475328506]
  We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. We animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos.
  arXiv  Detail & Related papers  (2023-07-27T17:58:11Z)
• ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild [57.37891682117178]
  We present a robust dense indirect structure-from-motion method for videos that is based on dense correspondence from pairwise optical flow. A novel neural network architecture is proposed for processing irregular point trajectory data. Experiments on MPI Sintel dataset show that our system produces significantly more accurate camera trajectories.
  arXiv  Detail & Related papers  (2022-07-19T09:19:45Z)
• Particle Videos Revisited: Tracking Through Occlusions Using Point Trajectories [29.258861811749103]
  We revisit Sand and Teller's "particle video" approach, and study pixel tracking as a long-range motion estimation problem. We re-build this classic approach using components that drive the current state-of-the-art in flow and object tracking. We train our models using long-range amodal point trajectories mined from existing optical flow datasets.
  arXiv  Detail & Related papers  (2022-04-08T16:05:48Z)
• Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videos [115.71874459429381]
  We propose a method for jointly estimating the 3D motion, 3D shape, and appearance of highly motion-blurred objects from a video. Experiments on benchmark datasets demonstrate that our method outperforms previous methods for fast moving object deblurring and 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-29T11:25:14Z)
• Consistent Depth of Moving Objects in Video [52.72092264848864]
  We present a method to estimate depth of a dynamic scene, containing arbitrary moving objects, from an ordinary video captured with a moving camera. We formulate this objective in a new test-time training framework where a depth-prediction CNN is trained in tandem with an auxiliary scene-flow prediction over the entire input video. We demonstrate accurate and temporally coherent results on a variety of challenging videos containing diverse moving objects (pets, people, cars) as well as camera motion.
  arXiv  Detail & Related papers  (2021-08-02T20:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.