TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos

• URL: http://arxiv.org/abs/2403.17346v1
• Date: Tue, 26 Mar 2024 03:10:45 GMT
• Title: TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos
• Authors: Yufu Wang, Ziyun Wang, Lingjie Liu, Kostas Daniilidis,
• Abstract summary: TRAM robustifies SLAM to recover the camera motion in the presence of dynamic humans. We introduce a video transformer model to regress the kinematic body motion of a human.
• Score: 46.11545135199594
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose TRAM, a two-stage method to reconstruct a human's global trajectory and motion from in-the-wild videos. TRAM robustifies SLAM to recover the camera motion in the presence of dynamic humans and uses the scene background to derive the motion scale. Using the recovered camera as a metric-scale reference frame, we introduce a video transformer model (VIMO) to regress the kinematic body motion of a human. By composing the two motions, we achieve accurate recovery of 3D humans in the world space, reducing global motion errors by 60% from prior work. https://yufu-wang.github.io/tram4d/

Related papers

• World-Grounded Human Motion Recovery via Gravity-View Coordinates [60.618543026949226]
  We propose estimating human poses in a novel Gravity-View coordinate system. The proposed GV system is naturally gravity-aligned and uniquely defined for each video frame. Our method recovers more realistic motion in both the camera space and world-grounded settings, outperforming state-of-the-art methods in both accuracy and speed.
  arXiv  Detail & Related papers  (2024-09-10T17:25:47Z)
• WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion [43.95997922499137]
  WHAM (World-grounded Humans with Accurate Motion) reconstructs 3D human motion in a global coordinate system from video. Uses camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks.
  arXiv  Detail & Related papers  (2023-12-12T18:57:46Z)
• PACE: Human and Camera Motion Estimation from in-the-wild Videos [113.76041632912577]
  We present a method to estimate human motion in a global scene from moving cameras. This is a highly challenging task due to the coupling of human and camera motions in the video. We propose a joint optimization framework that disentangles human and camera motions using both foreground human motion priors and background scene features.
  arXiv  Detail & Related papers  (2023-10-20T19:04:14Z)
• Humans in 4D: Reconstructing and Tracking Humans with Transformers [72.50856500760352]
  We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images.
  arXiv  Detail & Related papers  (2023-05-31T17:59:52Z)
• Decoupling Human and Camera Motion from Videos in the Wild [67.39432972193929]
  We propose a method to reconstruct global human trajectories from videos in the wild. Our method decouples the camera and human motion, which allows us to place people in the same world coordinate frame.
  arXiv  Detail & Related papers  (2023-02-24T18:59:15Z)
• MotionBERT: A Unified Perspective on Learning Human Motion Representations [46.67364057245364]
  We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. We propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. We implement motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network.
  arXiv  Detail & Related papers  (2022-10-12T19:46:25Z)
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
  We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
  arXiv  Detail & Related papers  (2021-12-02T18:59:54Z)
• 3D Human Motion Estimation via Motion Compression and Refinement [27.49664453166726]
  We develop a technique for generating smooth and accurate 3D human pose and motion estimates from RGB video sequences. Our method, which we call Motion Estimation via Variational Autoencoder (MEVA), decomposes a temporal sequence of human motion into a smooth motion representation.
  arXiv  Detail & Related papers  (2020-08-09T19:02:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.