Everybody Tracking Every Body

• URL: http://arxiv.org/abs/2608.29927v1
• Date: Sun, 30 Aug 2026 17:42:15 GMT
• Title: Everybody Tracking Every Body
• Abstract summary: We address the problem of 3D body pose estimation of multiple people interacting from their egocentric views with centralized coordination.<n>Processing this video with VIO SLAM provides high-quality tracking of each egocentric camera through space.<n>We propose a diffusion-based approach that fuses estimates of pose based on head motion with exocentric pose observations.
• Score: 54.76361724681694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of 3D body pose estimation of multiple interacting people from their egocentric views with centralized coordination. Each individual wears a camera recording egocentric video and IMU data. Processing this video with VIO SLAM provides high-quality tracking of each egocentric camera through space. The first-person view from one individual provides third-person observations of other people, although these exocentric observations are sparse, intermittent, and of highly variable reliability as both cameras and subjects move. To integrate these synchronized data streams, we propose a diffusion-based approach that fuses estimates of pose based on head motion derived from egocentric camera motion with exocentric pose observations, conditioning on both observation content and reliability. Our model is trained on a mixture of single-person motion-capture data and multi-person video in order to learn rich priors for body motion trajectories and video observation reliability. Evaluation on challenging multi-person datasets suggests our fusion approach improves over motion-only and vision-only baselines in terms of both absolute and relative pose accuracy.

Related papers

• Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context [17.735273173582716]
  We present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues.<n>We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction.
  arXiv  Detail & Related papers  (2025-12-22T11:26:41Z)
• Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data [16.431101717478796]
  Current methods for ego-body pose estimation rely on temporally dense sensor data. We develop a two-stage approach that decomposes the problem into temporal completion and spatial completion.
  arXiv  Detail & Related papers  (2024-11-05T23:53:19Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• WHAC: World-grounded Humans and Cameras [37.877565981937586]
  We aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly. We introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation. We present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras.
  arXiv  Detail & Related papers  (2024-03-19T17:58:02Z)
• Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement [65.08165593201437]
  We explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion. This task presents significant challenges due to the lack of high-quality datasets, fisheye camera distortion, and human body self-occlusion. We propose a novel approach that leverages FisheyeViT to extract fisheye image features, which are converted into pixel-aligned 3D heatmap representations for 3D human body pose prediction.
  arXiv  Detail & Related papers  (2023-11-28T07:13:47Z)
• Ego-Body Pose Estimation via Ego-Head Pose Estimation [22.08240141115053]
  Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. We propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion.
  arXiv  Detail & Related papers  (2022-12-09T02:25:20Z)
• Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision [72.36132924512299]
  We present a new egocentric pose estimation method, which can be trained on a large-scale in-the-wild egocentric dataset. We propose a novel learning strategy to supervise the egocentric features with the high-quality features extracted by a pretrained external-view pose estimation model. Experiments show that our method predicts accurate 3D poses from a single in-the-wild egocentric image and outperforms the state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2022-01-20T00:45:13Z)
• Enhancing Egocentric 3D Pose Estimation with Third Person Views [37.9683439632693]
  We propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. We introduce First2Third-Pose, a new paired synchronized dataset of nearly 2,000 videos depicting human activities captured from both first- and third-view perspectives. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos.
  arXiv  Detail & Related papers  (2022-01-06T11:42:01Z)
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
  We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
  arXiv  Detail & Related papers  (2021-12-02T18:59:54Z)
• Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation [76.58256020932312]
  Estimating the motion of the camera together with the 3D structure of the scene from a monocular vision system is a complex task. We present a self-supervised learning framework for 3D object motion field estimation from monocular videos.
  arXiv  Detail & Related papers  (2021-10-13T16:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.