EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity

• URL: http://arxiv.org/abs/2502.18373v1
• Date: Tue, 25 Feb 2025 17:11:14 GMT
• Title: EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity
• Abstract summary: We introduce EgoSim, a novel simulator of body-worn cameras that generates realistic egocentric renderings from multiple perspectives across a wearer's body.<n>A key feature of EgoSim is its use of real motion capture data to render motion artifacts, which are especially noticeable with arm- or leg-worn cameras.<n>We augment with 5 hours of real-world motion data from 13 participants using six GoPro cameras and 3D body pose references from an Xsens motion capture suit.
• Score: 23.361792553311055
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Research on egocentric tasks in computer vision has mostly focused on head-mounted cameras, such as fisheye cameras or embedded cameras inside immersive headsets. We argue that the increasing miniaturization of optical sensors will lead to the prolific integration of cameras into many more body-worn devices at various locations. This will bring fresh perspectives to established tasks in computer vision and benefit key areas such as human motion tracking, body pose estimation, or action recognition -- particularly for the lower body, which is typically occluded. In this paper, we introduce EgoSim, a novel simulator of body-worn cameras that generates realistic egocentric renderings from multiple perspectives across a wearer's body. A key feature of EgoSim is its use of real motion capture data to render motion artifacts, which are especially noticeable with arm- or leg-worn cameras. In addition, we introduce MultiEgoView, a dataset of egocentric footage from six body-worn cameras and ground-truth full-body 3D poses during several activities: 119 hours of data are derived from AMASS motion sequences in four high-fidelity virtual environments, which we augment with 5 hours of real-world motion data from 13 participants using six GoPro cameras and 3D body pose references from an Xsens motion capture suit. We demonstrate EgoSim's effectiveness by training an end-to-end video-only 3D pose estimation network. Analyzing its domain gap, we show that our dataset and simulator substantially aid training for inference on real-world data. EgoSim code & MultiEgoView dataset: https://siplab.org/projects/EgoSim

Related papers

• EgoSim: Egocentric World Simulator for Embodied Interaction Generation [93.11209644808783]
  EgoSim is a closed-loop egocentric world simulator that generates spatially consistent interaction videos.<n>It persistently updates the underlying 3D scene state for continuous simulation.<n>EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization.
  arXiv  Detail & Related papers  (2026-04-01T15:00:46Z)
• Ego-1K -- A Large-Scale Multiview Video Dataset for Egocentric Vision [7.318705822469006]
  We present Egohugging-1K, a large-scale collection of time-synchronized egocentric multiview videos.<n>The dataset contains nearly 1,000 short egocentric videos captured with a custom rig with 12 synchronized cameras surrounding a 4-camera VR headset worn by the user.
  arXiv  Detail & Related papers  (2026-03-14T04:00:26Z)
• FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video [52.33896173943054]
  Egocentric motion capture with a head-mounted body-facing stereo camera is crucial for VR and AR applications. Existing methods rely on synthetic pretraining and struggle to generate smooth and accurate predictions in real-world settings. We propose FRAME, a simple yet effective architecture that combines device pose and camera feeds for state-of-the-art body pose prediction.
  arXiv  Detail & Related papers  (2025-03-29T14:26:06Z)
• HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos [9.513100627302755]
  The dataset offers over 833 minutes (3.7M+ images) of recordings that feature 19 subjects interacting with 33 diverse rigid objects. The recordings include multiple synchronized data streams containing egocentric multi-view RGB/monochrome images, eye gaze signal, scene point clouds, and 3D poses of cameras, hands, and objects. In our experiments, we demonstrate the effectiveness of multi-view egocentric data for three popular tasks: 3D hand tracking, model-based 6DoF object pose estimation, and 3D lifting of unknown in-hand objects.
  arXiv  Detail & Related papers  (2024-11-28T14:09:42Z)
• EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars [56.56236652774294]
  We propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods.
  arXiv  Detail & Related papers  (2024-09-22T22:50:27Z)
• EgoGen: An Egocentric Synthetic Data Generator [53.32942235801499]
  EgoGen is a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. We demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views.
  arXiv  Detail & Related papers  (2024-01-16T18:55:22Z)
• EgoHumans: An Egocentric 3D Multi-Human Benchmark [37.375846688453514]
  We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking. We propose a novel 3D capture setup to construct a comprehensive egocentric multi-human benchmark in the wild. We leverage consumer-grade wearable camera-equipped glasses for the egocentric view, which enables us to capture dynamic activities like playing tennis, fencing, volleyball, etc.
  arXiv  Detail & Related papers  (2023-05-25T21:37:36Z)
• Ego-Body Pose Estimation via Ego-Head Pose Estimation [22.08240141115053]
  Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. We propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion.
  arXiv  Detail & Related papers  (2022-12-09T02:25:20Z)
• UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture [70.59984501516084]
  UnrealEgo is a new large-scale naturalistic dataset for egocentric 3D human pose estimation. It is based on an advanced concept of eyeglasses equipped with two fisheye cameras that can be used in unconstrained environments. We propose a new benchmark method with a simple but effective idea of devising a 2D keypoint estimation module for stereo inputs to improve 3D human pose estimation.
  arXiv  Detail & Related papers  (2022-08-02T17:59:54Z)
• EgoBody: Human Body Shape, Motion and Social Interactions from Head-Mounted Devices [76.50816193153098]
  EgoBody is a novel large-scale dataset for social interactions in complex 3D scenes. We employ Microsoft HoloLens2 headsets to record rich egocentric data streams including RGB, depth, eye gaze, head and hand tracking. To obtain accurate 3D ground-truth, we calibrate the headset with a multi-Kinect rig and fit expressive SMPL-X body meshes to multi-view RGB-D frames.
  arXiv  Detail & Related papers  (2021-12-14T18:41:28Z)
• Egocentric Pose Estimation from Human Vision Span [7.786789357900368]
  Estimating camera wearer's body pose from an egocentric view (egopose) is a vital task in augmented and virtual reality. Existing approaches either use a narrow field of view front facing camera that barely captures the wearer, or an extruded head-mounted top-down camera for maximal wearer visibility. We propose a novel deep learning system taking advantage of both the dynamic features from camera SLAM and the body shape imagery.
  arXiv  Detail & Related papers  (2021-04-12T02:38:22Z)
• Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors [71.29186299435423]
  We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment. We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift. HPS could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera.
  arXiv  Detail & Related papers  (2021-03-31T17:58:31Z)
• UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality [12.349749717823736]
  In this paper, we propose a new data-driven framework for inside-out body tracking. We first collect a large-scale motion capture dataset with both body and finger motions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts.
  arXiv  Detail & Related papers  (2020-11-12T09:31:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.