Seeing the World and the Self from Egocentric Video

• URL: http://arxiv.org/abs/2609.01276v1
• Date: Tue, 01 Sep 2026 14:10:13 GMT
• Title: Seeing the World and the Self from Egocentric Video
• Abstract summary: Existing methods address scene reconstruction and motion estimation separately.<n>We propose RESELF, a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation.<n> Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks.
• Score: 9.841166134702467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Existing methods typically address scene reconstruction and motion estimation separately: scene reconstruction methods ignore the wearer, whereas motion estimation methods lack explicit scene geometry and often depend on external trajectories. Joint recovery is challenging because the two tasks exhibit asymmetric visibility and require different prediction paradigms. The largely visible scene supports deterministic geometric regression, whereas the severely occluded body requires generative motion inference. We therefore propose RESELF (REconstructing the Scene and the sELF), a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation. RESELF adapts a geometry foundation model pre-trained on large-scale exocentric data to egocentric video using frame-wise scale and relative-pose consistency objectives. The resulting camera trajectory and latent geometric features condition a diffusion model that recovers the wearer's motion. A subsequent closed-loop kinematic feedback stage further refines the camera head while preserving the reconstructed scene geometry. To support training and evaluation, we curate EE4D-JSM from EgoExo4D by aligning egocentric video, sparse metric scene geometry, camera trajectories, and full-body motion annotations. Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation. Code, models, and datasets will be available at https://ka1guan.github.io/RESELF/.

Related papers

• TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Image [53.206752160427065]
  TourPhysics is an online framework from a single image and a declarative physical configuration.<n>For each action, the simulator computes a finite physical and camera trajectory before the corresponding observation is generated.<n>TourPhysics follows prescribed camera and object trajectories more closely than the evaluated baselines.
  arXiv  Detail & Related papers  (2026-09-04T09:15:03Z)
• TRIG: Trajectory-Rig Decoupled Metric Geometry Learning [33.69899077340302]
  Trajectory-Rig Decoupled Metric Geometry Learning (TRIG) is a geometry perception framework for autonomous driving.<n>We introduce decoupled pose encoding and supervision, which separately constrain trajectory evolution and rig geometry for metric-consistent learning.<n>Experiments on five autonomous driving benchmarks show that TRIG achieves state-of-the-art performance in pose estimation, metric depth prediction, and 3D reconstruction.
  arXiv  Detail & Related papers  (2026-07-07T03:55:13Z)
• GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation [46.507099021313074]
  We introduce a geometry-consistency reward that measures whether motion in a generated video is compatible with a coherent scene.<n>We operationalize this using optical flow, depth--pose predictions, and feature-based correspondence to separate rigid and dynamic regions.<n>Experiments show substantial reductions in temporal geometric artifacts over strong baselines while preserving perceptual quality.
  arXiv  Detail & Related papers  (2026-05-18T13:17:08Z)
• Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction [17.903916530175533]
  Egocentric video provides a unique view into human perception and interaction.<n>Rapid camera motion and complex scene dynamics pose major challenges for 3D reconstruction from this perspective.<n>We evaluate dynamic monocular 3DGS models on egocentric and exocentric video using paired ego-exo recordings.
  arXiv  Detail & Related papers  (2026-04-26T16:58:19Z)
• LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models [52.656349227001925]
  Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory.<n>Existing methods face two distinct challenges.<n>We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model.
  arXiv  Detail & Related papers  (2026-01-21T05:46:03Z)
• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives [65.89192712575797]
  We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video.<n>Our approach reduces motion tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks.<n>This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale.
  arXiv  Detail & Related papers  (2025-12-16T18:59:50Z)
• GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering [54.489285024494855]
  Video stabilization is pivotal for video processing, as it removes unwanted shakiness while preserving the original user motion intent.<n>Existing approaches, depending on the domain they operate, suffer from several issues that degrade the user experience.<n>We introduce textbfGaVS, a novel 3D-grounded approach that reformulates video stabilization as a temporally-consistent local reconstruction and rendering' paradigm.
  arXiv  Detail & Related papers  (2025-06-30T15:24:27Z)
• Self-Supervised Monocular 4D Scene Reconstruction for Egocentric Videos [25.41337525728398]
  EgoMono4D is a novel model that unifies the estimation of multiple variables necessary for Egocentric Monocular 4D reconstruction.<n>It achieves superior performance in dense pointclouds sequence reconstruction compared to all baselines.
  arXiv  Detail & Related papers  (2024-11-14T02:57:11Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation [76.58256020932312]
  Estimating the motion of the camera together with the 3D structure of the scene from a monocular vision system is a complex task. We present a self-supervised learning framework for 3D object motion field estimation from monocular videos.
  arXiv  Detail & Related papers  (2021-10-13T16:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.