Mirror-Aware Neural Humans
- URL: http://arxiv.org/abs/2309.04750v2
- Date: Wed, 15 May 2024 21:51:46 GMT
- Title: Mirror-Aware Neural Humans
- Authors: Daniel Ajisafe, James Tang, Shih-Yang Su, Bastian Wandt, Helge Rhodin,
- Abstract summary: We develop a consumer-level 3D motion capture system that starts from off-the-shelf 2D poses by automatically calibrating the camera.
We empirically demonstrate the benefit of learning a body model and accounting for occlusion in challenging mirror scenes.
- Score: 21.0548144424571
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Human motion capture either requires multi-camera systems or is unreliable when using single-view input due to depth ambiguities. Meanwhile, mirrors are readily available in urban environments and form an affordable alternative by recording two views with only a single camera. However, the mirror setting poses the additional challenge of handling occlusions of real and mirror image. Going beyond existing mirror approaches for 3D human pose estimation, we utilize mirrors for learning a complete body model, including shape and dense appearance. Our main contributions are extending articulated neural radiance fields to include a notion of a mirror, making it sample-efficient over potential occlusion regions. Together, our contributions realize a consumer-level 3D motion capture system that starts from off-the-shelf 2D poses by automatically calibrating the camera, estimating mirror orientation, and subsequently lifting 2D keypoint detections to 3D skeleton pose that is used to condition the mirror-aware NeRF. We empirically demonstrate the benefit of learning a body model and accounting for occlusion in challenging mirror scenes.
Related papers
- Gaussian Splatting in Mirrors: Reflection-Aware Rendering via Virtual Camera Optimization [14.324573496923792]
3D-GS often misinterprets reflections as virtual spaces, resulting in blurred and inconsistent multi-view rendering within mirrors.
Our paper presents a novel method aimed at obtaining high-quality multi-view consistent reflection rendering by modelling reflections as physically-based virtual cameras.
arXiv Detail & Related papers (2024-10-02T14:53:24Z) - MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections [58.003014868772254]
MirrorGaussian is the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting.
We introduce an intuitive dual-rendering strategy that enables differentiableization of both the real-world 3D Gaussians and the mirrored counterpart.
Our approach significantly outperforms existing methods, achieving state-of-the-art results.
arXiv Detail & Related papers (2024-05-20T09:58:03Z) - Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting [27.361324194709155]
Mirror-3DGS is an innovative rendering framework devised to master the intricacies of mirror geometries and reflections.
By incorporating mirror attributes into the 3DGS, Mirror-3DGS crafts a mirrored viewpoint to observe from behind the mirror, enriching the realism of scene renderings.
arXiv Detail & Related papers (2024-04-01T15:16:33Z) - iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse
Views [61.707755434165335]
iFusion is a novel 3D object reconstruction framework that requires only two views with unknown camera poses.
We harness a pre-trained novel view synthesis diffusion model, which embeds implicit knowledge about the geometry and appearance of diverse objects.
Experiments demonstrate strong performance in both pose estimation and novel view synthesis.
arXiv Detail & Related papers (2023-12-28T18:59:57Z) - State of the Art in Dense Monocular Non-Rigid 3D Reconstruction [100.9586977875698]
3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics.
This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views.
arXiv Detail & Related papers (2022-10-27T17:59:53Z) - MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision [72.5863451123577]
We show how to train a neural model that can perform accurate 3D pose and camera estimation.
Our method outperforms both classical bundle adjustment and weakly-supervised monocular 3D baselines.
arXiv Detail & Related papers (2021-08-10T18:39:56Z) - Reconstructing 3D Human Pose by Watching Humans in the Mirror [41.894948553970245]
We introduce the new task of reconstructing 3D human pose from a single image in which we can see the person and the person's image through a mirror.
We develop an optimization-based approach that exploits mirror symmetry constraints for accurate 3D pose reconstruction.
arXiv Detail & Related papers (2021-04-01T08:42:51Z) - Structure of Multiple Mirror System from Kaleidoscopic Projections of
Single 3D Point [14.345346642066511]
This paper proposes a novel algorithm of discovering the structure of a kaleidoscopic imaging system that consists of multiple planar mirrors and a camera.
The key contribution of this paper is to propose novel algorithms for these problems using a single 3D point of unknown geometry.
arXiv Detail & Related papers (2021-03-29T11:12:15Z) - Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image
Synthesis [72.34794624243281]
We propose a self-supervised learning framework to disentangle variations from unlabeled video frames.
Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
arXiv Detail & Related papers (2020-04-09T07:55:01Z) - Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the
Wild [101.70320427145388]
We propose a weakly-supervised approach that does not require 3D annotations and learns to estimate 3D poses from unlabeled multi-view data.
We evaluate our proposed approach on two large scale datasets.
arXiv Detail & Related papers (2020-03-17T08:47:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.