NeuralPassthrough: Learned Real-Time View Synthesis for VR
- URL: http://arxiv.org/abs/2207.02186v1
- Date: Tue, 5 Jul 2022 17:39:22 GMT
- Title: NeuralPassthrough: Learned Real-Time View Synthesis for VR
- Authors: Lei Xiao, Salah Nouri, Joel Hegland, Alberto Garcia Garcia, Douglas
Lanman
- Abstract summary: We propose the first learned passthrough method and assess its performance using a custom VR headset with a stereo pair of RGB cameras.
We demonstrate that our learned passthrough method delivers superior image quality compared to state-of-the-art methods.
- Score: 3.907767419763815
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Virtual reality (VR) headsets provide an immersive, stereoscopic visual
experience, but at the cost of blocking users from directly observing their
physical environment. Passthrough techniques are intended to address this
limitation by leveraging outward-facing cameras to reconstruct the images that
would otherwise be seen by the user without the headset. This is inherently a
real-time view synthesis challenge, since passthrough cameras cannot be
physically co-located with the eyes. Existing passthrough techniques suffer
from distracting reconstruction artifacts, largely due to the lack of accurate
depth information (especially for near-field and disoccluded objects), and also
exhibit limited image quality (e.g., being low resolution and monochromatic).
In this paper, we propose the first learned passthrough method and assess its
performance using a custom VR headset that contains a stereo pair of RGB
cameras. Through both simulations and experiments, we demonstrate that our
learned passthrough method delivers superior image quality compared to
state-of-the-art methods, while meeting strict VR requirements for real-time,
perspective-correct stereoscopic view synthesis over a wide field of view for
desktop-connected headsets.
Related papers
- Universal Facial Encoding of Codec Avatars from VR Headsets [32.60236093340087]
We present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset.
We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency.
arXiv Detail & Related papers (2024-07-17T22:08:15Z) - VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence [14.010324388059866]
VOODOO XP is a 3D-aware one-shot head reenactment method that can generate highly expressive facial expressions from any input driver video and a single 2D portrait.
We show our solution on a monocular video setting and an end-to-end VR telepresence system for two-way communication.
arXiv Detail & Related papers (2024-05-25T12:33:40Z) - Real-Time Simulated Avatar from Head-Mounted Sensors [70.41580295721525]
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets.
To synergize headset poses with cameras, we control a humanoid to track headset movement while analyzing input images to decide body movement.
When body parts are seen, the movements of hands and feet will be guided by the images; when unseen, the laws of physics guide the controller to generate plausible motion.
arXiv Detail & Related papers (2024-03-11T16:15:51Z) - Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended
Reality [65.70936336240554]
Real-time Stereo Matching is a cornerstone algorithm for many Extended Reality (XR) applications, such as indoor 3D understanding, video pass-through, and mixed-reality games.
One of the major difficulties is the lack of high-quality indoor video stereo training datasets captured by head-mounted VR/AR glasses.
We introduce a novel video stereo synthetic dataset that comprises renderings of various indoor scenes and realistic camera motion captured by a 6-DoF moving VR/AR head-mounted display (HMD).
This facilitates the evaluation of existing approaches and promotes further research on indoor augmented reality scenarios.
arXiv Detail & Related papers (2023-09-08T07:53:58Z) - Towards a Pipeline for Real-Time Visualization of Faces for VR-based
Telepresence and Live Broadcasting Utilizing Neural Rendering [58.720142291102135]
Head-mounted displays (HMDs) for Virtual Reality pose a considerable obstacle for a realistic face-to-face conversation in VR.
We present an approach that focuses on low-cost hardware and can be used on a commodity gaming computer with a single GPU.
arXiv Detail & Related papers (2023-01-04T08:49:51Z) - ChromaCorrect: Prescription Correction in Virtual Reality Headsets
through Perceptual Guidance [3.365646526465954]
eyeglasses causes additional bulk and discomfort when used with augmented and virtual reality headsets.
We propose a prescription-aware rendering approach for providing sharper and immersive VR imagery.
We evaluate our approach on various displays, including desktops and VR headsets, and show significant quality and contrast improvements for users with vision impairments.
arXiv Detail & Related papers (2022-12-08T13:30:17Z) - Facial De-occlusion Network for Virtual Telepresence Systems [6.501857679289835]
State-of-the-art image inpainting methods for de-occluding the eye region does not give usable results.
We propose a working solution that gives usable results to tackle this problem enabling the use of the real-time photo-realistic de-occluded face of the user in VR settings.
arXiv Detail & Related papers (2022-10-23T05:34:17Z) - Deep 3D Mask Volume for View Synthesis of Dynamic Scenes [49.45028543279115]
We introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS.
The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes.
We develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable view extrapolation from binocular videos of dynamic scenes, captured by static cameras.
arXiv Detail & Related papers (2021-08-30T17:55:28Z) - Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual
Reality [68.18446501943585]
Social presence will fuel the next generation of communication systems driven by digital humans in virtual reality (VR)
The best 3D video-realistic VR avatars that minimize the uncanny effect rely on person-specific (PS) models.
This paper makes progress in overcoming these limitations by proposing an end-to-end multi-identity architecture.
arXiv Detail & Related papers (2021-04-10T15:48:53Z) - Unmasking Communication Partners: A Low-Cost AI Solution for Digitally
Removing Head-Mounted Displays in VR-Based Telepresence [62.997667081978825]
Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD)
Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware.
We propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware.
arXiv Detail & Related papers (2020-11-06T23:17:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.