Eye-See-You: Reverse Pass-Through VR and Head Avatars

• URL: http://arxiv.org/abs/2505.18869v2
• Date: Wed, 28 May 2025 02:10:43 GMT
• Title: Eye-See-You: Reverse Pass-Through VR and Head Avatars
• Authors: Ankan Dash, Jingyi Gu, Guiling Wang, Chen Chen,
• Abstract summary: RevAvatar is an innovative framework that leverages AI methodologies to enable reverse pass-through technology.<n>RevAvatar integrates state-of-the-art generative models and multimodal AI techniques to reconstruct high-fidelity 2D facial images.<n>We present VR-Face, a novel dataset comprising 200,000 samples designed to emulate diverse VR-specific conditions.
• Score: 6.514203459446675
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Virtual Reality (VR) headsets, while integral to the evolving digital ecosystem, present a critical challenge: the occlusion of users' eyes and portions of their faces, which hinders visual communication and may contribute to social isolation. To address this, we introduce RevAvatar, an innovative framework that leverages AI methodologies to enable reverse pass-through technology, fundamentally transforming VR headset design and interaction paradigms. RevAvatar integrates state-of-the-art generative models and multimodal AI techniques to reconstruct high-fidelity 2D facial images and generate accurate 3D head avatars from partially observed eye and lower-face regions. This framework represents a significant advancement in AI4Tech by enabling seamless interaction between virtual and physical environments, fostering immersive experiences such as VR meetings and social engagements. Additionally, we present VR-Face, a novel dataset comprising 200,000 samples designed to emulate diverse VR-specific conditions, including occlusions, lighting variations, and distortions. By addressing fundamental limitations in current VR systems, RevAvatar exemplifies the transformative synergy between AI and next-generation technologies, offering a robust platform for enhancing human connection and interaction in virtual environments.

Related papers

• How LLMs are Shaping the Future of Virtual Reality [2.4150871564195007]
  The integration of Large Language Models (LLMs) into Virtual Reality (VR) games marks a paradigm shift in the design of immersive, adaptive, and intelligent digital experiences.<n>This paper examines how these models are transforming narrative generation, non-player character (NPC) interactions, accessibility, personalization, and game mastering.
  arXiv  Detail & Related papers  (2025-08-01T16:08:05Z)
• EVA: Expressive Virtual Avatars from Multi-view Videos [51.33851869426057]
  We introduce Expressive Virtual Avatars (EVA), an actor-specific, fully controllable, and expressive human avatar framework.<n>EVA achieves high-fidelity, lifelike renderings in real time while enabling independent control of facial expressions, body movements, and hand gestures.<n>This work represents a significant advancement towards fully drivable digital human models.
  arXiv  Detail & Related papers  (2025-05-21T11:22:52Z)
• Universal Facial Encoding of Codec Avatars from VR Headsets [32.60236093340087]
  We present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency.
  arXiv  Detail & Related papers  (2024-07-17T22:08:15Z)
• VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality [39.53150683721031]
  Our proposed VR-GS system represents a leap forward in human-centered 3D content interaction. The components of our Virtual Reality system are designed for high efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-01-30T01:28:36Z)
• Facial De-occlusion Network for Virtual Telepresence Systems [6.501857679289835]
  State-of-the-art image inpainting methods for de-occluding the eye region does not give usable results. We propose a working solution that gives usable results to tackle this problem enabling the use of the real-time photo-realistic de-occluded face of the user in VR settings.
  arXiv  Detail & Related papers  (2022-10-23T05:34:17Z)
• Force-Aware Interface via Electromyography for Natural VR/AR Interaction [69.1332992637271]
  We design a learning-based neural interface for natural and intuitive force inputs in VR/AR. We show that our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. We envision our findings to push forward research towards more realistic physicality in future VR/AR.
  arXiv  Detail & Related papers  (2022-10-03T20:51:25Z)
• Wireless Edge-Empowered Metaverse: A Learning-Based Incentive Mechanism for Virtual Reality [102.4151387131726]
  We propose a learning-based Incentive Mechanism framework for VR services in the Metaverse. First, we propose the quality of perception as the metric for VR users in the virtual world. Second, for quick trading of VR services between VR users (i.e., buyers) and VR SPs (i.e., sellers), we design a double Dutch auction mechanism. Third, for auction communication reduction, we design a deep reinforcement learning-based auctioneer to accelerate this auction process.
  arXiv  Detail & Related papers  (2021-11-07T13:02:52Z)
• Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Reality [68.18446501943585]
  Social presence will fuel the next generation of communication systems driven by digital humans in virtual reality (VR) The best 3D video-realistic VR avatars that minimize the uncanny effect rely on person-specific (PS) models. This paper makes progress in overcoming these limitations by proposing an end-to-end multi-identity architecture.
  arXiv  Detail & Related papers  (2021-04-10T15:48:53Z)
• Pixel Codec Avatars [99.36561532588831]
  Pixel Codec Avatars (PiCA) is a deep generative model of 3D human faces. On a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.
  arXiv  Detail & Related papers  (2021-04-09T23:17:36Z)
• Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence [62.997667081978825]
  Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD) Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. We propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware.
  arXiv  Detail & Related papers  (2020-11-06T23:17:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.