Towards Privacy-preserving Photorealistic Self-avatars in Mixed Reality

• URL: http://arxiv.org/abs/2507.22153v1
• Date: Tue, 29 Jul 2025 18:37:24 GMT
• Title: Towards Privacy-preserving Photorealistic Self-avatars in Mixed Reality
• Authors: Ethan Wilson, Vincent Bindschaedler, Sophie Jörg, Sean Sheikholeslam, Kevin Butler, Eakta Jain,
• Abstract summary: Photorealistic 3D avatar generation has rapidly improved in recent years, and realistic avatars that match a user's true appearance are more feasible in Mixed Reality (MR) than ever before.<n>Yet, there are known risks to sharing one's likeness online, and photorealistic MR avatars could exacerbate these risks.<n>We propose an alternate avatar rendering scheme for broader social MR -- synthesizing realistic avatars that preserve a user's demographic identity while being distinct enough from the individual user to protect facial biometric information.
• Score: 8.591721920594441
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Photorealistic 3D avatar generation has rapidly improved in recent years, and realistic avatars that match a user's true appearance are more feasible in Mixed Reality (MR) than ever before. Yet, there are known risks to sharing one's likeness online, and photorealistic MR avatars could exacerbate these risks. If user likenesses were to be shared broadly, there are risks for cyber abuse or targeted fraud based on user appearances. We propose an alternate avatar rendering scheme for broader social MR -- synthesizing realistic avatars that preserve a user's demographic identity while being distinct enough from the individual user to protect facial biometric information. We introduce a methodology for privatizing appearance by isolating identity within the feature space of identity-encoding generative models. We develop two algorithms that then obfuscate identity: \epsmethod{} provides differential privacy guarantees and \thetamethod{} provides fine-grained control for the level of identity offset. These methods are shown to successfully generate de-identified virtual avatars across multiple generative architectures in 2D and 3D. With these techniques, it is possible to protect user privacy while largely preserving attributes related to sense of self. Employing these techniques in public settings could enable the use of photorealistic avatars broadly in MR, maintaining high realism and immersion without privacy risk.

Related papers

• Is It Really You? Exploring Biometric Verification Scenarios in Photorealistic Talking-Head Avatar Videos [12.12643642515884]
  An attacker can steal a user's avatar, preserving his appearance and voice, making it nearly impossible to detect its usage by sight or sound alone.<n>Our main question is whether an individual's facial motion patterns can serve as reliable behavioral biometrics to verify their identity when the avatar's visual appearance is a facsimile of its owner.<n> Experimental results demonstrate that facial motion landmarks enable meaningful identity verification with AUC values approaching 80%.
  arXiv  Detail & Related papers  (2025-08-01T16:23:27Z)
• SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents [91.26239311240873]
  SmartAvatar is a vision-language-agent-driven framework for generating fully rigged, animation-ready 3D human avatars.<n>A key innovation is an autonomous verification loop, where the agent renders draft avatars.<n>The generated avatars are fully rigged and support pose manipulation with consistent identity and appearance.
  arXiv  Detail & Related papers  (2025-06-05T03:49:01Z)
• GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars [44.8290935585746]
  Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar. We propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities.
  arXiv  Detail & Related papers  (2024-08-24T21:25:22Z)
• OPOM: Customized Invisible Cloak towards Face Privacy Protection [58.07786010689529]
  We investigate the face privacy protection from a technology standpoint based on a new type of customized cloak. We propose a new method, named one person one mask (OPOM), to generate person-specific (class-wise) universal masks. The effectiveness of the proposed method is evaluated on both common and celebrity datasets.
  arXiv  Detail & Related papers  (2022-05-24T11:29:37Z)
• Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Reality [68.18446501943585]
  Social presence will fuel the next generation of communication systems driven by digital humans in virtual reality (VR) The best 3D video-realistic VR avatars that minimize the uncanny effect rely on person-specific (PS) models. This paper makes progress in overcoming these limitations by proposing an end-to-end multi-identity architecture.
  arXiv  Detail & Related papers  (2021-04-10T15:48:53Z)
• High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation [117.32310997522394]
  3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Existing person-specific 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar.
  arXiv  Detail & Related papers  (2021-03-29T18:33:49Z)
• Differentially Private Imaging via Latent Space Manipulation [5.446368808660037]
  We present a novel approach for image obfuscation by manipulating latent spaces of an unconditionally trained generative model. This is the first approach to image privacy that satisfies $varepsilon$-differential privacy emphfor the person.
  arXiv  Detail & Related papers  (2021-03-08T17:32:08Z)
• IdentityDP: Differential Private Identification Protection for Face Images [17.33916392050051]
  Face de-identification, also known as face anonymization, refers to generating another image with similar appearance and the same background, while the real identity is hidden. We propose IdentityDP, a face anonymization framework that combines a data-driven deep neural network with a differential privacy mechanism. Our model can effectively obfuscate the identity-related information of faces, preserve significant visual similarity, and generate high-quality images.
  arXiv  Detail & Related papers  (2021-03-02T14:26:00Z)
• Towards Face Encryption by Generating Adversarial Identity Masks [53.82211571716117]
  We propose a targeted identity-protection iterative method (TIP-IM) to generate adversarial identity masks. TIP-IM provides 95%+ protection success rate against various state-of-the-art face recognition models.
  arXiv  Detail & Related papers  (2020-03-15T12:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.