MetaFi: Device-Free Pose Estimation via Commodity WiFi for Metaverse Avatar Simulation

• URL: http://arxiv.org/abs/2208.10414v1
• Date: Mon, 22 Aug 2022 15:50:54 GMT
• Title: MetaFi: Device-Free Pose Estimation via Commodity WiFi for Metaverse Avatar Simulation
• Authors: Jianfei Yang, Yunjiao Zhou, He Huang, Han Zou, Lihua Xie
• Abstract summary: WiFi is ubiquitous and robust to illumination, making it a feasible solution for avatar applications in smart home. Deep neural network is designed with customized convolutional layers and residual blocks to map the channel state information to human pose landmarks. Experiments are conducted in the real world, and the results show that the MetaFi achieves very high performance with a PCK@50 of 95.23%.
• Score: 35.1943579208943
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Avatar refers to a representative of a physical user in the virtual world that can engage in different activities and interact with other objects in metaverse. Simulating the avatar requires accurate human pose estimation. Though camera-based solutions yield remarkable performance, they encounter the privacy issue and degraded performance caused by varying illumination, especially in smart home. In this paper, we propose a WiFi-based IoT-enabled human pose estimation scheme for metaverse avatar simulation, namely MetaFi. Specifically, a deep neural network is designed with customized convolutional layers and residual blocks to map the channel state information to human pose landmarks. It is enforced to learn the annotations from the accurate computer vision model, thus achieving cross-modal supervision. WiFi is ubiquitous and robust to illumination, making it a feasible solution for avatar applications in smart home. The experiments are conducted in the real world, and the results show that the MetaFi achieves very high performance with a PCK@50 of 95.23%.

Related papers

• POSE: Pose estimation Of virtual Sync Exhibit system [0.0]
  The motivation is that we find it inconvenient to use joysticks and sensors when playing with fitness rings. In order to replace joysticks and reduce costs, we developed a platform that can control virtual avatars through pose estimation to identify the movements of real people.
  arXiv  Detail & Related papers  (2024-10-20T09:34:15Z)
• EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars [56.56236652774294]
  We propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods.
  arXiv  Detail & Related papers  (2024-09-22T22:50:27Z)
• Vision Reimagined: AI-Powered Breakthroughs in WiFi Indoor Imaging [4.236383297604285]
  WiFi as an omnipresent signal is a promising candidate for carrying out passive imaging and synchronizing the up-to-date information to all connected devices. This is the first research work to consider WiFi indoor imaging as a multi-modal image generation task that converts the measured WiFi power into a high-resolution indoor image. Our proposed WiFi-GEN network achieves a shape reconstruction accuracy that is 275% of that achieved by physical model-based methods.
  arXiv  Detail & Related papers  (2024-01-09T02:20:30Z)
• Physics-based Motion Retargeting from Sparse Inputs [73.94570049637717]
  Commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available.
  arXiv  Detail & Related papers  (2023-07-04T21:57:05Z)
• AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing [24.053096294334694]
  We present AvatarPoser, the first learning-based method that predicts full-body poses in world coordinates using only motion input from the user's head and hands. Our method builds on a Transformer encoder to extract deep features from the input signals and decouples global motion from the learned local joint orientations. In our evaluation, AvatarPoser achieved new state-of-the-art results in evaluations on large motion capture datasets.
  arXiv  Detail & Related papers  (2022-07-27T20:52:39Z)
• Drivable Volumetric Avatars using Texel-Aligned Features [52.89305658071045]
  Photo telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance. We propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people.
  arXiv  Detail & Related papers  (2022-07-20T09:28:16Z)
• Edge-enabled Metaverse: The Convergence of Metaverse and Mobile Edge Computing [6.335949956497453]
  State-of-the-art Metaverse architectures rely on a cloud-based approach for avatar physics emulation and graphics rendering computation. We propose a Fog-Edge hybrid computing architecture for Metaverse applications that leverage an edge-enabled distributed computing paradigm. We show that the proposed architecture can reduce the latency by 50% when compared with the legacy cloud-based Metaverse applications.
  arXiv  Detail & Related papers  (2022-04-13T11:38:57Z)
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images [87.96474006263692]
  We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation.
  arXiv  Detail & Related papers  (2021-11-24T18:33:02Z)
• Pixel Codec Avatars [99.36561532588831]
  Pixel Codec Avatars (PiCA) is a deep generative model of 3D human faces. On a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.
  arXiv  Detail & Related papers  (2021-04-09T23:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.