HumanSplatHMR: Closing the Loop Between Human Mesh Recovery and Gaussian Splatting Avatar

• URL: http://arxiv.org/abs/2605.02784v1
• Date: Mon, 04 May 2026 16:26:11 GMT
• Title: HumanSplatHMR: Closing the Loop Between Human Mesh Recovery and Gaussian Splatting Avatar
• Abstract summary: Accurately recovering human pose and appearance from video is an essential component of scene reconstruction.<n>This paper proposes HumanSplatHMR, a joint optimization framework that refines 3D human poses.<n> Experiments show consistent improvements over pose recovery baselines that decouple pose estimation from avatar reconstruction.
• Score: 19.538398405782996
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurately recovering human pose and appearance from video is an essential component of scene reconstruction, with applications to motion capture, motion prediction, virtual reality, and digital twinning. Despite significant interest in building realistic human avatars from video, this paper demonstrates that existing methods do not accurately recover the 3D geometry of humans. ViT-based approaches are not consistently reliable and can overfit to 2D views, while NeRF- and Gaussian Splatting-based avatars treat pose and appearance separately, limiting rendering generalization to new poses. To resolve these shortcomings, this paper proposes HumanSplatHMR, a joint optimization framework that refines 3D human poses while simultaneously learning a high-fidelity avatar for novel-view and novel-pose synthesis. Our key insight is to close the loop between geometric pose estimation and differentiable rendering. Unlike prior human avatar methods that rely on accurate human pose obtained through motion capture systems or offline refinement, which are impractical in in-the-wild scenarios, our approach uses only human mesh estimates from a state-of-the-art human pose estimator to better reflect real-world conditions. Therefore, instead of using the human pose only as a deformation prior, HumanSplatHMR backpropagates photometric, segmentation, and depth losses through a differentiable renderer to the pose parameters and global position. This coupling refines the global 3D pose over time, improving accuracy and alignment while producing better renderings from novel views. Experiments show consistent improvements over pose recovery baselines that omit image-level refinement and avatar baselines that decouple pose estimation from avatar reconstruction.

Related papers

• F3G-Avatar : Face Focused Full-body Gaussian Avatar [5.5263731799099425]
  F3G-Avatar is a full-body, face-aware avatar synthesis method.<n>It reconstructs animatable human representations from multi-view RGB video and regressed pose/shape parameters.<n>Experiments demonstrate strong rendering quality, with face-view performance reaching PSNR/SSIM/LPIPS of 26.243/0.964/0.084 on the AvatarReX dataset.
  arXiv  Detail & Related papers  (2026-04-10T19:08:46Z)
• PF-LHM: 3D Animatable Avatar Reconstruction from Pose-free Articulated Human Images [23.745241278910946]
  PF-LHM is a large human reconstruction model that generates high-quality 3D avatars in seconds from one or multiple casually captured pose-free images.<n>Our method unifies single- and multi-image 3D human reconstruction, achieving high-fidelity and animatable 3D human avatars without requiring camera and human pose annotations.
  arXiv  Detail & Related papers  (2025-06-16T17:59:56Z)
• HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers [60.86393841247567]
  HumanRAM is a novel feed-forward approach for generalizable human reconstruction and animation from monocular or sparse human images.<n>Our approach integrates human reconstruction and animation into a unified framework by introducing explicit pose conditions.<n> Experiments show that HumanRAM significantly surpasses previous methods in terms of reconstruction accuracy, animation fidelity, and generalization performance on real-world datasets.
  arXiv  Detail & Related papers  (2025-06-03T17:50:05Z)
• ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos [18.73641648585445]
  Recent neural rendering advances have enabled holistic human-scene reconstruction but require pre-calibrated camera and human poses.<n>We introduce a novel unified framework that simultaneously performs camera tracking, human pose estimation and human-scene reconstruction in an online fashion.<n>Specifically, we design a human deformation module to reconstruct the details and enhance generalizability to out-of-distribution poses faithfully.
  arXiv  Detail & Related papers  (2025-04-17T17:59:02Z)
• FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images [74.86864398919467]
  We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images.<n>We learn a universal prior from over a thousand clothed humans to achieve instant feedforward generation and zero-shot generalization.<n>Our method generates more authentic reconstruction and animation than state-of-the-arts, and can be directly generalized to inputs from casually taken phone photos.
  arXiv  Detail & Related papers  (2025-03-24T23:20:47Z)
• Better Together: Unified Motion Capture and 3D Avatar Reconstruction [6.329917162442801]
  We present a method that simultaneously solves the human pose estimation problem while reconstructing a 3D human avatar from multi-view videos.<n>We introduce a novel animatable avatar with 3D Gaussians rigged on a personalized mesh.<n>We first evaluate our method on highly challenging yoga poses and demonstrate state-of-the-art accuracy on multi-view human pose estimation.
  arXiv  Detail & Related papers  (2025-03-12T11:39:43Z)
• AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos [31.904839609743448]
  Existing multi-view methods often face challenges in estimating the 3D pose and shape of multiple closely interacting people. We propose a novel method leveraging the personalized implicit neural avatar of each individual as a prior. Our experimental results demonstrate state-of-the-art performance on several public datasets.
  arXiv  Detail & Related papers  (2024-08-04T18:41:35Z)
• GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos [56.40776739573832]
  We present a novel method that facilitates the creation of vivid 3D Gaussian avatars from monocular video inputs (GVA) Our innovation lies in addressing the intricate challenges of delivering high-fidelity human body reconstructions. We introduce a pose refinement technique to improve hand and foot pose accuracy by aligning normal maps and silhouettes.
  arXiv  Detail & Related papers  (2024-02-26T14:40:15Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• ARCH++: Animation-Ready Clothed Human Reconstruction Revisited [82.83445332309238]
  We present ARCH++, an image-based method to reconstruct 3D avatars with arbitrary clothing styles. Our reconstructed avatars are animation-ready and highly realistic, in both the visible regions from input views and the unseen regions.
  arXiv  Detail & Related papers  (2021-08-17T19:27:12Z)
• Animatable Neural Radiance Fields from Monocular RGB Video [72.6101766407013]
  We present animatable neural radiance fields for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields to the dynamic scenes with human movements via introducing explicit pose-guided deformation. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from arbitrary views, and 3) animation of the human with arbitrary poses.
  arXiv  Detail & Related papers  (2021-06-25T13:32:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.