LUNA: Learning Universal 3D Human Animation Beyond Skinning
Abstract Overview
This paper presents LUNA, an end-to-end 3D human animation framework that avoids Linear Blend Skinning (LBS) and instead maps diverse 2D driving inputs directly to deformations of a canonical 3D Gaussian avatar. The method reconstructs canonical Gaussians from a small set of identity images, then uses a transformer-based implicit neural animator that separates global rigid motion from local non-rigid deformation. To address the ambiguity of 2D-to-3D lifting, the training scheme combines photometric rendering losses with LBS-teacher distillation and supports both labeled data and large-scale unlabeled in-the-wild videos. The reported goal is to preserve photorealism while reducing pose-induced artifacts and enabling cross-identity, cross-modality animation from images, keypoints, sketches, and out-of-domain characters.
Novelty
The main novelty is an LBS-free, end-to-end 3D animatable model that supports implicit 2D driving across multiple modalities, rather than relying on explicit body fitting or parametric pose control. The paper also introduces a hybrid supervision strategy that uses LBS only as a soft structural prior while learning a direct 2D-to-3D deformation mapping.
Results
Experimentally, LUNA outperforms the compared methods on the non-rigid Cloth10K benchmark, reaching 22.072 PSNR, 0.027 L1, and 0.131 LPIPS, while remaining comparable to strong feed-forward baselines on NeuMan. It also improves temporal stability substantially, reducing MSJ to 0.0032 and MAE to 0.0225, including a reported 4.5x MSJ reduction over the strongest baseline (MV-LHM). In cross-identity driving on NeuMan, its performance stays close to self-reenactment and achieves the lowest reported normalized 2D keypoint error among compared methods.
Key Points
- LUNA directly animates canonical 3D Gaussians from heterogeneous 2D controls, including RGB images, 2D keypoints, sketches, and unseen character images, without explicit body fitting or preprocessing.
- The architecture disentangles global rotation/translation from local per-Gaussian residual deformation, aiming to capture both articulated motion and fine non-rigid effects such as loose clothing.
- Hybrid supervision is important to performance: LBS-guided distillation prevents geometric collapse, and multiview fine-tuning improves spatial compactness and visual quality in the ablations.