AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors

• URL: http://arxiv.org/abs/2603.17975v1
• Date: Wed, 18 Mar 2026 17:39:05 GMT
• Title: AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors
• Abstract summary: AHOY is a method for reconstructing complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion.<n>We develop a hallucination-as-supervision pipeline that uses identity-finetuned diffusion models to generate dense supervision for previously unobserved body regions.<n>The resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes captured using cell-phone video.
• Score: 58.22950708041041
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present AHOY, a method for reconstructing complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion. Existing methods assume unoccluded input-a fully visible subject, often in a canonical pose-excluding the vast majority of real-world footage where people are routinely occluded by furniture, objects, or other people. Reconstructing from such footage poses fundamental challenges: large body regions may never be observed, and multi-view supervision per pose is unavailable. We address these challenges with four contributions: (i) a hallucination-as-supervision pipeline that uses identity-finetuned diffusion models to generate dense supervision for previously unobserved body regions; (ii) a two-stage canonical-to-pose-dependent architecture that bootstraps from sparse observations to full pose-dependent Gaussian maps; (iii) a map-pose/LBS-pose decoupling that absorbs multi-view inconsistencies from the generated data; (iv) a head/body split supervision strategy that preserves facial identity. We evaluate on YouTube videos and on multi-view capture data with significant occlusion and demonstrate state-of-the-art reconstruction quality. We also demonstrate that the resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes captured using cell-phone video. Our project page is available at https://miraymen.github.io/ahoy/

Related papers

• Human Video Generation from a Single Image with 3D Pose and View Control [62.676151243249556]
  We present Human Video Generation in 4D (HVG), a latent video diffusion model capable of generating high-quality multi-view,temporally coherent human videos from a single image.<n>HVG achieves this through three key designs: (i) Articulated Pose Modulation, which captures the anatomical relationships of 3D joints via a novel dual-dimensional bone map and resolves self-occlusions across views by introducing 3D information; (ii) View and Temporal Alignment, which ensures multi-view consistency and alignment between a reference image and pose sequences for frame-to-frame stability; and (iii)
  arXiv  Detail & Related papers  (2026-02-24T18:42:20Z)
• LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models [52.656349227001925]
  Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory.<n>Existing methods face two distinct challenges.<n>We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model.
  arXiv  Detail & Related papers  (2026-01-21T05:46:03Z)
• Dream, Lift, Animate: From Single Images to Animatable Gaussian Avatars [20.807609264738865]
  We introduce Dream, Lift, Animate (DLA), a novel framework that reconstructs animatable 3D human avatars from a single image.<n>This is achieved by leveraging multi-view generation, 3D Gaussian lifting, and pose-aware UV-space mapping of 3D Gaussians.<n>Our method outperforms state-of-the-art approaches on ActorsHQ and 4D-Dress datasets in both perceptual quality and photometric accuracy.
  arXiv  Detail & Related papers  (2025-07-21T18:20:09Z)
• UAV4D: Dynamic Neural Rendering of Human-Centric UAV Imagery using Gaussian Splatting [54.883935964137706]
  We introduce UAV4D, a framework for enabling photorealistic rendering for dynamic real-world scenes captured by UAVs.<n>We use a combination of a 3D foundation model and a human mesh reconstruction model to reconstruct both the scene background and humans.<n>Our results demonstrate the benefits of our approach over existing methods in novel view synthesis, achieving a 1.5 dB PSNR improvement and superior visual sharpness.
  arXiv  Detail & Related papers  (2025-06-05T13:21:09Z)
• Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior [31.780579293685797]
  We present Vid2Avatar-Pro, a method to create photorealistic and animatable 3D human avatars from monocular in-the-wild videos.
  arXiv  Detail & Related papers  (2025-03-03T14:45:35Z)
• Can Generative Video Models Help Pose Estimation? [42.10672365565019]
  Pairwise pose estimation from images with little or no overlap is an open challenge in computer vision.<n>Inspired by the human ability to infer spatial relationships from diverse scenes, we propose a novel approach, InterPose.<n>We propose a video model to hallucinate intermediate frames between two input images, effectively creating a dense, visual transition.
  arXiv  Detail & Related papers  (2024-12-20T18:58:24Z)
• GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion [5.49003371165534]
  Photorealistic 3D head avatar reconstruction from recordings is challenging due to limited observations.<n>We introduce a multi-view head diffusion model, leveraging its priors to fill in missing regions and ensure view consistency.<n>We evaluate our method on the NeRSemble dataset, showing that our method outperforms previous state-of-the-art methods in novel view synthesis.
  arXiv  Detail & Related papers  (2024-12-13T15:31:22Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision.<n>Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data.<n>We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision [75.38953287579616]
  We present a novel framework to exploit Multi-view Occlusion-aware supervision from hand-object videos for Hand-held Object reconstruction. We tackle two predominant challenges in such setting: hand-induced occlusion and object's self-occlusion. Experiments on HO3D and DexYCB datasets demonstrate 2D-supervised MOHO gains superior results against 3D-supervised methods by a large margin.
  arXiv  Detail & Related papers  (2023-10-18T03:57:06Z)
• Towards Robust and Smooth 3D Multi-Person Pose Estimation from Monocular Videos in the Wild [10.849750765175754]
  POTR-3D is a sequence-to-sequence 2D-to-3D lifting model for 3DMPPE. It robustly generalizes to diverse unseen views, robustly recovers the poses against heavy occlusions, and reliably generates more natural and smoother outputs.
  arXiv  Detail & Related papers  (2023-09-15T06:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.