VideoMDM: Towards 3D Human Motion Generation From 2D Supervision

• URL: http://arxiv.org/abs/2606.13364v1
• Date: Thu, 11 Jun 2026 13:49:23 GMT
• Title: VideoMDM: Towards 3D Human Motion Generation From 2D Supervision
• Abstract summary: VideoMDM is a framework that trains 3D human motion priors directly from accurate 2D poses extracted from monocular videos.<n>A pretrained 2D-to-3D lifter provides approximate 3D pose sequences that serve as a noisy teacher.<n>We show that, under mild assumptions, a depth-weighted 2D reprojection loss is equivalent in expectation to direct 3D supervision.
• Score: 13.408796377211559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce VideoMDM, a diffusion-based framework that trains 3D human motion priors directly from accurate 2D poses extracted from monocular videos, without any 3D ground truth. A pretrained 2D-to-3D lifter provides approximate 3D pose sequences that serve as a noisy teacher: these are diffused, denoised by the model in 3D, and supervised in 2D by reprojecting the prediction and comparing against accurate keypoints. We show that, under mild assumptions, a depth-weighted 2D reprojection loss is equivalent in expectation to direct 3D supervision, and we adapt standard 3D motion regularizers - velocity consistency and over-parameterized representation alignment - to this 2D setting. Unlike methods that lift 2D to 3D only at inference, VideoMDM learns a coherent 3D motion manifold during training. On HumanML3D it nearly closes the gap to fully 3D-supervised MDM (FID 0.88 vs 0.54); On real video datasets Fit3D and NBA the method learns to generate motions consistently preferred by humans, with strong quantitative results.

Related papers

• MeshFM: 2D Features Are All You Need for 3D Shape Understanding [15.930410067984198]
  MeshFM is an efficient framework for extracting rich features from 3D inputs.<n>We train a feedforward network to directly predict 3D features without requiring optimization during inference.
  arXiv  Detail & Related papers  (2026-07-30T02:22:34Z)
• Unsupervised 3D Human Pose Estimation via Conditional Multi-view Ancestral Sampling [13.79571134197704]
  We propose a method of estimating a 3D human pose from a single view without 3D supervision.<n>We leverage the 2D diffusion priors of motion diffusion models pre-trained on large 2D human pose datasets.<n>Our method achieves better cross-domain performance compared to state-of-the-art supervised and unsupervised 3D pose estimation methods.
  arXiv  Detail & Related papers  (2026-05-15T03:47:04Z)
• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight [105.9472902251177]
  We present a VLM-native recipe that casts 3D detection as a next-token prediction problem.<n>Our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement.
  arXiv  Detail & Related papers  (2025-11-25T18:59:45Z)
• Free3D: 3D Human Motion Emerges from Single-View 2D Supervision [24.238045609839983]
  Free3D is a framework that synthesizes realistic 3D motions without any 3D motion annotations.<n> trained entirely on 2D motion data, Free3D generates diverse, temporally coherent, and semantically aligned 3D motions.
  arXiv  Detail & Related papers  (2025-11-14T14:49:19Z)
• Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs [72.11701578308804]
  This paper categorizes recent 3D Vision-Language Models into 3D object-centric, 2D image-based, and 3D scene-centric approaches.<n>Despite the architectural similarity of 3D scene-centric VLMs to their 2D counterparts, they have exhibited comparatively lower performance compared with the latest 3D object-centric and 2D image-based approaches.<n>Our investigation suggests that while these models possess cross-modal alignment capabilities, they tend to over-rely on linguistic cues and overfit to frequent answer distributions.
  arXiv  Detail & Related papers  (2025-06-05T17:56:12Z)
• Unifying 2D and 3D Vision-Language Understanding [85.84054120018625]
  We introduce UniVLG, a unified architecture for 2D and 3D vision-language learning.<n>UniVLG bridges the gap between existing 2D-centric models and the rich 3D sensory data available in embodied systems.
  arXiv  Detail & Related papers  (2025-03-13T17:56:22Z)
• A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision [65.33043028101471]
  We present a novel framework for training 3D image-conditioned diffusion models using only 2D supervision.<n>Most existing 3D generative models rely on full 3D supervision, which is impractical due to the scarcity of large-scale 3D datasets.
  arXiv  Detail & Related papers  (2024-12-01T00:29:57Z)
• 2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation? [5.408549711581793]
  We study the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models. We employ a lifting model for converting generated 2D pose sequences into 3D and assess how gestures created directly in 3D stack up against those initially generated in 2D and then converted to 3D.
  arXiv  Detail & Related papers  (2024-09-16T15:06:12Z)
• EmbodiedSAM: Online Segment Any 3D Thing in Real Time [61.2321497708998]
  Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration.<n>An online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed.
  arXiv  Detail & Related papers  (2024-08-21T17:57:06Z)
• M3D-VTON: A Monocular-to-3D Virtual Try-On Network [62.77413639627565]
  Existing 3D virtual try-on methods mainly rely on annotated 3D human shapes and garment templates. We propose a novel Monocular-to-3D Virtual Try-On Network (M3D-VTON) that builds on the merits of both 2D and 3D approaches.
  arXiv  Detail & Related papers  (2021-08-11T10:05:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.