Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

• URL: http://arxiv.org/abs/2606.23688v1
• Date: Mon, 22 Jun 2026 17:59:54 GMT
• Title: Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
• Abstract summary: Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance.<n>Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence.<n>We present Lift4D, a test-time optimization framework that addresses both limitations.
• Score: 87.49244020695505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.

Related papers

• Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video [69.94904173314505]
  We propose Ground4D, a geometry-grounded framework built on two stages.<n>First, we reconstruct multi-view-consistent 3D geometry and camera poses from monocular video.<n>Second, we conduct geometry-consistency-aware refinement via dynamic Gaussian Splatting.
  arXiv  Detail & Related papers  (2026-06-27T09:27:42Z)
• Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video [81.44600627066747]
  Mesh4D is a feed-forward model for monocular 4D mesh reconstruction.<n>Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass.<n>We evaluate Mesh4D on reconstruction and view novel benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.
  arXiv  Detail & Related papers  (2026-01-08T18:59:56Z)
• Flux4D: Flow-based Unsupervised 4D Reconstruction [30.764886648248222]
  Reconstructing large-scale dynamic scenes from visual observations is a fundamental challenge in computer vision.<n>We introduce Flux4D, a simple and scalable framework for 4D reconstruction of large-scale dynamic scenes.<n>Our approach enables efficient reconstruction of dynamic scenes within seconds, scales effectively to large datasets, and generalizes well to unseen environments.
  arXiv  Detail & Related papers  (2025-12-02T20:28:45Z)
• Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models [79.06910348413861]
  We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image.<n>Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion.
  arXiv  Detail & Related papers  (2025-11-01T11:16:25Z)
• Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency [49.875459658889355]
  Free4D is a tuning-free framework for 4D scene generation from a single image.<n>Our key insight is to distill pre-trained foundation models for consistent 4D scene representation.<n>The resulting 4D representation enables real-time, controllable rendering.
  arXiv  Detail & Related papers  (2025-03-26T17:59:44Z)
• AR4D: Autoregressive 4D Generation from Monocular Videos [27.61057927559143]
  Existing approaches primarily rely on Score Distillation Sampling to infer novel-view videos.<n>We propose AR4D, a novel paradigm for SDS-free 4D generation.<n>We show that AR4D can achieve state-of-the-art 4D generation without SDS, delivering greater diversity, improved spatial-temporal consistency, and better alignment with input prompts.
  arXiv  Detail & Related papers  (2025-01-03T09:27:36Z)
• Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model [76.64071133839862]
  Capturing general deforming scenes from monocular RGB video is crucial for many computer graphics and vision applications. Our method, Ub4D, handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering. Results on our new dataset, which will be made publicly available, demonstrate a clear improvement over the state of the art in terms of surface reconstruction accuracy and robustness to large deformations.
  arXiv  Detail & Related papers  (2022-06-16T17:59:54Z)
• 4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface [7.637832293935966]
  We introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D.
  arXiv  Detail & Related papers  (2021-05-05T07:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.