Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering

• URL: http://arxiv.org/abs/2211.11610v2
• Date: Thu, 13 Apr 2023 11:42:12 GMT
• Title: Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering
• Authors: Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, Yebin Liu
• Abstract summary: We propose an efficient 4D tensor decomposition method for dynamic scenes. We show that our method is able to achieve high-quality dynamic reconstruction and rendering from sparse-view camera or even a monocular camera. The code and dataset will be released atliuyebin.com/tensor4d-tensor4d.html.
• Score: 31.928844354349117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Tensor4D, an efficient yet effective approach to dynamic scene modeling. The key of our solution is an efficient 4D tensor decomposition method so that the dynamic scene can be directly represented as a 4D spatio-temporal tensor. To tackle the accompanying memory issue, we decompose the 4D tensor hierarchically by projecting it first into three time-aware volumes and then nine compact feature planes. In this way, spatial information over time can be simultaneously captured in a compact and memory-efficient manner. When applying Tensor4D for dynamic scene reconstruction and rendering, we further factorize the 4D fields to different scales in the sense that structural motions and dynamic detailed changes can be learned from coarse to fine. The effectiveness of our method is validated on both synthetic and real-world scenes. Extensive experiments show that our method is able to achieve high-quality dynamic reconstruction and rendering from sparse-view camera rigs or even a monocular camera. The code and dataset will be released at https://liuyebin.com/tensor4d/tensor4d.html.

Related papers

• S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points [30.46796069720543]
  We introduce a novel approach for streaming 4D real-world reconstruction utilizing discrete 3D control points. This method physically models local rays and establishes a motion-decoupling coordinate system. By effectively merging traditional graphics with learnable pipelines, it provides a robust and efficient local 6-degrees-of-freedom (6 DoF) motion representation.
  arXiv  Detail & Related papers  (2024-08-23T12:51:49Z)
• Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models [116.31344506738816]
  We present a novel framework, textbfDiffusion4D, for efficient and scalable 4D content generation. We develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. Our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency.
  arXiv  Detail & Related papers  (2024-05-26T17:47:34Z)
• Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video [42.10482273572879]
  We propose an efficient video-to-4D object generation framework called Efficient4D. It generates high-quality spacetime-consistent images under different camera views, and then uses them as labeled data. Experiments on both synthetic and real videos show that Efficient4D offers a remarkable 10-fold increase in speed.
  arXiv  Detail & Related papers  (2024-01-16T18:58:36Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency [118.15258850780417]
  This work introduces 4DGen, a novel framework for grounded 4D content creation. We identify static 3D assets and monocular video sequences as key components in constructing the 4D content. Our pipeline facilitates conditional 4D generation, enabling users to specify geometry (3D assets) and motion (monocular videos)
  arXiv  Detail & Related papers  (2023-12-28T18:53:39Z)
• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models [94.07744207257653]
  We focus on the underexplored text-to-4D setting and synthesize dynamic, animated 3D objects. We combine text-to-image, text-to-video, and 3D-aware multiview diffusion models to provide feedback during 4D object optimization.
  arXiv  Detail & Related papers  (2023-12-21T11:41:02Z)
• Consistent4D: Consistent 360{\deg} Dynamic Object Generation from Monocular Video [15.621374353364468]
  Consistent4D is a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. We cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration.
  arXiv  Detail & Related papers  (2023-11-06T03:26:43Z)
• Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting [8.078460597825142]
  Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. We propose to approximate the underlying-temporal rendering volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics.
  arXiv  Detail & Related papers  (2023-10-16T17:57:43Z)
• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields [99.57774680640581]
  We present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. We propose to decompose the 4D space according to temporal characteristics. Points in the 4D space are associated with probabilities belonging to three categories: static, deforming, and new areas.
  arXiv  Detail & Related papers  (2022-10-28T07:11:05Z)
• LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling [69.56581851211841]
  We propose a novel Local 4D implicit Representation for Dynamic clothed human, named LoRD. Our key insight is to encourage the network to learn the latent codes of local part-level representation. LoRD has strong capability for representing 4D human, and outperforms state-of-the-art methods on practical applications.
  arXiv  Detail & Related papers  (2022-08-18T03:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.