4K4D: Real-Time 4D View Synthesis at 4K Resolution
- URL: http://arxiv.org/abs/2310.11448v3
- Date: Sat, 28 Oct 2023 06:41:48 GMT
- Title: 4K4D: Real-Time 4D View Synthesis at 4K Resolution
- Authors: Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun
Shen, Hujun Bao, Xiaowei Zhou
- Abstract summary: This paper targets high-fidelity and real-time view of dynamic 3D scenes at 4K resolution.
We propose a 4D point cloud representation that supports hardwareization and enables unprecedented rendering speed.
Our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an 4090 GPU.
- Score: 86.6582179227016
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper targets high-fidelity and real-time view synthesis of dynamic 3D
scenes at 4K resolution. Recently, some methods on dynamic view synthesis have
shown impressive rendering quality. However, their speed is still limited when
rendering high-resolution images. To overcome this problem, we propose 4K4D, a
4D point cloud representation that supports hardware rasterization and enables
unprecedented rendering speed. Our representation is built on a 4D feature grid
so that the points are naturally regularized and can be robustly optimized. In
addition, we design a novel hybrid appearance model that significantly boosts
the rendering quality while preserving efficiency. Moreover, we develop a
differentiable depth peeling algorithm to effectively learn the proposed model
from RGB videos. Experiments show that our representation can be rendered at
over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the
ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x
faster than previous methods and achieves the state-of-the-art rendering
quality. Our project page is available at https://zju3dv.github.io/4k4d/.
Related papers
- 4K4DGen: Panoramic 4D Generation at 4K Resolution [66.32782301285256]
We propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360-degree images.
We achieve high-quality Panorama-to-4D generation at a resolution of (4096 $times$ 2048) for the first time.
arXiv Detail & Related papers (2024-06-19T13:11:02Z) - Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video [42.10482273572879]
We propose an efficient video-to-4D object generation framework called Efficient4D.
It generates high-quality spacetime-consistent images under different camera views, and then uses them as labeled data.
Experiments on both synthetic and real videos show that Efficient4D offers a remarkable 10-fold increase in speed.
arXiv Detail & Related papers (2024-01-16T18:58:36Z) - UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale
Scene [52.21184153832739]
We propose a novel neural rendering system called UE4-NeRF, specifically designed for real-time rendering of large-scale scenes.
Our approach combines with the Unrealization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS.
arXiv Detail & Related papers (2023-10-20T04:01:35Z) - Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic
Scenes [69.52540205439989]
We introduce Im4D, a hybrid representation that consists of a grid-based geometry representation and a multi-view image-based appearance representation.
We represent the scene appearance by the original multi-view videos and a network that learns to predict the color of a 3D point from image features.
We show that Im4D state-of-the-art performance in rendering quality and can be trained efficiently, while realizing real-time rendering with a speed of 79.8 FPS for 512x512 images.
arXiv Detail & Related papers (2023-10-12T17:59:57Z) - 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering [103.32717396287751]
We propose 4D Gaussian Splatting (4D-GS) as a holistic representation for dynamic scenes.
A neuralvoxel encoding algorithm inspired by HexPlane is proposed to efficiently build features from 4D neural voxels.
Our 4D-GS method achieves real-time rendering under high resolutions, 82 FPS at an 800$times$800 resolution on an 3090 GPU.
arXiv Detail & Related papers (2023-10-12T17:21:41Z) - 4K-HAZE: A Dehazing Benchmark with 4K Resolution Hazy and Haze-Free
Images [12.402054374952485]
We develop a novel method to simulate 4K hazy images from clear images, which first estimates the scene depth, simulates the light rays and object reflectance, then migrates the synthetic images to real domains by using a GAN.
We wrap these synthesized images into a benchmark called the 4K-HAZE dataset.
The most appealing aspect of our approach is the capability to run a 4K image on a single GPU with 24G RAM in real-time (33fps)
arXiv Detail & Related papers (2023-03-28T09:39:29Z) - NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed
Neural Radiance Fields [99.57774680640581]
We present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering.
We propose to decompose the 4D space according to temporal characteristics. Points in the 4D space are associated with probabilities belonging to three categories: static, deforming, and new areas.
arXiv Detail & Related papers (2022-10-28T07:11:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.