Instant-NeRF: Instant On-Device Neural Radiance Field Training via
Algorithm-Accelerator Co-Designed Near-Memory Processing
- URL: http://arxiv.org/abs/2305.05766v1
- Date: Tue, 9 May 2023 20:59:14 GMT
- Title: Instant-NeRF: Instant On-Device Neural Radiance Field Training via
Algorithm-Accelerator Co-Designed Near-Memory Processing
- Authors: Yang Zhao, Shang Wu, Jingqun Zhang, Sixu Li, Chaojian Li, Yingyan Lin
- Abstract summary: Instant on-device Neural Radiance Fields (NeRFs) are in growing demand for unleashing the promise of immersive AR/VR experiences.
Our profiling analysis reveals a memory-bound inefficiency in NeRF training.
To tackle this inefficiency, near-memory processing (NMP) promises to be an effective solution.
We propose the first NMP framework, Instant-NeRF, dedicated to enabling instant on-device NeRF training.
- Score: 24.3997207834401
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Instant on-device Neural Radiance Fields (NeRFs) are in growing demand for
unleashing the promise of immersive AR/VR experiences, but are still limited by
their prohibitive training time. Our profiling analysis reveals a memory-bound
inefficiency in NeRF training. To tackle this inefficiency, near-memory
processing (NMP) promises to be an effective solution, but also faces
challenges due to the unique workloads of NeRFs, including the random hash
table lookup, random point processing sequence, and heterogeneous bottleneck
steps. Therefore, we propose the first NMP framework, Instant-NeRF, dedicated
to enabling instant on-device NeRF training. Experiments on eight datasets
consistently validate the effectiveness of Instant-NeRF.
Related papers
- Spatial Annealing Smoothing for Efficient Few-shot Neural Rendering [106.0057551634008]
We introduce an accurate and efficient few-shot neural rendering method named Spatial Annealing smoothing regularized NeRF (SANeRF)
By adding merely one line of code, SANeRF delivers superior rendering quality and much faster reconstruction speed compared to current few-shot NeRF methods.
arXiv Detail & Related papers (2024-06-12T02:48:52Z) - Deep Learning for Low-Latency, Quantum-Ready RF Sensing [2.5393702482222813]
Recent work has shown the promise of applying deep learning to enhance software processing of radio frequency (RF) signals.
In this paper, we describe our implementations of quantum-ready machine learning approaches for RF signal classification.
arXiv Detail & Related papers (2024-04-27T17:22:12Z) - Prompt2NeRF-PIL: Fast NeRF Generation via Pretrained Implicit Latent [61.56387277538849]
This paper explores promptable NeRF generation for direct conditioning and fast generation of NeRF parameters for the underlying 3D scenes.
Prompt2NeRF-PIL is capable of generating a variety of 3D objects with a single forward pass.
We will show that our approach speeds up the text-to-NeRF model DreamFusion and the 3D reconstruction speed of the image-to-NeRF method Zero-1-to-3 by 3 to 5 times.
arXiv Detail & Related papers (2023-12-05T08:32:46Z) - Efficient View Synthesis with Neural Radiance Distribution Field [61.22920276806721]
We propose a new representation called Neural Radiance Distribution Field (NeRDF) that targets efficient view synthesis in real-time.
We use a small network similar to NeRF while preserving the rendering speed with a single network forwarding per pixel as in NeLF.
Experiments show that our proposed method offers a better trade-off among speed, quality, and network size than existing methods.
arXiv Detail & Related papers (2023-08-22T02:23:28Z) - MF-NeRF: Memory Efficient NeRF with Mixed-Feature Hash Table [62.164549651134465]
We propose MF-NeRF, a memory-efficient NeRF framework that employs a Mixed-Feature hash table to improve memory efficiency and reduce training time while maintaining reconstruction quality.
Our experiments with state-of-the-art Instant-NGP, TensoRF, and DVGO, indicate our MF-NeRF could achieve the fastest training time on the same GPU hardware with similar or even higher reconstruction quality.
arXiv Detail & Related papers (2023-04-25T05:44:50Z) - Re-ReND: Real-time Rendering of NeRFs across Devices [56.081995086924216]
Re-ReND is designed to achieve real-time performance by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines.
We find that Re-ReND can achieve over a 2.6-fold increase in rendering speed versus the state-of-the-art without perceptible losses in quality.
arXiv Detail & Related papers (2023-03-15T15:59:41Z) - FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency
Regularization [32.1581416980828]
We present Frequency regularized NeRF (FreeNeRF), a surprisingly simple baseline that outperforms previous methods.
We analyze the key challenges in few-shot neural rendering and find that frequency plays an important role in NeRF's training.
arXiv Detail & Related papers (2023-03-13T18:59:03Z) - MEIL-NeRF: Memory-Efficient Incremental Learning of Neural Radiance
Fields [49.68916478541697]
We develop a Memory-Efficient Incremental Learning algorithm for NeRF (MEIL-NeRF)
MEIL-NeRF takes inspiration from NeRF itself in that a neural network can serve as a memory that provides the pixel RGB values, given rays as queries.
As a result, MEIL-NeRF demonstrates constant memory consumption and competitive performance.
arXiv Detail & Related papers (2022-12-16T08:04:56Z) - Compressing Explicit Voxel Grid Representations: fast NeRFs become also
small [3.1473798197405944]
Re:NeRF aims to reduce memory storage of NeRF models while maintaining comparable performance.
We benchmark our approach with three different EVG-NeRF architectures on four popular benchmarks.
arXiv Detail & Related papers (2022-10-23T16:42:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.