Towards Real-Time Neural Volumetric Rendering on Mobile Devices: A Measurement Study

• URL: http://arxiv.org/abs/2406.16068v1
• Date: Sun, 23 Jun 2024 10:33:26 GMT
• Title: Towards Real-Time Neural Volumetric Rendering on Mobile Devices: A Measurement Study
• Authors: Zhe Wang, Yifei Zhu,
• Abstract summary: We take the first initiative to examine the state-of-the-art real-time NeRF rendering technique from a system perspective. We first define the entire working pipeline of the NeRF serving system. We then identify possible control knobs that are critical to the system from the communication, computation, and visual performance perspective.
• Score: 12.392923990003753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) is an emerging technique to synthesize 3D objects from 2D images with a wide range of potential applications. However, rendering existing NeRF models is extremely computation intensive, making it challenging to support real-time interaction on mobile devices. In this paper, we take the first initiative to examine the state-of-the-art real-time NeRF rendering technique from a system perspective. We first define the entire working pipeline of the NeRF serving system. We then identify possible control knobs that are critical to the system from the communication, computation, and visual performance perspective. Furthermore, an extensive measurement study is conducted to reveal the effects of these control knobs on system performance. Our measurement results reveal that different control knobs contribute differently towards improving the system performance, with the mesh granularity being the most effective knob and the quantization being the least effective knob. In addition, diverse hardware device settings and network conditions have to be considered to fully unleash the benefit of operating under the appropriate knobs

Related papers

• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• RS-NeRF: Neural Radiance Fields from Rolling Shutter Images [30.719764073204423]
  We present RS-NeRF, a method designed to synthesize normal images from novel views using input with RS distortions. This involves a physical model that replicates the image formation process under RS conditions. We further address the inherent shortcomings of the basic RS-NeRF model by delving into the RS characteristics and developing algorithms to enhance its functionality.
  arXiv  Detail & Related papers  (2024-07-14T16:27:11Z)
• NeRF-DetS: Enhancing Multi-View 3D Object Detection with Sampling-adaptive Network of Continuous NeRF-based Representation [60.47114985993196]
  NeRF-Det unifies the tasks of novel view arithmetic and 3D perception. We introduce a novel 3D perception network structure, NeRF-DetS. NeRF-DetS outperforms competitive NeRF-Det on the ScanNetV2 dataset.
  arXiv  Detail & Related papers  (2024-04-22T06:59:03Z)
• 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands [51.305421495638434]
  Neural radiance fields (NeRFs) are promising 3D representations for scenes, objects, and humans. This paper proposes a generalizable visibility-aware NeRF framework for interacting hands. Experiments on the Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms conventional NeRFs significantly.
  arXiv  Detail & Related papers  (2024-01-02T00:42:06Z)
• Agile gesture recognition for capacitive sensing devices: adapting on-the-job [55.40855017016652]
  We demonstrate a hand gesture recognition system that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer five fingers. We use a machine learning technique to analyse the time series signals and identify three features that can represent 5 fingers within 500 ms.
  arXiv  Detail & Related papers  (2023-05-12T17:24:02Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• LGC-Net: A Lightweight Gyroscope Calibration Network for Efficient Attitude Estimation [10.468378902106613]
  We present a calibration neural network model for denoising low-cost microelectromechanical system (MEMS) gyroscope and estimating the attitude of a robot in real-time. Key idea is extracting local and global features from the time window of inertial measurement units (IMU) measurements to regress the output compensation components for the gyroscope dynamically. The proposed algorithm is evaluated in the EuRoC and TUM-VI datasets and achieves state-of-the-art on the (unseen) test sequences with a more lightweight model structure.
  arXiv  Detail & Related papers  (2022-09-19T08:03:03Z)
• NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields [54.27264716713327]
  We show that a Neural Radiance Fields (NeRF) representation of a scene can be used to train dense object descriptors. We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object. Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106%.
  arXiv  Detail & Related papers  (2022-03-03T18:49:57Z)
• Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series [2.7793394375935088]
  High-Frequency (HF) signal are ubiquitous in the industrial world and are of great use for the monitoring of industrial assets. Most deep learning tools are designed for inputs of fixed and/or very limited size and successful applications of deep learning to the industrial context use as inputs extracted features. We propose a fully unsupervised deep learning framework that is able to extract meaningful and sparse representation of raw HF signals.
  arXiv  Detail & Related papers  (2021-05-03T14:35:06Z)
• A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data [3.4806267677524896]
  We propose a method to deploy state of the art neural networks for real time 3D object localization on augmented reality devices. We focus on fast 2D detection approaches which are extracting the 3D pose of the object fast and accurately by using only 2D input. For the 6D annotation of 2D images, we developed an annotation tool, which is, to our knowledge, the first open source tool to be available.
  arXiv  Detail & Related papers  (2020-01-16T09:13:31Z)
• Lightweight Residual Densely Connected Convolutional Neural Network [18.310331378001397]
  The lightweight residual densely connected blocks are proposed to guaranty the deep supervision, efficient gradient flow, and feature reuse abilities of convolutional neural network. The proposed method decreases the cost of training and inference processes without using any special hardware-software equipment.
  arXiv  Detail & Related papers  (2020-01-02T17:15:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.