Neural Rendering and Its Hardware Acceleration: A Review
- URL: http://arxiv.org/abs/2402.00028v1
- Date: Sat, 6 Jan 2024 07:57:11 GMT
- Title: Neural Rendering and Its Hardware Acceleration: A Review
- Authors: Xinkai Yan, Jieting Xu, Yuchi Huo, Hujun Bao
- Abstract summary: Neural rendering is a new image and video generation method based on deep learning.
In this paper, we review the technical connotation, main challenges, and research progress of neural rendering.
- Score: 39.6466512858213
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural rendering is a new image and video generation method based on deep
learning. It combines the deep learning model with the physical knowledge of
computer graphics, to obtain a controllable and realistic scene model, and
realize the control of scene attributes such as lighting, camera parameters,
posture and so on. On the one hand, neural rendering can not only make full use
of the advantages of deep learning to accelerate the traditional forward
rendering process, but also provide new solutions for specific tasks such as
inverse rendering and 3D reconstruction. On the other hand, the design of
innovative hardware structures that adapt to the neural rendering pipeline
breaks through the parallel computing and power consumption bottleneck of
existing graphics processors, which is expected to provide important support
for future key areas such as virtual and augmented reality, film and television
creation and digital entertainment, artificial intelligence and the metaverse.
In this paper, we review the technical connotation, main challenges, and
research progress of neural rendering. On this basis, we analyze the common
requirements of neural rendering pipeline for hardware acceleration and the
characteristics of the current hardware acceleration architecture, and then
discuss the design challenges of neural rendering processor architecture.
Finally, the future development trend of neural rendering processor
architecture is prospected.
Related papers
- Recent Trends in 3D Reconstruction of General Non-Rigid Scenes [104.07781871008186]
Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision.
It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications.
This state-of-the-art report (STAR) offers the reader a comprehensive summary of state-of-the-art techniques with monocular and multi-view inputs.
arXiv Detail & Related papers (2024-03-22T09:46:11Z) - Artificial intelligence optical hardware empowers high-resolution
hyperspectral video understanding at 1.2 Tb/s [53.91923493664551]
This work introduces a hardware-accelerated integrated optoelectronic platform for multidimensional video understanding in real-time.
The technology platform combines artificial intelligence hardware, processing information optically, with state-of-the-art machine vision networks.
Such performance surpasses the speed of the closest technologies with similar spectral resolution by three to four orders of magnitude.
arXiv Detail & Related papers (2023-12-17T07:51:38Z) - ENVIDR: Implicit Differentiable Renderer with Neural Environment
Lighting [9.145875902703345]
We introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections.
We first propose a novel neural with decomposed rendering to learn the interaction between surface and environment lighting.
We then propose an SDF-based neural surface model that leverages this learned neural to represent general scenes.
arXiv Detail & Related papers (2023-03-23T04:12:07Z) - FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks [33.489890950757975]
FoVolNet is a method to significantly increase the performance of volume data visualization.
We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network.
arXiv Detail & Related papers (2022-09-20T19:48:56Z) - Human Performance Modeling and Rendering via Neural Animated Mesh [40.25449482006199]
We bridge the traditional mesh with a new class of neural rendering.
In this paper, we present a novel approach for rendering human views from video.
We demonstrate our approach on various platforms, inserting virtual human performances into AR headsets.
arXiv Detail & Related papers (2022-09-18T03:58:00Z) - Efficient Neural Architecture Search with Performance Prediction [0.0]
We use a neural architecture search to find the best network architecture for the task at hand.
Existing NAS algorithms generally evaluate the fitness of a new architecture by fully training from scratch.
An end-to-end offline performance predictor is proposed to accelerate the evaluation of sampled architectures.
arXiv Detail & Related papers (2021-08-04T05:44:16Z) - Fast Training of Neural Lumigraph Representations using Meta Learning [109.92233234681319]
We develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time.
Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection.
We show that MetaNLR++ achieves similar or better photorealistic novel view synthesis results in a fraction of the time that competing methods require.
arXiv Detail & Related papers (2021-06-28T18:55:50Z) - Deep Direct Volume Rendering: Learning Visual Feature Mappings From
Exemplary Images [57.253447453301796]
We introduce Deep Direct Volume Rendering (DeepDVR), a generalization of Direct Volume Rendering (DVR) that allows for the integration of deep neural networks into the DVR algorithm.
We conceptualize the rendering in a latent color space, thus enabling the use of deep architectures to learn implicit mappings for feature extraction and classification.
Our generalization serves to derive novel volume rendering architectures that can be trained end-to-end directly from examples in image space.
arXiv Detail & Related papers (2021-06-09T23:03:00Z) - State of the Art on Neural Rendering [141.22760314536438]
We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs.
This report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence.
arXiv Detail & Related papers (2020-04-08T04:36:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.