CUDA-Optimized real-time rendering of a Foveated Visual System
- URL: http://arxiv.org/abs/2012.08655v1
- Date: Tue, 15 Dec 2020 22:43:04 GMT
- Title: CUDA-Optimized real-time rendering of a Foveated Visual System
- Authors: Elian Malkin, Arturo Deza, Tomaso Poggio
- Abstract summary: We present a technique that exploits the GPU to efficiently generate Gaussian-based foveated images at high definition (1920x1080) in real-time (165 Hz)
Our algorithm can meet demand for spatially-varying processing across biological artificial agents so that foveation can be added easily on top of existing systems.
- Score: 5.260841516691153
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The spatially-varying field of the human visual system has recently received
a resurgence of interest with the development of virtual reality (VR) and
neural networks. The computational demands of high resolution rendering desired
for VR can be offset by savings in the periphery, while neural networks trained
with foveated input have shown perceptual gains in i.i.d and o.o.d
generalization. In this paper, we present a technique that exploits the CUDA
GPU architecture to efficiently generate Gaussian-based foveated images at high
definition (1920x1080 px) in real-time (165 Hz), with a larger number of
pooling regions than previous Gaussian-based foveation algorithms by several
orders of magnitude, producing a smoothly foveated image that requires no
further blending or stitching, and that can be well fit for any contrast
sensitivity function. The approach described can be adapted from Gaussian
blurring to any eccentricity-dependent image processing and our algorithm can
meet demand for experimentation to evaluate the role of spatially-varying
processing across biological and artificial agents, so that foveation can be
added easily on top of existing systems rather than forcing their redesign
(emulated foveated renderer). Altogether, this paper demonstrates how a GPU,
with a CUDA block-wise architecture, can be employed for radially-variant
rendering, with opportunities for more complex post-processing to ensure a
metameric foveation scheme. Code is provided.
Related papers
- Image-GS: Content-Adaptive Image Representation via 2D Gaussians [55.15950594752051]
We propose Image-GS, a content-adaptive image representation.
Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack.
General efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors.
We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.
arXiv Detail & Related papers (2024-07-02T00:45:21Z) - N-BVH: Neural ray queries with bounding volume hierarchies [51.430495562430565]
In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures.
We devise N-BVH, a neural compression architecture designed to answer arbitrary ray queries in 3D.
Our method provides faithful approximations of visibility, depth, and appearance attributes.
arXiv Detail & Related papers (2024-05-25T13:54:34Z) - Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple
Scale Neural Radiance Field Rendering [3.8200916793910973]
Recent advances in Neural Radiance Fields (NeRF) have demonstrated significant potential for representing 3D scene appearances as implicit neural networks.
However, the lengthy training and rendering process hinders the widespread adoption of this promising technique for real-time rendering applications.
We present an effective adaptive multi-NeRF method designed to accelerate the neural rendering process for large scenes.
arXiv Detail & Related papers (2023-10-03T08:34:49Z) - Online Neural Path Guiding with Normalized Anisotropic Spherical
Gaussians [20.68953631807367]
We propose a novel online framework to learn the spatial-varying density model with a single small neural network.
Our framework learns the distribution in a progressive manner and does not need any warm-up phases.
arXiv Detail & Related papers (2023-03-11T05:22:42Z) - Convolutional Neural Generative Coding: Scaling Predictive Coding to
Natural Images [79.07468367923619]
We develop convolutional neural generative coding (Conv-NGC)
We implement a flexible neurobiologically-motivated algorithm that progressively refines latent state maps.
We study the effectiveness of our brain-inspired neural system on the tasks of reconstruction and image denoising.
arXiv Detail & Related papers (2022-11-22T06:42:41Z) - A Novel Algorithm for Exact Concave Hull Extraction [0.0]
Region extraction is necessary in a wide range of applications, from object detection in autonomous driving to analysis of subcellular morphology in cell biology.
There exist two main approaches: convex hull extraction, for which exact and efficient algorithms exist and concave hulls, which are better at capturing real-world shapes but do not have a single solution.
In this study, we present a novel algorithm that can provide concave hulls with maximal (i.e. pixel-perfect) resolution and is tunable for speed-efficiency tradeoffs.
arXiv Detail & Related papers (2022-06-23T05:26:48Z) - Image-specific Convolutional Kernel Modulation for Single Image
Super-resolution [85.09413241502209]
In this issue, we propose a novel image-specific convolutional modulation kernel (IKM)
We exploit the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels.
Experiments on single image super-resolution show that the proposed methods achieve superior performances over state-of-the-art methods.
arXiv Detail & Related papers (2021-11-16T11:05:10Z) - An Energy-Efficient Edge Computing Paradigm for Convolution-based Image
Upsampling [0.0]
A novel energy-efficient edge computing paradigm is proposed for real-time deep learning-based image upsampling applications.
We transform learned convolution kernels to deconvolution kernels before deploying them for inference as a functionally equivalent deconvolution.
arXiv Detail & Related papers (2021-07-15T23:49:37Z) - AutoInt: Automatic Integration for Fast Neural Volume Rendering [51.46232518888791]
We propose a new framework for learning efficient, closed-form solutions to integrals using implicit neural representation networks.
We demonstrate a greater than 10x improvement in photorealistic requirements, enabling fast neural volume rendering.
arXiv Detail & Related papers (2020-12-03T05:46:10Z) - A Flexible Framework for Designing Trainable Priors with Adaptive
Smoothing and Game Encoding [57.1077544780653]
We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems.
We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions.
This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
arXiv Detail & Related papers (2020-06-26T08:34:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.