Real-Time Neural Materials using Block-Compressed Features

• URL: http://arxiv.org/abs/2311.16121v2
• Date: Sun, 18 Feb 2024 02:41:26 GMT
• Title: Real-Time Neural Materials using Block-Compressed Features
• Abstract summary: We present a neural material model whose features and decoder are specifically designed to be used in real-time rendering pipelines. Our framework leverages hardware-based block compression (BC) texture formats to store the learned features and trains the model to output the material information continuously in space and scale.
• Score: 0.8520624117635328
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural materials typically consist of a collection of neural features along with a decoder network. The main challenge in integrating such models in real-time rendering pipelines lies in the large size required to store their features in GPU memory and the complexity of evaluating the network efficiently. We present a neural material model whose features and decoder are specifically designed to be used in real-time rendering pipelines. Our framework leverages hardware-based block compression (BC) texture formats to store the learned features and trains the model to output the material information continuously in space and scale. To achieve this, we organize the features in a block-based manner and emulate BC6 decompression during training, making it possible to export them as regular BC6 textures. This structure allows us to use high resolution features while maintaining a low memory footprint. Consequently, this enhances our model's overall capability, enabling the use of a lightweight and simple decoder architecture that can be evaluated directly in a shader. Furthermore, since the learned features can be decoded continuously, it allows for random uv sampling and smooth transition between scales without needing any subsequent filtering. As a result, our neural material has a small memory footprint, can be decoded extremely fast adding a minimal computational overhead to the rendering pipeline.

Related papers

• Neural Texture Compression using Hypernetworks [0.0]
  It is possible to learn small, per-material texture representations that can be decoded in real-time during shading.<n>We train a single hypernetwork that outputs both the latent features and the shading's weights and biases.<n>We extend this approach to infer multiple decoders at once or even produce decoders that learn super-resolution.
  arXiv  Detail & Related papers  (2026-06-25T11:52:11Z)
• Towards Practical Lossless Neural Compression for LiDAR Point Clouds [84.36825469211375]
  extreme sparsity of high-precision geometric details hinders efficient context modeling.<n>Our framework comprises two lightweight modules.<n>Experiments demonstrate competitive compression performance at real-time speed.
  arXiv  Detail & Related papers  (2026-03-26T10:02:07Z)
• Quantile Rendering: Efficiently Embedding High-dimensional Feature on 3D Gaussian Splatting [52.18697134979677]
  Recent advancements in computer vision have successfully extended Open-vocabulary segmentation (OVS) to the 3D domain by leveraging 3D Gaussian Splatting (3D-GS)<n>Existing methods employ codebooks or feature compression, causing information loss, thereby degrading segmentation quality.<n>We introduce Quantile Rendering (Q-Render), a novel rendering strategy for 3D Gaussians that efficiently handles high-dimensional features while maintaining high fidelity.<n>Our framework outperforms state-of-the-art methods, while enabling real-time rendering with an approximate 43.7x speedup on 512-D feature maps.
  arXiv  Detail & Related papers  (2025-12-24T04:16:18Z)
• Re-Densification Meets Cross-Scale Propagation: Real-Time Neural Compression of LiDAR Point Clouds [83.39320394656855]
  LiDAR point clouds are fundamental to various applications, yet high-precision scans incur substantial storage and transmission overhead.<n>Existing methods typically convert unordered points into hierarchical octree or voxel structures for dense-to-sparse predictive coding.<n>Our framework comprises two lightweight modules. First, the Geometry Re-Densification Module re-densifies encoded sparse geometry, extracts features at denser scale, and then re-sparsifies the features for predictive coding.
  arXiv  Detail & Related papers  (2025-08-28T06:36:10Z)
• LODGE: Level-of-Detail Large-Scale Gaussian Splatting with Efficient Rendering [68.93333348474988]
  We present a novel level-of-detail (LOD) method for 3D Gaussian Splatting on memory-constrained devices.<n>Our approach iteratively selects optimal subsets of Gaussians based on camera distance.<n>Our method achieves state-of-the-art performance on both outdoor (Hierarchical 3DGS) and indoor (Zip-NeRF) datasets.
  arXiv  Detail & Related papers  (2025-05-29T06:50:57Z)
• Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB [29.267039546199094]
  This paper aims to encode a 3D scene into an extremely compact representation from 2D images. It enables its transmittance, decoding and rendering in real-time across various platforms.
  arXiv  Detail & Related papers  (2024-09-24T03:06:22Z)
• N-BVH: Neural ray queries with bounding volume hierarchies [51.430495562430565]
  In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures. We devise N-BVH, a neural compression architecture designed to answer arbitrary ray queries in 3D. Our method provides faithful approximations of visibility, depth, and appearance attributes.
  arXiv  Detail & Related papers  (2024-05-25T13:54:34Z)
• Neural Graphics Texture Compression Supporting Random Access [34.974631096947284]
  We introduce a novel approach to texture set compression that integrates traditional GPU texture representation and NIC techniques. We propose an asymmetric auto-encoder framework that employs a convolutional encoder to capture detailed information in a bottleneck-latent space. Experimental results demonstrate that this approach provides much better results than conventional texture compression.
  arXiv  Detail & Related papers  (2024-05-06T19:44:13Z)
• NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation [22.151167286623416]
  We propose an end-to-end NeRF compression framework that integrates non-linear transform, quantization, and entropy coding for memory-efficient scene representation. We demonstrate our method outperforms existing NeRF compression methods, enabling high-quality novel view synthesis with a memory budget of 0.5 MB.
  arXiv  Detail & Related papers  (2024-04-02T15:49:00Z)
• Volume Feature Rendering for Fast Neural Radiance Field Reconstruction [11.05302598034426]
  Neural radiance fields (NeRFs) are able to synthesize realistic novel views from multi-view images captured from distinct positions and perspectives. In NeRF's rendering pipeline, neural networks are used to represent a scene independently or transform queried learnable feature vector of a point to the expected color or density. We propose to render the queried feature vectors of a ray first and then transform the rendered feature vector to the final pixel color by a neural network.
  arXiv  Detail & Related papers  (2023-05-29T06:58:27Z)
• Variable Bitrate Neural Fields [75.24672452527795]
  We present a dictionary method for compressing feature grids, reducing their memory consumption by up to 100x. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available.
  arXiv  Detail & Related papers  (2022-06-15T17:58:34Z)
• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding [67.33850633281803]
  We present a versatile new input encoding that permits the use of a smaller network without sacrificing quality. A small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through a gradient descent. We achieve a combined speed of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds.
  arXiv  Detail & Related papers  (2022-01-16T07:22:47Z)
• Small Lesion Segmentation in Brain MRIs with Subpixel Embedding [105.1223735549524]
  We present a method to segment MRI scans of the human brain into ischemic stroke lesion and normal tissues. We propose a neural network architecture in the form of a standard encoder-decoder where predictions are guided by a spatial expansion embedding network.
  arXiv  Detail & Related papers  (2021-09-18T00:21:17Z)
• HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation [95.47168925127089]
  We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. We design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features.
  arXiv  Detail & Related papers  (2020-12-21T18:58:18Z)
• Neural Network Compression Framework for fast model inference [59.65531492759006]
  We present a new framework for neural networks compression with fine-tuning, which we called Neural Network Compression Framework (NNCF) It leverages recent advances of various network compression methods and implements some of them, such as sparsity, quantization, and binarization. The framework can be used within the training samples, which are supplied with it, or as a standalone package that can be seamlessly integrated into the existing training code.
  arXiv  Detail & Related papers  (2020-02-20T11:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.