Expansive Supervision for Neural Radiance Field

• URL: http://arxiv.org/abs/2409.08056v1
• Date: Thu, 12 Sep 2024 14:05:13 GMT
• Title: Expansive Supervision for Neural Radiance Field
• Authors: Weixiang Zhang, Shuzhao Xie, Shijia Ge, Wei Yao, Chen Tang, Zhi Wang,
• Abstract summary: We introduce an expansive supervision mechanism that efficiently balances computational load, rendering quality and flexibility for neural radiance field training. Compared to conventional supervision, our method effectively bypasses redundant rendering processes, resulting in notable reductions in both time and memory consumption.
• Score: 12.510474224361504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields have achieved success in creating powerful 3D media representations with their exceptional reconstruction capabilities. However, the computational demands of volume rendering pose significant challenges during model training. Existing acceleration techniques often involve redesigning the model architecture, leading to limitations in compatibility across different frameworks. Furthermore, these methods tend to overlook the substantial memory costs incurred. In response to these challenges, we introduce an expansive supervision mechanism that efficiently balances computational load, rendering quality and flexibility for neural radiance field training. This mechanism operates by selectively rendering a small but crucial subset of pixels and expanding their values to estimate the error across the entire area for each iteration. Compare to conventional supervision, our method effectively bypasses redundant rendering processes, resulting in notable reductions in both time and memory consumption. Experimental results demonstrate that integrating expansive supervision within existing state-of-the-art acceleration frameworks can achieve 69% memory savings and 42% time savings, with negligible compromise in visual quality.

Related papers

• LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization [17.190984773586745]
  Current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. We propose efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance.
  arXiv  Detail & Related papers  (2024-11-26T07:32:36Z)
• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• Magnituder Layers for Implicit Neural Representations in 3D [23.135779936528333]
  We introduce a novel neural network layer called the "magnituder" By integrating magnituders into standard feed-forward layer stacks, we achieve improved inference speed and adaptability. Our approach enables a zero-shot performance boost in trained implicit neural representation models.
  arXiv  Detail & Related papers  (2024-10-13T08:06:41Z)
• Temporal Feature Matters: A Framework for Diffusion Model Quantization [105.3033493564844]
  Diffusion models rely on the time-step for the multi-round denoising. We introduce a novel quantization framework that includes three strategies. This framework preserves most of the temporal information and ensures high-quality end-to-end generation.
  arXiv  Detail & Related papers  (2024-07-28T17:46:15Z)
• Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network [0.6675733925327885]
  Existing methods for imputing missing values in remote sensing images fail to fully exploit auxiliary information. This paper proposes a deep learning-based novel approach called MS2 for reconstructing time-series remote sensing images.
  arXiv  Detail & Related papers  (2024-06-19T09:05:05Z)
• Spatial Annealing Smoothing for Efficient Few-shot Neural Rendering [106.0057551634008]
  We introduce an accurate and efficient few-shot neural rendering method named Spatial Annealing smoothing regularized NeRF (SANeRF) By adding merely one line of code, SANeRF delivers superior rendering quality and much faster reconstruction speed compared to current few-shot NeRF methods.
  arXiv  Detail & Related papers  (2024-06-12T02:48:52Z)
• Oblique-MERF: Revisiting and Improving MERF for Oblique Photography [32.864777068264665]
  We introduce an innovative adaptive occupancy plane optimized during the volume rendering process and a smoothness regularization term for view-dependent color to address these issues. Our approach, termed oblique-MERF, surpasses state-of-the-art real-time methods by approximately 0.7 dB, reduces VRAM usage by about 40%, and achieves higher rendering frame rates with more realistic rendering outcomes across most viewpoints.
  arXiv  Detail & Related papers  (2024-04-15T07:51:29Z)
• Efficient Diffusion Model for Image Restoration by Residual Shifting [63.02725947015132]
  This study proposes a novel and efficient diffusion model for image restoration. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Our method achieves superior or comparable performance to current state-of-the-art methods on three classical IR tasks.
  arXiv  Detail & Related papers  (2024-03-12T05:06:07Z)
• Compensation Sampling for Improved Convergence in Diffusion Models [12.311434647047427]
  Diffusion models achieve remarkable quality in image generation, but at a cost. Iterative denoising requires many time steps to produce high fidelity images. We argue that the denoising process is crucially limited by an accumulation of the reconstruction error due to an initial inaccurate reconstruction of the target data.
  arXiv  Detail & Related papers  (2023-12-11T10:39:01Z)
• Diffusion Models Without Attention [110.5623058129782]
  Diffusion State Space Model (DiffuSSM) is an architecture that supplants attention mechanisms with a more scalable state space model backbone. Our focus on FLOP-efficient architectures in diffusion training marks a significant step forward.
  arXiv  Detail & Related papers  (2023-11-30T05:15:35Z)
• VoxNeRF: Bridging Voxel Representation and Neural Radiance Fields for Enhanced Indoor View Synthesis [51.49008959209671]
  We introduce VoxNeRF, a novel approach that leverages volumetric representations to enhance the quality and efficiency of indoor view synthesis. We employ multi-resolution hash grids to adaptively capture spatial features, effectively managing occlusions and the intricate geometry of indoor scenes. We validate our approach against three public indoor datasets and demonstrate that VoxNeRF outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-09T11:32:49Z)
• VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations [25.88881764546414]
  VQ-NeRF is an efficient pipeline for enhancing implicit neural representations via vector quantization. We present an innovative multi-scale NeRF sampling scheme that concurrently optimize the NeRF model at both compressed and original scales. We incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions.
  arXiv  Detail & Related papers  (2023-10-23T01:41:38Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• Weakly Supervised Face and Whole Body Recognition in Turbulent Environments [2.2263723609685773]
  We propose a new weakly supervised framework that generates domain representations, aligning turbulent and pristine images into a common subspace. We also introduce a new tilt map estimator that predicts geometric distortions observed in turbulent images. Our method does not require synthesizing turbulent-free images or ground-truth paired images, and requires significantly fewer annotated samples.
  arXiv  Detail & Related papers  (2023-08-22T19:58:02Z)
• Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration [59.6021678234829]
  We propose a novel method to restore the intermediate features for two sparsely sampled and adjacent video frames. With the integration of our method, the efficiency of three commonly used baselines has been improved by over 50%, with a mere 0.5% reduction in recognition accuracy.
  arXiv  Detail & Related papers  (2023-07-27T13:52:42Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes [61.01853377661283]
  We present a Memory-Efficient Radiance Field representation that achieves real-time rendering of large-scale scenes in a browser. We introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection.
  arXiv  Detail & Related papers  (2023-02-23T18:59:07Z)
• UNeRF: Time and Memory Conscious U-Shaped Network for Training Neural Radiance Fields [16.826691448973367]
  Neural Radiance Fields (NeRFs) increase reconstruction detail for novel view synthesis and scene reconstruction. However, the increased resolution and model-free nature of such neural fields come at the cost of high training times and excessive memory requirements. We propose a method to exploit the redundancy of NeRF's sample-based computations by partially sharing evaluations across neighboring sample points.
  arXiv  Detail & Related papers  (2022-06-23T19:57:07Z)
• Investigating Tradeoffs in Real-World Video Super-Resolution [90.81396836308085]
  Real-world video super-resolution (VSR) models are often trained with diverse degradations to improve generalizability. To alleviate the first tradeoff, we propose a degradation scheme that reduces up to 40% of training time without sacrificing performance. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences.
  arXiv  Detail & Related papers  (2021-11-24T18:58:21Z)
• Monocular Real-Time Volumetric Performance Capture [28.481131687883256]
  We present the first approach to volumetric performance capture and novel-view rendering at real-time speed from monocular video. Our system reconstructs a fully textured 3D human from each frame by leveraging Pixel-Aligned Implicit Function (PIFu) We also introduce an Online Hard Example Mining (OHEM) technique that effectively suppresses failure modes due to the rare occurrence of challenging examples.
  arXiv  Detail & Related papers  (2020-07-28T04:45:13Z)
• Towards Practical Lipreading with Distilled and Efficient Models [57.41253104365274]
  Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. There is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. We propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5% and 46.6%, respectively using self-distillation.
  arXiv  Detail & Related papers  (2020-07-13T16:56:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.