Freq-Mip-AA : Frequency Mip Representation for Anti-Aliasing Neural Radiance Fields

• URL: http://arxiv.org/abs/2406.13251v1
• Date: Wed, 19 Jun 2024 06:33:56 GMT
• Title: Freq-Mip-AA : Frequency Mip Representation for Anti-Aliasing Neural Radiance Fields
• Authors: Youngin Park, Seungtae Nam, Cheul-hee Hahm, Eunbyung Park,
• Abstract summary: Mip-NeRF proposed using frustums to render a pixel and suggested integrated positional encoding (IPE) While effective, this approach requires long training times due to its reliance on volumetric architecture. We propose a novel anti-aliasing technique that utilizes grid-based representations, usually showing significantly faster training time.
• Score: 3.796287987989994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) have shown remarkable success in representing 3D scenes and generating novel views. However, they often struggle with aliasing artifacts, especially when rendering images from different camera distances from the training views. To address the issue, Mip-NeRF proposed using volumetric frustums to render a pixel and suggested integrated positional encoding (IPE). While effective, this approach requires long training times due to its reliance on MLP architecture. In this work, we propose a novel anti-aliasing technique that utilizes grid-based representations, usually showing significantly faster training time. In addition, we exploit frequency-domain representation to handle the aliasing problem inspired by the sampling theorem. The proposed method, FreqMipAA, utilizes scale-specific low-pass filtering (LPF) and learnable frequency masks. Scale-specific low-pass filters (LPF) prevent aliasing and prioritize important image details, and learnable masks effectively remove problematic high-frequency elements while retaining essential information. By employing a scale-specific LPF and trainable masks, FreqMipAA can effectively eliminate the aliasing factor while retaining important details. We validated the proposed technique by incorporating it into a widely used grid-based method. The experimental results have shown that the FreqMipAA effectively resolved the aliasing issues and achieved state-of-the-art results in the multi-scale Blender dataset. Our code is available at https://github.com/yi0109/FreqMipAA .

Related papers

• When Semantic Segmentation Meets Frequency Aliasing [14.066404173580864]
  We conduct a comprehensive analysis of hard pixel errors, categorizing them into three types: false responses, merging mistakes, and displacements. Our findings reveal a quantitative association between hard pixels and aliasing, which is distortion caused by the overlapping of frequency components in the Fourier domain during downsampling. Here, we propose two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing by accurately removing or adjusting frequencies higher than the Nyquist frequency.
  arXiv  Detail & Related papers  (2024-03-14T03:12:02Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields [12.910072009005065]
  We present mip-blur, a novel approach that integrates anti-aliasing techniques into grid-based representations for radiance fields. The proposed method generates multi-scale grids by applying simple convolution operations over a shared grid representation and uses the scale coordinate to retrieve features at different scales from the generated multi-scale grids.
  arXiv  Detail & Related papers  (2024-02-22T00:45:40Z)
• PyNeRF: Pyramidal Neural Radiance Fields [51.25406129834537]
  We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. At render time, we simply use coarser grids to render samples that cover larger volumes. Compared to Mip-NeRF, we reduce error rates by 20% while training over 60x faster.
  arXiv  Detail & Related papers  (2023-11-30T23:52:46Z)
• Neural Fields with Thermal Activations for Arbitrary-Scale Super-Resolution [56.089473862929886]
  We present a novel way to design neural fields such that points can be queried with an adaptive Gaussian PSF. With its theoretically guaranteed anti-aliasing, our method sets a new state of the art for arbitrary-scale single image super-resolution.
  arXiv  Detail & Related papers  (2023-11-29T14:01:28Z)
• Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling [11.72025865314187]
  Convolutional neural networks encode images through a sequence of convolutions, normalizations and non-linearities as well as downsampling operations. Previous work showed that even slight mistakes during sampling, leading to aliasing, can be directly attributed to the networks' lack in robustness. We propose aliasing and spectral artifact-free pooling, short ASAP.
  arXiv  Detail & Related papers  (2023-07-19T07:47:23Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields [64.13207562222094]
  We show how a technique that combines mip-NeRF 360 and grid-based models can yield error rates that are 8% - 77% lower than either prior technique, and that trains 24x faster than mip-NeRF 360.
  arXiv  Detail & Related papers  (2023-04-13T17:55:12Z)
• Re-ReND: Real-time Rendering of NeRFs across Devices [56.081995086924216]
  Re-ReND is designed to achieve real-time performance by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines. We find that Re-ReND can achieve over a 2.6-fold increase in rendering speed versus the state-of-the-art without perceptible losses in quality.
  arXiv  Detail & Related papers  (2023-03-15T15:59:41Z)
• Low Pass Filter for Anti-aliasing in Temporal Action Localization [15.139834271977913]
  This paper aims to verify the existence of aliasing in temporal action localization methods. It investigates utilizing low pass filters to solve this problem by inhibiting the high-frequency band. Experiments demonstrate that anti-aliasing with low pass filters in TAL is advantageous and efficient.
  arXiv  Detail & Related papers  (2021-04-23T03:57:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.