Voxel-Mesh Hybrid Representation for Real-Time View Synthesis

• URL: http://arxiv.org/abs/2403.06505v2
• Date: Wed, 20 Nov 2024 08:11:36 GMT
• Title: Voxel-Mesh Hybrid Representation for Real-Time View Synthesis
• Authors: Chenhao Zhang, Yongyang Zhou, Lei Zhang,
• Abstract summary: We propose a hybrid representation named Vosh, seamlessly combining both voxel and mesh components in hybrid rendering for view synthesis. Vosh excels in fast rendering scenes with simple geometry and textures through its mesh component, while simultaneously enabling high-quality rendering in intricate regions by leveraging voxel component. The flexibility of Vosh is showcased through the ability to adjust hybrid ratios, providing users the ability to control the balance between rendering quality and speed based on flexible usage.
• Score: 5.528874948395173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The neural radiance fields (NeRF) have emerged as a prominent methodology for synthesizing realistic images of novel views. While neural radiance representations based on voxels or mesh individually offer distinct advantages, excelling in either rendering quality or speed, each has limitations in the other aspect. In response, we propose a hybrid representation named Vosh, seamlessly combining both voxel and mesh components in hybrid rendering for view synthesis. Vosh is meticulously crafted by optimizing the voxel grid based on neural rendering, strategically meshing a portion of the volumetric density field to surface. Therefore, it excels in fast rendering scenes with simple geometry and textures through its mesh component, while simultaneously enabling high-quality rendering in intricate regions by leveraging voxel component. The flexibility of Vosh is showcased through the ability to adjust hybrid ratios, providing users the ability to control the balance between rendering quality and speed based on flexible usage. Experimental results demonstrate that our method achieves commendable trade-off between rendering quality and speed, and notably has real-time performance on mobile devices. The interactive web demo and code are available at https://zyyzyy06.github.io/Vosh.

Related papers

• Radiance Meshes for Volumetric Reconstruction [56.51690637804858]
  We introduce radiance meshes, a technique for representing radiance fields with constant density tetrahedral cells.<n>Our model is able to perform exact and fast volume rendering using both synthesisization and ray-tracing.<n>Our rendering method exactly evaluates the volume equation and enables high quality, real-time view on standard consumer hardware.
  arXiv  Detail & Related papers  (2025-12-03T18:57:03Z)
• UniVoxel: Fast Inverse Rendering by Unified Voxelization of Scene Representation [66.95976870627064]
  We design a Unified Voxelization framework for explicit learning of scene representations, dubbed UniVoxel. We propose to encode a scene into a latent volumetric representation, based on which the geometry, materials and illumination can be readily learned via lightweight neural networks. Experiments show that UniVoxel boosts the optimization efficiency significantly compared to other methods, reducing the per-scene training time from hours to 18 minutes, while achieving favorable reconstruction quality.
  arXiv  Detail & Related papers  (2024-07-28T17:24:14Z)
• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces [71.1071688018433]
  Neural radiance fields provide state-of-the-art view synthesis quality but tend to be slow to render. We propose a method, HybridNeRF, that leverages the strengths of both representations by rendering most objects as surfaces. We improve error rates by 15-30% while achieving real-time framerates (at least 36 FPS) for virtual-reality resolutions (2Kx2K)
  arXiv  Detail & Related papers  (2023-12-05T22:04:49Z)
• VoxNeRF: Bridging Voxel Representation and Neural Radiance Fields for Enhanced Indoor View Synthesis [51.49008959209671]
  We introduce VoxNeRF, a novel approach that leverages volumetric representations to enhance the quality and efficiency of indoor view synthesis. We employ multi-resolution hash grids to adaptively capture spatial features, effectively managing occlusions and the intricate geometry of indoor scenes. We validate our approach against three public indoor datasets and demonstrate that VoxNeRF outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-09T11:32:49Z)
• Learning Neural Duplex Radiance Fields for Real-Time View Synthesis [33.54507228895688]
  We propose a novel approach to distill and bake NeRFs into highly efficient mesh-based neural representations. We demonstrate the effectiveness and superiority of our approach via extensive experiments on a range of standard datasets.
  arXiv  Detail & Related papers  (2023-04-20T17:59:52Z)
• VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesis [31.52438192597056]
  We present a hybrid volume-mesh representation, VMesh, which depicts an object with a textured mesh along with an auxiliary sparse volume. VMesh can be obtained from multi-view images of an object and renders at 2K 60FPS on common consumer devices with high fidelity.
  arXiv  Detail & Related papers  (2023-03-28T17:49:42Z)
• BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis [42.93055827628597]
  We present a method for reconstructing high-quality meshes of large real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians.
  arXiv  Detail & Related papers  (2023-02-28T18:58:03Z)
• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels [106.69049089979433]
  We propose a radiance field framework by representing scenes with time-aware voxel features, named as TiNeuVox. Our framework accelerates the optimization of dynamic radiance fields while maintaining high rendering quality. Our TiNeuVox completes training with only 8 minutes and 8-MB storage cost while showing similar or even better rendering performance than previous dynamic NeRF methods.
  arXiv  Detail & Related papers  (2022-05-30T17:47:31Z)
• Neural Sparse Voxel Fields [151.20366604586403]
  We introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results.
  arXiv  Detail & Related papers  (2020-07-22T17:51:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.