VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for
Analysis-by-Synthesis
- URL: http://arxiv.org/abs/2205.15401v3
- Date: Sun, 28 Jan 2024 20:25:55 GMT
- Title: VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for
Analysis-by-Synthesis
- Authors: Angtian Wang, Peng Wang, Jian Sun, Adam Kortylewski, Alan Yuille
- Abstract summary: We propose VoGE, which utilizes the Gaussian reconstruction kernels as volumetric primitives.
To efficiently render via VoGE, we propose an approximate closeform solution for the volume density aggregation and a coarse-to-fine rendering strategy.
VoGE outperforms SoTA when applied to various vision tasks, e.g., object pose estimation, shape/texture fitting, and reasoning.
- Score: 62.47221232706105
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The Gaussian reconstruction kernels have been proposed by Westover (1990) and
studied by the computer graphics community back in the 90s, which gives an
alternative representation of object 3D geometry from meshes and point clouds.
On the other hand, current state-of-the-art (SoTA) differentiable renderers,
Liu et al. (2019), use rasterization to collect triangles or points on each
image pixel and blend them based on the viewing distance. In this paper, we
propose VoGE, which utilizes the volumetric Gaussian reconstruction kernels as
geometric primitives. The VoGE rendering pipeline uses ray tracing to capture
the nearest primitives and blends them as mixtures based on their volume
density distributions along the rays. To efficiently render via VoGE, we
propose an approximate closeform solution for the volume density aggregation
and a coarse-to-fine rendering strategy. Finally, we provide a CUDA
implementation of VoGE, which enables real-time level rendering with a
competitive rendering speed in comparison to PyTorch3D. Quantitative and
qualitative experiment results show VoGE outperforms SoTA counterparts when
applied to various vision tasks, e.g., object pose estimation, shape/texture
fitting, and occlusion reasoning. The VoGE library and demos are available at:
https://github.com/Angtian/VoGE.
Related papers
- PFGS: High Fidelity Point Cloud Rendering via Feature Splatting [5.866747029417274]
We propose a novel framework to render high-quality images from sparse points.
This method first attempts to bridge the 3D Gaussian Splatting and point cloud rendering.
Experiments on different benchmarks show the superiority of our method in terms of rendering qualities and the necessities of our main components.
arXiv Detail & Related papers (2024-07-04T11:42:54Z) - Bridging 3D Gaussian and Mesh for Freeview Video Rendering [57.21847030980905]
GauMesh bridges the 3D Gaussian and Mesh for modeling and rendering the dynamic scenes.
We show that our approach adapts the appropriate type of primitives to represent the different parts of the dynamic scene.
arXiv Detail & Related papers (2024-03-18T04:01:26Z) - Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information.
We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering.
Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
arXiv Detail & Related papers (2023-04-20T04:05:22Z) - Multi-View Reconstruction using Signed Ray Distance Functions (SRDF) [22.75986869918975]
We investigate a new computational approach that builds on a novel shape representation that is volumetric.
The shape energy associated to this representation evaluates 3D geometry given color images and does not need appearance prediction.
In practice we propose an implicit shape representation, the SRDF, based on signed distances which we parameterize by depths along camera rays.
arXiv Detail & Related papers (2022-08-31T19:32:17Z) - Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
We present an efficient method for joint optimization of materials and lighting from multi-view image observations.
We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
arXiv Detail & Related papers (2021-11-24T13:58:20Z) - Volume Rendering of Neural Implicit Surfaces [57.802056954935495]
This paper aims to improve geometry representation and reconstruction in neural volume rendering.
We achieve that by modeling the volume density as a function of the geometry.
Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions.
arXiv Detail & Related papers (2021-06-22T20:23:16Z) - Geometric Correspondence Fields: Learned Differentiable Rendering for 3D
Pose Refinement in the Wild [96.09941542587865]
We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild.
In this way, we precisely align 3D models to objects in RGB images which results in significantly improved 3D pose estimates.
We evaluate our approach on the challenging Pix3D dataset and achieve up to 55% relative improvement compared to state-of-the-art refinement methods in multiple metrics.
arXiv Detail & Related papers (2020-07-17T12:34:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.