Improving Robustness for Joint Optimization of Camera Poses and
Decomposed Low-Rank Tensorial Radiance Fields
- URL: http://arxiv.org/abs/2402.13252v1
- Date: Tue, 20 Feb 2024 18:59:02 GMT
- Title: Improving Robustness for Joint Optimization of Camera Poses and
Decomposed Low-Rank Tensorial Radiance Fields
- Authors: Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu
- Abstract summary: We propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor.
We also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask.
- Score: 26.4340697184666
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we propose an algorithm that allows joint refinement of camera
pose and scene geometry represented by decomposed low-rank tensor, using only
2D images as supervision. First, we conduct a pilot study based on a 1D signal
and relate our findings to 3D scenarios, where the naive joint pose
optimization on voxel-based NeRFs can easily lead to sub-optimal solutions.
Moreover, based on the analysis of the frequency spectrum, we propose to apply
convolutional Gaussian filters on 2D and 3D radiance fields for a
coarse-to-fine training schedule that enables joint camera pose optimization.
Leveraging the decomposition property in decomposed low-rank tensor, our method
achieves an equivalent effect to brute-force 3D convolution with only incurring
little computational overhead. To further improve the robustness and stability
of joint optimization, we also propose techniques of smoothed 2D supervision,
randomly scaled kernel parameters, and edge-guided loss mask. Extensive
quantitative and qualitative evaluations demonstrate that our proposed
framework achieves superior performance in novel view synthesis as well as
rapid convergence for optimization.
Related papers
- GaussianRoom: Improving 3D Gaussian Splatting with SDF Guidance and Monocular Cues for Indoor Scene Reconstruction [3.043712258792239]
We present a unified framework integrating neural SDF with 3DGS.
This framework incorporates a learnable neural SDF field to guide the densification and pruning of Gaussians.
Our method achieves state-of-the-art performance in both surface reconstruction and novel view synthesis.
arXiv Detail & Related papers (2024-05-30T03:46:59Z) - 2D Gaussian Splatting for Geometrically Accurate Radiance Fields [50.056790168812114]
3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking.
We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images.
We demonstrate that our differentiable terms allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.
arXiv Detail & Related papers (2024-03-26T17:21:24Z) - GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time [112.32349668385635]
GGRt is a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses.
As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $ge$ 5 FPS and real-time rendering at $ge$ 100 FPS.
arXiv Detail & Related papers (2024-03-15T09:47:35Z) - GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis [70.24111297192057]
We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner.
The proposed method enables 2K-resolution rendering under a sparse-view camera setting.
arXiv Detail & Related papers (2023-12-04T18:59:55Z) - Mip-Splatting: Alias-free 3D Gaussian Splatting [52.366815964832426]
3D Gaussian Splatting has demonstrated impressive novel view synthesis results, reaching high fidelity and efficiency.
Strong artifacts can be observed when changing the sampling rate, eg, by changing focal length or camera distance.
We find that the source for this phenomenon can be attributed to the lack of 3D frequency constraints and the usage of a 2D dilation filter.
arXiv Detail & Related papers (2023-11-27T13:03:09Z) - Towards Scalable Multi-View Reconstruction of Geometry and Materials [27.660389147094715]
We propose a novel method for joint recovery of camera pose, object geometry and spatially-varying Bidirectional Reflectance Distribution Function (svBRDF) of 3D scenes.
The input are high-resolution RGBD images captured by a mobile, hand-held capture system with point lights for active illumination.
arXiv Detail & Related papers (2023-06-06T15:07:39Z) - HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion
Guidance [19.252300247300145]
This work proposes holistic sampling and smoothing approaches to achieve high-quality text-to-3D generation.
We compute denoising scores in the text-to-image diffusion model's latent and image spaces.
To generate high-quality renderings in a single-stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays.
arXiv Detail & Related papers (2023-05-30T05:56:58Z) - Adaptive Joint Optimization for 3D Reconstruction with Differentiable
Rendering [22.2095090385119]
Given an imperfect reconstructed 3D model, most previous methods have focused on the refinement of either geometry, texture, or camera pose.
We propose a novel optimization approach based on differentiable rendering, which integrates the optimization of camera pose, geometry, and texture into a unified framework.
Using differentiable rendering, an image-level adversarial loss is applied to further improve the 3D model, making it more photorealistic.
arXiv Detail & Related papers (2022-08-15T04:32:41Z) - IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from
Photometric Images [52.021529273866896]
We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content.
Our method adopts neural representations for geometry as signed distance fields (SDFs) and materials during optimization to enjoy their flexibility and compactness.
We show that our IRON achieves significantly better inverse rendering quality compared to prior works.
arXiv Detail & Related papers (2022-04-05T14:14:18Z) - Differentiable Rendering with Perturbed Optimizers [85.66675707599782]
Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision.
Our work highlights the link between some well-known differentiable formulations and randomly smoothed renderings.
We apply our method to 3D scene reconstruction and demonstrate its advantages on the tasks of 6D pose estimation and 3D mesh reconstruction.
arXiv Detail & Related papers (2021-10-18T08:56:23Z) - Snapshot Hyperspectral Imaging Based on Weighted High-order Singular
Value Regularization [22.5033027930853]
Snapshot hyperspectral imaging can capture the 3D hyperspectral image (HSI) with a single 2D measurement.
Existing reconstruction methods cannot fully exploit the structurally spectral-spatial nature in 3D HSI.
We propose an effective high-order tensor optimization based method to boost the reconstruction fidelity for snapshot hyperspectral imaging.
arXiv Detail & Related papers (2021-01-22T02:54:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.