SmileSplat: Generalizable Gaussian Splats for Unconstrained Sparse Images

• URL: http://arxiv.org/abs/2411.18072v1
• Date: Wed, 27 Nov 2024 05:52:28 GMT
• Title: SmileSplat: Generalizable Gaussian Splats for Unconstrained Sparse Images
• Authors: Yanyan Li, Yixin Fang, Federico Tombari, Gim Hee Lee,
• Abstract summary: A novel generalizable Gaussian Splatting method, SmileSplat, is proposed to reconstruct pixel-aligned Gaussian surfels for diverse scenarios. The proposed method achieves state-of-the-art performance in various 3D vision tasks.
• Score: 91.28365943547703
• License:
• Abstract: Sparse Multi-view Images can be Learned to predict explicit radiance fields via Generalizable Gaussian Splatting approaches, which can achieve wider application prospects in real-life when ground-truth camera parameters are not required as inputs. In this paper, a novel generalizable Gaussian Splatting method, SmileSplat, is proposed to reconstruct pixel-aligned Gaussian surfels for diverse scenarios only requiring unconstrained sparse multi-view images. First, Gaussian surfels are predicted based on the multi-head Gaussian regression decoder, which can are represented with less degree-of-freedom but have better multi-view consistency. Furthermore, the normal vectors of Gaussian surfel are enhanced based on high-quality of normal priors. Second, the Gaussians and camera parameters (both extrinsic and intrinsic) are optimized to obtain high-quality Gaussian radiance fields for novel view synthesis tasks based on the proposed Bundle-Adjusting Gaussian Splatting module. Extensive experiments on novel view rendering and depth map prediction tasks are conducted on public datasets, demonstrating that the proposed method achieves state-of-the-art performance in various 3D vision tasks. More information can be found on our project page (https://yanyan-li.github.io/project/gs/smilesplat)

Related papers

• GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance [43.97159590077809]
  We present GaussianPainter, the first method to paint a point cloud into 3D Gaussians given a reference image. Our method addresses the non-uniqueness problem inherent in the large parameter space of 3D Gaussian splatting.
  arXiv  Detail & Related papers  (2024-12-23T16:45:37Z)
• GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction [55.60972844777044]
  3D semantic occupancy prediction is an important task for robust vision-centric autonomous driving. Most existing methods leverage dense grid-based scene representations, overlooking the spatial sparsity of the driving scenes. We propose a probabilistic Gaussian superposition model which interprets each Gaussian as a probability distribution of its neighborhood being occupied.
  arXiv  Detail & Related papers  (2024-12-05T17:59:58Z)
• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views [67.34073368933814]
  We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
  arXiv  Detail & Related papers  (2024-11-18T08:18:44Z)
• PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views [116.10577967146762]
  PixelGaussian is an efficient framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Our method achieves state-of-the-art performance with good generalization to various numbers of views.
  arXiv  Detail & Related papers  (2024-10-24T17:59:58Z)
• HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction [46.269350101349715]
  HiSplat is a novel framework for generalizable 3D Gaussian Splatting. It generates hierarchical 3D Gaussians via a coarse-to-fine strategy. It significantly enhances reconstruction quality and cross-dataset generalization.
  arXiv  Detail & Related papers  (2024-10-08T17:59:32Z)
• GaussianForest: Hierarchical-Hybrid 3D Gaussian Splatting for Compressed Scene Modeling [40.743135560583816]
  We introduce the Gaussian-Forest modeling framework, which hierarchically represents a scene as a forest of hybrid 3D Gaussians. Experiments demonstrate that Gaussian-Forest not only maintains comparable speed and quality but also achieves a compression rate surpassing 10 times.
  arXiv  Detail & Related papers  (2024-06-13T02:41:11Z)
• MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images [102.7646120414055]
  We introduce MVSplat, an efficient model that, given sparse multi-view images as input, predicts clean feed-forward 3D Gaussians. On the large-scale RealEstate10K and ACID benchmarks, MVSplat achieves state-of-the-art performance with the fastest feed-forward inference speed (22fps)
  arXiv  Detail & Related papers  (2024-03-21T17:59:58Z)
• GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis [70.24111297192057]
  We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting.
  arXiv  Detail & Related papers  (2023-12-04T18:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.