SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model

• URL: http://arxiv.org/abs/2409.17345v1
• Date: Wed, 25 Sep 2024 20:45:19 GMT
• Title: SeaSplat: Representing Underwater Scenes with 3D Gaussian Splatting and a Physically Grounded Image Formation Model
• Authors: Daniel Yang, John J. Leonard, Yogesh Girdhar,
• Abstract summary: We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation.
• Score: 11.57677379828992
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Underwater scenes are challenging visual environments, as rendering through a medium such as water introduces both range and color dependent effects on image capture. We constrain 3D Gaussian Splatting (3DGS), a recent advance in radiance fields enabling rapid training and real-time rendering of full 3D scenes, with a physically grounded underwater image formation model. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands, and simulation-degraded real-world scenes, not only do we see increased quantitative performance on rendering novel viewpoints from the scene with the medium present, but are also able to recover the underlying true color of the scene and restore renders to be without the presence of the intervening medium. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation.

Related papers

• Aquatic-GS: A Hybrid 3D Representation for Underwater Scenes [6.549998173302729]
  We propose Aquatic-GS, a hybrid 3D representation approach for underwater scenes that effectively represents both the objects and the water medium. Specifically, we construct a Neural Water Field (NWF) to implicitly model the water parameters, while extending the latest 3D Gaussian Splatting (3DGS) to model the objects explicitly. Both components are integrated through a physics-based underwater image formation model to represent complex underwater scenes.
  arXiv  Detail & Related papers  (2024-10-31T22:24:56Z)
• WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting [39.58317527488534]
  We propose a novel approach that fuses volumetric rendering with 3DGS to handle underwater data effectively. Our method outperforms state-of-the-art NeRF-based methods in rendering quality on the underwater SeaThru-NeRF dataset.
  arXiv  Detail & Related papers  (2024-08-15T15:16:49Z)
• Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion [77.34078223594686]
  We propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.
  arXiv  Detail & Related papers  (2024-01-19T16:15:37Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• Ghost on the Shell: An Expressive Representation of General 3D Shapes [97.76840585617907]
  Meshes are appealing since they enable fast physics-based rendering with realistic material and lighting. Recent work on reconstructing and statistically modeling 3D shapes has critiqued meshes as being topologically inflexible. We parameterize open surfaces by defining a manifold signed distance field on watertight surfaces. G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks.
  arXiv  Detail & Related papers  (2023-10-23T17:59:52Z)
• SUCRe: Leveraging Scene Structure for Underwater Color Restoration [1.9490160607392462]
  We introduce SUCRe, a novel method that exploits the scene's 3D structure for underwater color restoration. We conduct extensive quantitative and qualitative analyses of our approach in a variety of scenarios ranging from natural light to deep-sea environments.
  arXiv  Detail & Related papers  (2022-12-18T16:53:13Z)
• PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes [84.66946637534089]
  PhotoScene is a framework that takes input image(s) of a scene and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes.
  arXiv  Detail & Related papers  (2022-07-02T06:52:44Z)
• Underwater Light Field Retention : Neural Rendering for Underwater Imaging [6.22867695581195]
  Underwater Image Rendering aims to generate a true-to-life underwater image from a given clean one. We propose a neural rendering method for underwater imaging, dubbed UWNR (Underwater Neural Rendering).
  arXiv  Detail & Related papers  (2022-03-21T14:22:05Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)
• Deep Sea Robotic Imaging Simulator [6.2122699483618]
  The largest portion of the ocean - the deep sea - still remains mostly unexplored. Deep sea images are very different from the images taken in shallow waters and this area did not get much attention from the community. This paper presents a physical model-based image simulation solution, which uses an in-air texture and depth information as inputs.
  arXiv  Detail & Related papers  (2020-06-27T16:18:32Z)
• 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
  We propose a method for converting a single RGB-D input image into a 3D photo. A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region. The resulting 3D photos can be efficiently rendered with motion parallax.
  arXiv  Detail & Related papers  (2020-04-09T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.