Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin

• URL: http://arxiv.org/abs/2511.20348v2
• Date: Fri, 28 Nov 2025 16:12:54 GMT
• Title: Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin
• Abstract summary: Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass.<n>We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images.<n>This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion.
• Score: 9.932207949877636
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: 3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.

Related papers

• PhysMirror: Physics-Aware Mirror Object Generation [27.32616510065338]
  We introduce a novel, end-to-end physics-aware generation framework namely PhysMirror.<n>Our method automatically lifts prompted objects into 3D meshes and constructs a lightweight, mathematically exact mirror scene.<n>We also introduce Mirror Consistency Score (MCS), reference-free, fully automated metric that quantifies physical correctness.
  arXiv  Detail & Related papers  (2026-07-03T16:32:02Z)
• Unposed-to-3D: Learning Simulation-Ready Vehicles from Real-World Images [29.849361869760713]
  We present Unposed-to-3D, a novel framework that learns to reconstruct 3D vehicles from real-world driving images using image-only supervision.<n>In experiments, Unposed-to-3D effectively reconstructs realistic, pose-consistent, and harmonized 3D vehicle models from real-world images.
  arXiv  Detail & Related papers  (2026-04-21T09:20:39Z)
• LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting [90.15229815175593]
  We present LightHarmony3D, a novel framework for illumination-consistent mesh insertion in 3DGS scenes.<n>By leveraging generative priors instead of iterative optimization, our method efficiently captures dominant scene illumination.<n>We introduce the first dedicated benchmark for mesh insertion in 3DGS, providing a standardized evaluation framework for assessing lighting consistency and photorealism.
  arXiv  Detail & Related papers  (2026-03-31T03:26:30Z)
• MatSpray: Fusing 2D Material World Knowledge on 3D Geometry [7.4570191712029965]
  We propose a framework for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches.<n>Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism.
  arXiv  Detail & Related papers  (2025-12-20T10:58:45Z)
• RePose-NeRF: Robust Radiance Fields for Mesh Reconstruction under Noisy Camera Poses [0.5328562411561214]
  We propose a robust framework that reconstructs high-quality, editable 3D meshes directly from multi-view images.<n>Our approach jointly refines camera poses while learning an implicit scene representation that captures fine geometric detail and appearance.
  arXiv  Detail & Related papers  (2025-11-11T18:25:58Z)
• Hybrid Rendering for Multimodal Autonomous Driving: Merging Neural and Physics-Based Simulation [1.0027737736304287]
  We introduce a hybrid approach that combines the strengths of neural reconstruction with physics-based rendering.<n>Our approach significantly enhances novel view synthesis quality, especially for road surfaces and lane markings.<n>We achieve this by training a customized NeRF model on the original images with depth regularization derived from a noisy LiDAR point cloud.
  arXiv  Detail & Related papers  (2025-03-12T15:18:50Z)
• FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction [69.63414788486578]
  FreeSplatter is a scalable feed-forward framework that generates high-quality 3D Gaussians from uncalibrated sparse-view images.<n>Our approach employs a streamlined transformer architecture where self-attention blocks facilitate information exchange.<n>We develop two specialized variants--for object-centric and scene-level reconstruction--trained on comprehensive datasets.
  arXiv  Detail & Related papers  (2024-12-12T18:52:53Z)
• LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives [44.06145846507639]
  LetsGo is a LiDAR-assisted Gaussian splatting framework for large-scale garage modeling and rendering. We present the GarageWorld dataset, consisting of eight expansive garage scenes with diverse geometric structures. We introduce a novel depth regularizer that effectively eliminates floating artifacts in rendered images.
  arXiv  Detail & Related papers  (2024-04-15T12:50:44Z)
• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements [59.70107451308687]
  We show for the first time that using 3D Gaussians for map representation with unposed camera images and inertial measurements can enable accurate SLAM. Our method, MM3DGS, addresses the limitations of prior rendering by enabling faster scale awareness, and improved trajectory tracking. We also release a multi-modal dataset, UT-MM, collected from a mobile robot equipped with a camera and an inertial measurement unit.
  arXiv  Detail & Related papers  (2024-04-01T04:57:41Z)
• Holistic Inverse Rendering of Complex Facade via Aerial 3D Scanning [38.72679977945778]
  We use multi-view aerial images to reconstruct the geometry, lighting, and material of facades using neural signed distance fields (SDFs) The experiment demonstrates the superior quality of our method on facade holistic inverse rendering, novel view synthesis, and scene editing compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-11-20T15:03:56Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes [48.642181362172906]
  We present an algorithm that learns a coarse 3D representation of objects from unposed multi-view 2D mask supervision. In contrast to existing voxel-based methods for unposed object reconstruction, our approach learns to represent the generated shape and pose. We show results on synthetic datasets with realistic lighting, and demonstrate object insertion with interactive posing.
  arXiv  Detail & Related papers  (2021-06-24T17:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.