Glass Surface Detection Grounded in 3D Visual Geometry
Abstract Overview
This paper reframes glass surface detection as a 3D visual geometry-grounded task rather than relying predominantly on 2D appearance cues. The proposed framework builds upon the Visual Geometry Grounded Transformer (VGGT), using its 3D priors while rectifying depth and point-cloud supervision within glass regions via boundary-anchored planar interpolation. The architecture incorporates a dedicated glass detection head combining a Frequency Self-Attention Module (FSAM) to capture glass-induced spectral patterns and a Geometry Grounding Block (GeGB) to fuse 2D features with 3D depth and point-cloud representations. The model is trained via Low-Rank Adaptation (LoRA) under a multi-task loss covering glass masks, boundaries, depth, and point clouds.
Novelty
The main novelty is the paradigm shift to explicitly ground glass surface detection in large-scale pre-trained 3D visual geometry instead of 2D appearance alone. It introduces a training scheme with planar depth/point-cloud rectification for glass regions, alongside a detection head coupling frequency-domain attention with 3D geometry grounding.
Results
The approach achieves state-of-the-art performance across seven standard single-image, multi-modal, and video glass surface detection benchmarks. It demonstrates strong generalization to video and multi-modal datasets without requiring extra physical sensors, and runs interactively at approximately 8.5 FPS on an NVIDIA RTX 4090 GPU.
Key Points
- The method adapts pre-trained VGGT 3D priors using LoRA and rectifies depth and point-cloud pseudo-ground truth within glass regions using planar interpolation.
- The glass detection head pairs a Frequency Self-Attention Module (FSAM) for spectral glass localization with a Geometry Grounding Block (GeGB) for geometry-guided feature fusion.
- Evaluations show consistent performance improvements across seven benchmarks, enhanced monocular 3D scene reconstruction around glass, and interactive runtime efficiency.