Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

• URL: http://arxiv.org/abs/2606.24353v1
• Date: Tue, 23 Jun 2026 09:43:12 GMT
• Title: Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints
• Abstract summary: We introduce open-vocabulary BEV segmentation (OVBS) to recognize categories beyond the training set.<n>OVBEVSeg is a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection.<n>On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories.
• Score: 20.26519299938903
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.

Related papers

• Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation [76.16065370700488]
  We introduce Lift3D-VLA, a unified VLA framework that equips models with explicit 3D point cloud reasoning and enables temporally coherent action generation.<n>We show that Lift3D-VLA achieves 10.8% and 11.1% higher mean success rates on MetaWorld and RLBench than the best-performing prior VLA methods.
  arXiv  Detail & Related papers  (2026-07-07T17:59:47Z)
• MVDGC: Joint 3D and 2D Multi-view Pedestrian Detection via Dual Geometric Constraints [11.583454260842524]
  We propose a unified framework that estimates pedestrian locations on the BEV plane and 2D bounding boxes in image views.<n>MVDGC employs a emphsparse set of 3D cylindrical queries that embraces geometric context across both BEV and image views.<n>These queries function as shape anchors that directly extract 2D features from the intact image-view features using camera projection.
  arXiv  Detail & Related papers  (2026-06-30T23:39:38Z)
• GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation [75.85672467847631]
  We introduce the Geometry-Aware BEV (GA-BEV) - a compact, 3D-grounded feature representation.<n>We construct BEV spatial maps from RGB-D inputs by projecting visual features into 3D space.<n>We incorporate features from a pretrained 3D foundation model into the BEV space, injecting structural priors learned from large-scale 3D reconstruction tasks.
  arXiv  Detail & Related papers  (2026-05-21T06:20:17Z)
• Reconstruction Matters: Learning Geometry-Aligned BEV Representation through 3D Gaussian Splatting [24.49684091836623]
  Splat2BEV aims to learn BEV feature representations that are both semantically rich and geometrically precise.<n>Experiments on nuScenes and argoverse dataset demonstrate that Splat2BEV achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-19T17:49:43Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• DualBEV: Unifying Dual View Transformation with Probabilistic Correspondences [0.02999888908665658]
  Camera-based Bird's-Eye-View (BEV) perception often struggles between adopting 3D-to-2D or 2D-to-3D view transformation (VT) We propose DualBEV, a unified framework that utilizes a shared feature transformation incorporating three probabilistic measurements for both strategies. Our method achieves state-of-the-art performance without Transformer, delivering comparable efficiency to the LSS approach, with 55.2% mAP and 63.4% NDS on the nuScenes test set.
  arXiv  Detail & Related papers  (2024-03-08T15:58:00Z)
• BEV-IO: Enhancing Bird's-Eye-View 3D Detection with Instance Occupancy [58.92659367605442]
  We present BEV-IO, a new 3D detection paradigm to enhance BEV representation with instance occupancy information. We show that BEV-IO can outperform state-of-the-art methods while only adding a negligible increase in parameters and computational overhead.
  arXiv  Detail & Related papers  (2023-05-26T11:16:12Z)
• GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation [105.19949897812494]
  Birds-eye-view (BEV) semantic segmentation is critical for autonomous driving. We present a novel two-stage Geometry Prior-based Transformation framework named GitNet.
  arXiv  Detail & Related papers  (2022-04-16T06:46:45Z)
• M^2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation [145.6041893646006]
  M$2$BEV is a unified framework that jointly performs 3D object detection and map segmentation. M$2$BEV infers both tasks with a unified model and improves efficiency.
  arXiv  Detail & Related papers  (2022-04-11T13:43:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.