SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

• URL: http://arxiv.org/abs/2604.20395v1
• Date: Wed, 22 Apr 2026 09:57:57 GMT
• Title: SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation
• Abstract summary: SpaCeFormer is a proposal-free space-curve transformer that runs at 0.14 seconds per scene.<n>SpaCeFormer reaches 21x higher mask recall than prior single-view pipelines.<n>On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8x improvement over the prior best proposal-free method.
• Score: 82.87586897359367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs at 0.14 seconds per scene, 2-3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches 21x higher mask recall than prior single-view pipelines (54.3% vs 2.5% at IoU > 0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8x improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.

Related papers

• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• Topologically Consistent Multi-view 3D Head Reconstruction via Coarse-Guided Layered Surface Sampling [15.24889101682807]
  SHELLS is an efficient feed-forward framework for 3D head reconstruction in dense semantic correspondence from multi-view images.<n>It reduces median registration error by 21% to 29% with a 3.5x inference speedup.<n>Our model is trained exclusively on synthetic data yet generalizes effectively to real-world captures.
  arXiv  Detail & Related papers  (2026-05-29T13:14:43Z)
• OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation [5.073373064582996]
  We introduce OpenTrack3D, a generalizable and accurate framework for open-vocabulary 3D instance segmentation.<n>Unlike methods that rely on pre-generated proposals, OpenTrack3D employs a novel visual-spatial tracker to construct cross-view consistent object proposals.<n>Our pipeline is entirely mesh-free, yet we also provide an optional superpoints refinement module to further enhance performance when scene mesh is available.
  arXiv  Detail & Related papers  (2025-12-03T07:51:03Z)
• PointGauss: Point Cloud-Guided Multi-Object Segmentation for Gaussian Splatting [18.042769428774676]
  We introduce PointGauss, a novel point cloud-guided framework for real-time multi-object segmentation in Gaussian Splatting representations.<n>The key innovation lies in two aspects: (1) a point cloud-based Gaussian primitive decoder that generates 3D instance masks within 1 minute, and (2) a GPU-accelerated 2D mask rendering system that ensures multi-view consistency.
  arXiv  Detail & Related papers  (2025-08-01T01:56:54Z)
• OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging [36.9859733771263]
  We propose an efficient method for lifting 2D masks into a unified 3D instance using a hashing technique.<n>By employing voxel hashing for efficient 3D scene querying, our approach reduces the time complexity of costly spatial overlap queries.<n>Our approach achieves state-of-the-art performance in online, zero-shot 3D instance segmentation with leading efficiency.
  arXiv  Detail & Related papers  (2025-03-03T08:48:06Z)
• Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation [92.17176311351469]
  We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework.<n>Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale.<n>Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs.
  arXiv  Detail & Related papers  (2025-02-04T18:18:50Z)
• Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking [6.599971425078935]
  Existing 3D instance segmentation methods frequently encounter issues with over-segmentation, leading to redundant and inaccurate 3D proposals that complicate downstream tasks. This challenge arises from their unsupervised merging approach, where dense 2D masks are lifted across frames into point clouds to form 3D candidate proposals without direct supervision. We propose a 3D-Aware 2D Mask Tracking module that uses robust 3D priors from a 2D mask segmentation and tracking foundation model (SAM-2) to ensure consistent object masks across video frames.
  arXiv  Detail & Related papers  (2024-11-25T08:26:31Z)
• Triple Point Masking [49.39218611030084]
  Existing 3D mask learning methods encounter performance bottlenecks under limited data. We introduce a triple point masking scheme, named TPM, which serves as a scalable framework for pre-training of masked autoencoders. Extensive experiments show that the four baselines equipped with the proposed TPM achieve comprehensive performance improvements on various downstream tasks.
  arXiv  Detail & Related papers  (2024-09-26T05:33:30Z)
• Mask Propagation for Efficient Video Semantic Segmentation [63.09523058489429]
  Video Semantic baseline degradation (VSS) involves assigning a semantic label to each pixel in a video sequence. We propose an efficient mask propagation framework for VSS, called SSSS. Our framework reduces up to 4x FLOPs compared to the per-frame Mask2Former with only up to 2% mIoU on the Cityscapes validation set.
  arXiv  Detail & Related papers  (2023-10-29T09:55:28Z)
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation [89.41640045953378]
  We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Mask3D sets a new state-of-the-art on ScanNet test (+6.2 mAP), S3DIS 6-fold (+10.1 mAP),LS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP)
  arXiv  Detail & Related papers  (2022-10-06T17:55:09Z)
• Pyramid Fusion Transformer for Semantic Segmentation [44.57867861592341]
  We propose a transformer-based Pyramid Fusion Transformer (PFT) for per-mask approach semantic segmentation with multi-scale features. We achieve competitive performance on three widely used semantic segmentation datasets.
  arXiv  Detail & Related papers  (2022-01-11T16:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.