MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation

• URL: http://arxiv.org/abs/2604.08916v1
• Date: Fri, 10 Apr 2026 03:26:52 GMT
• Title: MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation
• Abstract summary: Conventional 3D instance segmentation methods rely on labor-intensive 3D annotations for supervised training.<n>Recent approaches leverage multi-view 2D masks from the Segment Anything Model (SAM) to guide the merging of 3D geometric primitives.<n>We propose MV3DIS, a coarse-to-fine framework for zero-shot 3D instance segmentation that explicitly incorporates 3D priors.
• Score: 12.513474673735963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional 3D instance segmentation methods rely on labor-intensive 3D annotations for supervised training, which limits their scalability and generalization to novel objects. Recent approaches leverage multi-view 2D masks from the Segment Anything Model (SAM) to guide the merging of 3D geometric primitives, thereby enabling zero-shot 3D instance segmentation. However, these methods typically process each frame independently and rely solely on 2D metrics, such as SAM prediction scores, to produce segmentation maps. This design overlooks multi-view correlations and inherent 3D priors, leading to inconsistent 2D masks across views and ultimately fragmented 3D segmentation. In this paper, we propose MV3DIS, a coarse-to-fine framework for zero-shot 3D instance segmentation that explicitly incorporates 3D priors. Specifically, we introduce a 3D-guided mask matching strategy that uses coarse 3D segments as a common reference to match 2D masks across views and consolidates multi-view mask consistency via 3D coverage distributions. Guided by these view-consistent 2D masks, the coarse 3D segments are further refined into precise 3D instances. Additionally, we introduce a depth consistency weighting scheme that quantifies projection reliability to suppress ambiguities from inter-object occlusions, thereby improving the robustness of 3D-to-2D correspondence. Extensive experiments on the ScanNetV2, ScanNet200, ScanNet++, Replica, and Matterport3D datasets demonstrate the effectiveness of MV3DIS, which achieves superior performance over previous methods

Related papers

• CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging [24.37584731810519]
  Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments.<n>We introduce Cross-Dimensional Class-Agnostic 3D Instance (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints.<n>This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training.
  arXiv  Detail & Related papers  (2026-07-20T10:09:12Z)
• MV-SAM: Multi-view Promptable Segmentation using Pointmap Guidance [79.57732829495843]
  We introduce MV-SAM, a framework for multi-view segmentation that achieves 3D consistency using pointmaps.<n>MV-SAM lifts images and prompts into 3D space, eliminating the need for explicit 3D networks or annotated 3D data.
  arXiv  Detail & Related papers  (2026-01-25T15:00:37Z)
• Integrating SAM Supervision for 3D Weakly Supervised Point Cloud Segmentation [66.65719382619538]
  Current methods for 3D semantic segmentation propose training models with limited annotations to address the difficulty of annotating large, irregular, and unordered 3D point cloud data.<n>We present a novel approach that maximizes the utility of sparsely available 3D annotations incorporating segmentation masks generated by 2D foundation models.
  arXiv  Detail & Related papers  (2025-08-27T14:13:01Z)
• DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting [0.0]
  We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations.<n>We evaluate on synthetic and real-world indoor datasets, demonstrating improved performance over comparable NeRF-based pipelines.
  arXiv  Detail & Related papers  (2024-12-14T21:26:44Z)
• Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking [6.599971425078935]
  Existing 3D instance segmentation methods frequently encounter issues with over-segmentation, leading to redundant and inaccurate 3D proposals that complicate downstream tasks. This challenge arises from their unsupervised merging approach, where dense 2D masks are lifted across frames into point clouds to form 3D candidate proposals without direct supervision. We propose a 3D-Aware 2D Mask Tracking module that uses robust 3D priors from a 2D mask segmentation and tracking foundation model (SAM-2) to ensure consistent object masks across video frames.
  arXiv  Detail & Related papers  (2024-11-25T08:26:31Z)
• EmbodiedSAM: Online Segment Any 3D Thing in Real Time [61.2321497708998]
  Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration.<n>An online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed.
  arXiv  Detail & Related papers  (2024-08-21T17:57:06Z)
• MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation [11.123421412837336]
  Open-vocabulary 3D instance segmentation is cutting-edge for its ability to segment 3D instances without predefined categories. Recent works first generate 2D open-vocabulary masks through 2D models and then merge them into 3D instances based on metrics calculated between two neighboring frames. We propose a novel metric, view consensus rate, to enhance the utilization of multi-view observations.
  arXiv  Detail & Related papers  (2024-01-15T14:56:15Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• Segment Anything in 3D with Radiance Fields [83.14130158502493]
  This paper generalizes the Segment Anything Model (SAM) to segment 3D objects. We refer to the proposed solution as SA3D, short for Segment Anything in 3D. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within seconds.
  arXiv  Detail & Related papers  (2023-04-24T17:57:15Z)
• Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training [65.75399500494343]
  Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for 2D and 3D computer vision. We propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
  arXiv  Detail & Related papers  (2023-02-27T17:56:18Z)
• Virtual Multi-view Fusion for 3D Semantic Segmentation [11.259694096475766]
  We show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results.
  arXiv  Detail & Related papers  (2020-07-26T14:46:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.