MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation

• URL: http://arxiv.org/abs/2510.17866v1
• Date: Wed, 15 Oct 2025 22:16:09 GMT
• Title: MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation
• Abstract summary: MUSE is a training-free framework designed for model-based zero-shot 2D object detection and segmentation.<n>It exploits 2D multi-view templates rendered from 3D unseen objects and 2D object proposals extracted from input query images.<n>MUSE achieves state-of-the-art performance on the BOP Challenge 2025, ranking first across the Classic Core, H3, and Industrial tracks.
• Score: 3.862980857248212
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we introduce MUSE (Model-based Uncertainty-aware Similarity Estimation), a training-free framework designed for model-based zero-shot 2D object detection and segmentation. MUSE leverages 2D multi-view templates rendered from 3D unseen objects and 2D object proposals extracted from input query images. In the embedding stage, it integrates class and patch embeddings, where the patch embeddings are normalized using generalized mean pooling (GeM) to capture both global and local representations efficiently. During the matching stage, MUSE employs a joint similarity metric that combines absolute and relative similarity scores, enhancing the robustness of matching under challenging scenarios. Finally, the similarity score is refined through an uncertainty-aware object prior that adjusts for proposal reliability. Without any additional training or fine-tuning, MUSE achieves state-of-the-art performance on the BOP Challenge 2025, ranking first across the Classic Core, H3, and Industrial tracks. These results demonstrate that MUSE offers a powerful and generalizable framework for zero-shot 2D object detection and segmentation.

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding [2.1987601456703474]
  We introduce a unified, end-to-end framework that seamlessly integrates object detection and pose estimation.<n>Our system first employs the CNOS detector to localize target objects.<n>For each detection, our novel pose estimation module, OPFormer, infers the precise 6D pose.
  arXiv  Detail & Related papers  (2025-11-16T14:19:52Z)
• Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precision [1.2036459029967645]
  Out-of-Distribution (OoD) segmentation is critical for safety-sensitive applications like autonomous driving.<n>Existing mask-based methods often suffer from boundary imprecision, inconsistent anomaly scores within objects, and false positives from background noise.<n>We propose textbftextitObjectomaly, an objectness-aware refinement framework that incorporates object-level priors.
  arXiv  Detail & Related papers  (2025-07-10T06:23:35Z)
• NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation [42.84923737757637]
  We present a new training-free framework, called: Novel Object Cyclic Threshold based Instance (NOCTIS)<n> NOCTIS integrates two pre-trained models: Grounded-SAM 2 for object proposals with precise bounding boxes and corresponding segmentation masks; and DINOv2 for robust class and patch embeddings.<n>We empirically show that NOCTIS, without further training/fine tuning, attains state-of-the-art results regarding the mean AP score.
  arXiv  Detail & Related papers  (2025-07-02T08:23:14Z)
• Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes [71.2827490406779]
  We propose NPISeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges.<n>NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization.<n>We design a prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model's ability to capture object-aware context.
  arXiv  Detail & Related papers  (2025-05-03T07:43:23Z)
• PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection [65.84604846389624]
  We propose PointOBB-v3, a stronger single point-supervised OOD framework.<n>It generates pseudo rotated boxes without additional priors and incorporates support for the end-to-end paradigm.<n>Our method achieves an average improvement in accuracy of 3.56% in comparison to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-23T18:18:15Z)
• iComMa: Inverting 3D Gaussian Splatting for Camera Pose Estimation via Comparing and Matching [14.737266480464156]
  We present a method named iComMa to address the 6D camera pose estimation problem in computer vision. We propose an efficient method for accurate camera pose estimation by inverting 3D Gaussian Splatting (3DGS)
  arXiv  Detail & Related papers  (2023-12-14T15:31:33Z)
• Occ$^2$Net: Robust Image Matching Based on 3D Occupancy Estimation for Occluded Regions [14.217367037250296]
  Occ$2$Net is an image matching method that models occlusion relations using 3D occupancy and infers matching points in occluded regions. We evaluate our method on both real-world and simulated datasets and demonstrate its superior performance over state-of-the-art methods on several metrics.
  arXiv  Detail & Related papers  (2023-08-14T13:09:41Z)
• GOOD: General Optimization-based Fusion for 3D Object Detection via LiDAR-Camera Object Candidates [10.534984939225014]
  3D object detection serves as the core basis of the perception tasks in autonomous driving. Good is a general optimization-based fusion framework that can achieve satisfying detection without training additional models. Experiments on both nuScenes and KITTI datasets are carried out and the results show that GOOD outperforms by 9.1% on mAP score compared with PointPillars.
  arXiv  Detail & Related papers  (2023-03-17T07:05:04Z)
• 3DMODT: Attention-Guided Affinities for Joint Detection & Tracking in 3D Point Clouds [95.54285993019843]
  We propose a method for joint detection and tracking of multiple objects in 3D point clouds. Our model exploits temporal information employing multiple frames to detect objects and track them in a single network.
  arXiv  Detail & Related papers  (2022-11-01T20:59:38Z)
• Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes [82.4186966781934]
  We introduce a simple, efficient, and effective two-stage detector, termed as Ret3D. At the core of Ret3D is the utilization of novel intra-frame and inter-frame relation modules. With negligible extra overhead, Ret3D achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-18T03:48:58Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)
• Sim2Real Object-Centric Keypoint Detection and Description [40.58367357980036]
  Keypoint detection and description play a central role in computer vision. We propose the object-centric formulation, which requires further identifying which object each interest point belongs to. We develop a sim2real contrastive learning mechanism that can generalize the model trained in simulation to real-world applications.
  arXiv  Detail & Related papers  (2022-02-01T15:00:20Z)
• Similarity-Aware Fusion Network for 3D Semantic Segmentation [87.51314162700315]
  We propose a similarity-aware fusion network (SAFNet) to adaptively fuse 2D images and 3D point clouds for 3D semantic segmentation. We employ a late fusion strategy where we first learn the geometric and contextual similarities between the input and back-projected (from 2D pixels) point clouds. We show that SAFNet significantly outperforms existing state-of-the-art fusion-based approaches across various data integrity.
  arXiv  Detail & Related papers  (2021-07-04T09:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.