Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation

• URL: http://arxiv.org/abs/2605.02580v1
• Date: Mon, 04 May 2026 13:33:43 GMT
• Title: Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation
• Abstract summary: Open-Set Panoptic (OPS) aims to segment known thing and stuff classes while identifying valid unknown objects as separate instances.<n>Prior approaches largely treat known categories as a flat label set, ignoring the semantic hierarchy that provides valuable structural priors for distinguishing unknown objects from in-distribution classes.<n>We propose Hyp2Former, an end-to-end framework for OPS that does not require explicit modeling of unknowns during training, and instead learns hierarchical semantic similarities continuously in hyperbolic space.
• Score: 19.04466519384799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recognizing unknown objects is crucial for safety-critical applications such as autonomous driving and robotics. Open-Set Panoptic Segmentation (OPS) aims to segment known thing and stuff classes while identifying valid unknown objects as separate instances. Prior OPS approaches largely treat known categories as a flat label set, ignoring the semantic hierarchy that provides valuable structural priors for distinguishing unknown objects from in-distribution classes. In this work, we propose Hyp2Former, an end-to-end framework for OPS that does not require explicit modeling of unknowns during training, and instead learns hierarchical semantic similarities continuously in hyperbolic space. By explicitly encoding hierarchical relationships among known categories, the model learns a structured embedding space that captures multiple levels of semantic abstraction. As a result, unknown objects that cannot be confidently classified as known categories still remain in close proximity to higher-level concepts (e.g., an unknown animal remains closer to "animal" or "object" than to unrelated concepts such as "electronics" or "stuff") and can therefore be reliably detected, even if their fine-grained category was not represented during training. Empirical evaluations across multiple public datasets such as MS COCO, Cityscapes, and Lost&Found demonstrate that Hyp2Former outperforms existing methods on OPS, achieving the best balance between unknown object discovery and in-distribution robustness.

Related papers

• Multimodal Semantic-Probabilistic Objectness for Open World Object Detection [2.7962583061246327]
  MSPO is a semantic calibration framework for open-world object detection.<n>It calibrates known and unknown predictions without turning OWOD into open-vocabulary classification.<n>It also improves early unknown-confusion metrics and raises PASCAL VOC final mAP by up to 2.7 points.
  arXiv  Detail & Related papers  (2026-07-27T04:12:04Z)
• Language-Guided Open-World Anomaly Segmentation [81.13615952207685]
  We propose Clipomaly, the first CLIP-based open-world and anomaly segmentation method for autonomous driving.<n>Our zero-shot approach requires no anomaly-specific training data and leverages CLIP's shared image-text embedding space to segment unknown objects.<n>Our model dynamically extends its vocabulary at inference time without retraining, enabling robust detection and naming of anomalies beyond common class definitions.
  arXiv  Detail & Related papers  (2025-12-01T09:08:59Z)
• TARO: Toward Semantically Rich Open-World Object Detection [9.59690330728612]
  TARO is a novel detection framework that identifies unknown objects and classifies them into coarse parent categories within a semantic hierarchy.<n>We show TARO can categorize up to 29.9% of unknowns into meaningful coarse classes, significantly reduce confusion between unknown and known classes, and achieve competitive performance in both unknown recall and known mAP.
  arXiv  Detail & Related papers  (2025-10-10T09:15:26Z)
• Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts [81.68203255687051]
  Generalized Category Discovery is an open-world problem that clusters unlabeled data by leveraging knowledge from partially labeled categories.<n>Existing approaches fail to exploit multi-granularity conceptual information in visual data.<n>We propose a Multi-Granularity Experts framework that integrates multi-granularity knowledge for accurate category discovery.
  arXiv  Detail & Related papers  (2025-09-30T13:25:11Z)
• SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics [8.467086312715892]
  SCOPE is a diffusion-based category-level object pose estimation model.<n>It eliminates the need for discrete category labels by leveraging DINOv2 features as continuous semantic priors.<n>It achieves a relative improvement of 31.9% on the 5$circ$5cm metric.
  arXiv  Detail & Related papers  (2025-09-29T10:27:59Z)
• Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation [74.55677741919035]
  We propose Prior2Former (P2F), the first approach for segmentation vision transformers rooted in evidential learning.<n>P2F extends the mask vision transformer architecture by incorporating a Beta prior for computing model uncertainty in pixel-wise binary mask assignments.<n>Unlike most segmentation models addressing unknown classes, P2F operates without access to OOD data samples or contrastive training on void (i.e., unlabeled) classes.
  arXiv  Detail & Related papers  (2025-04-07T08:53:14Z)
• Lidar Panoptic Segmentation in an Open World [50.094491113541046]
  Lidar Panoptics (LPS) is crucial for safe deployment of autonomous vehicles. LPS aims to recognize and segment lidar points wr.t. a pre-defined vocabulary of semantic classes. We propose a class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification.
  arXiv  Detail & Related papers  (2024-09-22T00:10:20Z)
• Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detection [23.005760505169803]
  Open World Object Detection is a challenging and realistic task. It involves detecting both known and unknown objects. We propose Hyp-OW, a method that learns and models hierarchical representation of known items.
  arXiv  Detail & Related papers  (2023-06-25T16:45:20Z)
• Segmenting Known Objects and Unseen Unknowns without Prior Knowledge [86.46204148650328]
  holistic segmentation aims to identify and separate objects of unseen, unknown categories into instances without any prior knowledge about them. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. Experiments on public data from MS, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS.
  arXiv  Detail & Related papers  (2022-09-12T16:59:36Z)
• The Overlooked Classifier in Human-Object Interaction Recognition [82.20671129356037]
  We encode the semantic correlation among classes into the classification head by initializing the weights with language embeddings of HOIs. We propose a new loss named LSE-Sign to enhance multi-label learning on a long-tailed dataset. Our simple yet effective method enables detection-free HOI classification, outperforming the state-of-the-arts that require object detection and human pose by a clear margin.
  arXiv  Detail & Related papers  (2022-03-10T23:35:00Z)
• Hierarchical Image Classification using Entailment Cone Embeddings [68.82490011036263]
  We first inject label-hierarchy knowledge into an arbitrary CNN-based classifier. We empirically show that availability of such external semantic information in conjunction with the visual semantics from images boosts overall performance.
  arXiv  Detail & Related papers  (2020-04-02T10:22:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.