Language-Guided Open-World Anomaly Segmentation

• URL: http://arxiv.org/abs/2512.01427v1
• Date: Mon, 01 Dec 2025 09:08:59 GMT
• Title: Language-Guided Open-World Anomaly Segmentation
• Authors: Klara Reichard, Nikolas Brasch, Nassir Navab, Federico Tombari,
• Abstract summary: We propose Clipomaly, the first CLIP-based open-world and anomaly segmentation method for autonomous driving.<n>Our zero-shot approach requires no anomaly-specific training data and leverages CLIP's shared image-text embedding space to segment unknown objects.<n>Our model dynamically extends its vocabulary at inference time without retraining, enabling robust detection and naming of anomalies beyond common class definitions.
• Score: 81.13615952207685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-world and anomaly segmentation methods seek to enable autonomous driving systems to detect and segment both known and unknown objects in real-world scenes. However, existing methods do not assign semantically meaningful labels to unknown regions, and distinguishing and learning representations for unknown classes remains difficult. While open-vocabulary segmentation methods show promise in generalizing to novel classes, they require a fixed inference vocabulary and thus cannot be directly applied to anomaly segmentation where unknown classes are unconstrained. We propose Clipomaly, the first CLIP-based open-world and anomaly segmentation method for autonomous driving. Our zero-shot approach requires no anomaly-specific training data and leverages CLIP's shared image-text embedding space to both segment unknown objects and assign human-interpretable names to them. Unlike open-vocabulary methods, our model dynamically extends its vocabulary at inference time without retraining, enabling robust detection and naming of anomalies beyond common class definitions such as those in Cityscapes. Clipomaly achieves state-of-the-art performance on established anomaly segmentation benchmarks while providing interpretability and flexibility essential for practical deployment.

Related papers

• Clustering is back: Reaching state-of-the-art LiDAR instance segmentation without training [69.2787246878521]
  We show that competitive panoptic segmentation can be achieved using only semantic labels.<n>Our method outperforms state-of-the-art supervised methods on standard benchmarks.<n>It is fully explainable, and requires no learning or parameter tuning.
  arXiv  Detail & Related papers  (2025-03-17T14:12:08Z)
• A Novel Decomposed Feature-Oriented Framework for Open-Set Semantic Segmentation on LiDAR Data [6.427051055902494]
  We propose a feature-oriented framework for open-set semantic segmentation on LiDAR data.<n>We design a dual-decoder network to simultaneously perform closed-set semantic segmentation and generate distinctive features for unknown objects.<n>By integrating the results of close-set semantic segmentation and anomaly detection, we achieve effective feature-driven LiDAR open-set semantic segmentation.
  arXiv  Detail & Related papers  (2025-03-14T05:40:05Z)
• From Open-Vocabulary to Vocabulary-Free Semantic Segmentation [78.62232202171919]
  Open-vocabulary semantic segmentation enables models to identify novel object categories beyond their training data.<n>Current approaches still rely on manually specified class names as input, creating an inherent bottleneck in real-world applications.<n>This work proposes a Vocabulary-Free Semantic pipeline, eliminating the need for predefined class vocabularies.
  arXiv  Detail & Related papers  (2025-02-17T15:17:08Z)
• Lidar Panoptic Segmentation in an Open World [50.094491113541046]
  Lidar Panoptics (LPS) is crucial for safe deployment of autonomous vehicles. LPS aims to recognize and segment lidar points wr.t. a pre-defined vocabulary of semantic classes. We propose a class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification.
  arXiv  Detail & Related papers  (2024-09-22T00:10:20Z)
• Auto-Vocabulary Semantic Segmentation [13.410217680999462]
  Open-Vocabulary (OVS) methods are capable of performing semantic segmentation without relying on a fixed vocabulary, and in some cases, without training or fine-tuning.<n>In this paper, we introduce Auto-Vocabulary Semantic (AVS), advancing open-ended image understanding by eliminating the necessity to predefine object for segmentation.<n>Our approach, AutoSeg, presents a framework that autonomously identifies relevant class names using semantically enhanced BLIP embeddings and segments them afterwards.
  arXiv  Detail & Related papers  (2023-12-07T18:55:52Z)
• CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free [12.15899043709721]
  We propose an open-vocabulary semantic segmentation method, dubbed CLIP-DIY. It exploits CLIP classification abilities on patches of different sizes and aggregates the decision in a single map. We obtain state-of-the-art zero-shot semantic segmentation results on PASCAL VOC and perform on par with the best methods on COCO.
  arXiv  Detail & Related papers  (2023-09-25T16:52:59Z)
• Global Knowledge Calibration for Fast Open-Vocabulary Segmentation [124.74256749281625]
  We introduce a text diversification strategy that generates a set of synonyms for each training category. We also employ a text-guided knowledge distillation method to preserve the generalizable knowledge of CLIP. Our proposed model achieves robust generalization performance across various datasets.
  arXiv  Detail & Related papers  (2023-03-16T09:51:41Z)
• Segmenting Known Objects and Unseen Unknowns without Prior Knowledge [86.46204148650328]
  holistic segmentation aims to identify and separate objects of unseen, unknown categories into instances without any prior knowledge about them. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. Experiments on public data from MS, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS.
  arXiv  Detail & Related papers  (2022-09-12T16:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.