Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

• URL: http://arxiv.org/abs/2604.07723v1
• Date: Thu, 09 Apr 2026 02:14:35 GMT
• Title: Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
• Abstract summary: Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts.<n>We propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map.
• Score: 48.806000388608005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.

Related papers

• Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology [58.17907376475596]
  State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion.<n>InSeIn extracts explicit inclusion constraints that govern spatial class relations from the semantic segmentation training set at hand.<n>It then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility.
  arXiv  Detail & Related papers  (2024-08-26T22:39:08Z)
• Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations [15.59251297818324]
  We present an approach for analyzing grouping information contained within a neural network's activations. We exploit features from all layers and obviating the need to guess which part of the model contains relevant information.
  arXiv  Detail & Related papers  (2023-12-11T01:20:34Z)
• Stochastic Segmentation with Conditional Categorical Diffusion Models [3.8168879948759953]
  We propose a conditional categorical diffusion model (CCDM) for semantic segmentation based on Denoising Diffusion Probabilistic Models. Our results show that CCDM achieves state-of-the-art performance on LIDC, and outperforms established baselines on the classical segmentation dataset Cityscapes.
  arXiv  Detail & Related papers  (2023-03-15T19:16:47Z)
• Deterministic Decoupling of Global Features and its Application to Data Analysis [0.0]
  We propose a new formalism that is based on defining transformations on submanifolds. Through these transformations we define a normalization that, we demonstrate, allows for decoupling differentiable features. We apply this method in the original data domain and at the output of a filter bank to regression and classification problems based on global descriptors.
  arXiv  Detail & Related papers  (2022-07-05T15:54:39Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Semantic Distribution-aware Contrastive Adaptation for Semantic Segmentation [50.621269117524925]
  Domain adaptive semantic segmentation refers to making predictions on a certain target domain with only annotations of a specific source domain. We present a semantic distribution-aware contrastive adaptation algorithm that enables pixel-wise representation alignment. We evaluate SDCA on multiple benchmarks, achieving considerable improvements over existing algorithms.
  arXiv  Detail & Related papers  (2021-05-11T13:21:25Z)
• Latent Space Regularization for Unsupervised Domain Adaptation in Semantic Segmentation [14.050836886292869]
  We introduce feature-level space-shaping regularization strategies to reduce the domain discrepancy in semantic segmentation. We verify the effectiveness of such methods in the autonomous driving setting.
  arXiv  Detail & Related papers  (2021-04-06T16:07:22Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)
• FDA: Fourier Domain Adaptation for Semantic Segmentation [82.4963423086097]
  We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other. We illustrate the method in semantic segmentation, where densely annotated images are aplenty in one domain, but difficult to obtain in another. Our results indicate that even simple procedures can discount nuisance variability in the data that more sophisticated methods struggle to learn away.
  arXiv  Detail & Related papers  (2020-04-11T22:20:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.