Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation

• URL: http://arxiv.org/abs/2605.15942v1
• Date: Fri, 15 May 2026 13:27:01 GMT
• Title: Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
• Abstract summary: Open-vocabulary segmentation models often struggle to generalize to unseen combinations of categories object and attributes.<n>We propose a Decomposed Vision-Language Alignment framework that factorizes textual prompts into a concept token and multiple attribute tokens.<n>At the feature level, we introduce a Feature-Gated Cross-Attention module that generates attribute-specific gating maps to fuse information in a multiplicative manner.
• Score: 16.594496639787195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary segmentation models often struggle to generalize to unseen combinations of object categories and attributes, because fine-grained descriptions are typically encoded as holistic sentences that entangle multiple semantic units. We propose a Decomposed Vision-Language Alignment framework that explicitly factorizes textual prompts into a concept token and multiple attribute tokens, enabling separate cross-modal interactions for each semantic unit. At the feature level, we introduce a Feature-Gated Cross-Attention module that generates attribute-specific gating maps to fuse information in a multiplicative manner, effectively enforcing compositional semantics. At the scoring level, per-token similarities are aggregated in log-space, producing a stable and interpretable compositional matching. The method can be seamlessly integrated into existing transformer-based segmentation architectures and significantly improves generalization to unseen attribute-category compositions in fine-grained open-vocabulary segmentation benchmarks.

Related papers

• Does Syntax Matter? A Graph-Augmented Variational Topic Model for Computational Social Sciences [51.56484100374058]
  This paper introduces the Structural Contextual Probabilistic Topic Model ( SCPTM), an architecture that incorporates syntactic dependency relations into topic inference.<n>We evaluate seven topic modeling techniques across four corpora differing in register and discourse structure.<n>Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines.
  arXiv  Detail & Related papers  (2026-09-07T17:39:06Z)
• Controlling Embedding Spaces with Text-Conditioned Transformations [54.94594915955756]
  We propose a text-conditioned transformation of visual embeddings that makes attributes explicitly accessible.<n> Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface.
  arXiv  Detail & Related papers  (2026-07-24T21:14:29Z)
• Token-to-Token Alignment of Text Embeddings for Semantic Blending [55.88554156801394]
  We introduce Token-to-Token alignment, a framework that establishes explicit semantic correspondence between tokens across prompts.<n>Our results show that text embedding spaces in text-to-image models implicitly encode a continuous semantic structure that becomes accessible once representations are properly aligned.
  arXiv  Detail & Related papers  (2026-06-22T23:54:40Z)
• Learning a Semantic Calibration Network for Open-Vocabulary Semantic Segmentation [5.122331812021513]
  Open-Vocabulary (OVS) extends the segmentation task from a fixed set to an open set.<n>We propose a novel Semantic Network (SCN) for open-vocabulary semantic segmentation.<n>The proposed method achieves significant performance improvements compared to state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2026-06-06T06:42:03Z)
• Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representation [11.279399724798383]
  NAVI is a segment-centric pretraining framework that treats each header-value pair as the unit for aggregating schema-level structural evidence and column-level distributional evidence.<n>Masked Segment Modeling and Entropy-driven Segment Alignment jointly enforce structured header-value coupling and semantic alignment across stable and instance-specific attributes.
  arXiv  Detail & Related papers  (2026-06-01T08:34:25Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learning [86.58227205147546]
  The goal of Compositional Zero-Shot Learning (OV-CZSL) is to recognize iteration-object compositions in the open-vocabulary setting.<n>We propose Structure-aware Prompt Adaptation (SPA) method, which enables models to generalize from seen to unseen attributes and objects.
  arXiv  Detail & Related papers  (2026-03-04T07:54:28Z)
• Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence [6.991281327290525]
  Language models lack the notion of interchangeable tokens.<n>We formalize this machine learning problem and introduce alpha-covariance.<n>Our findings establish a foundation for designing language models that can learn interchangeable token representations.
  arXiv  Detail & Related papers  (2024-10-22T16:34:36Z)
• AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation [33.25304533086283]
  Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent studies have explored vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios. This work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts.
  arXiv  Detail & Related papers  (2023-08-31T19:34:09Z)
• DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment [124.57488600605822]
  Cross-modal garment synthesis and manipulation will significantly benefit the way fashion designers generate garments. We introduce DiffCloth, a diffusion-based pipeline for cross-modal garment synthesis and manipulation. Experiments on the CM-Fashion benchmark demonstrate that DiffCloth both yields state-of-the-art garment synthesis results.
  arXiv  Detail & Related papers  (2023-08-22T05:43:33Z)
• Hierarchical Open-vocabulary Universal Image Segmentation [48.008887320870244]
  Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. We propose a decoupled text-image fusion mechanism and representation learning modules for both "things" and "stuff" Our resulting model, named HIPIE tackles, HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework.
  arXiv  Detail & Related papers  (2023-07-03T06:02:15Z)
• Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation [13.001629605405954]
  We study universal zero-shot segmentation in this work to achieve panoptic, instance, and semantic segmentation for novel categories without any training samples. We introduce a generative model to synthesize features for unseen categories, which links semantic and visual spaces. The proposed approach achieves impressively state-of-the-art performance on zero-shot panoptic segmentation, instance segmentation, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-19T17:59:16Z)
• Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations [80.45474362071236]
  It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings.
  arXiv  Detail & Related papers  (2023-05-24T00:44:49Z)
• Framework-agnostic Semantically-aware Global Reasoning for Segmentation [29.69187816377079]
  We propose a component that learns to project image features into latent representations and reason between them. Our design encourages the latent regions to represent semantic concepts by ensuring that the activated regions are spatially disjoint. Our latent tokens are semantically interpretable and diverse and provide a rich set of features that can be transferred to downstream tasks.
  arXiv  Detail & Related papers  (2022-12-06T21:42:05Z)
• Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification [78.83284164605473]
  emphFunnelling (Fun) is a recently proposed method for cross-lingual text classification. We describe emphGeneralized Funnelling (gFun) as a generalization of Fun. We show that gFun substantially improves over Fun and over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-09-17T23:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.