What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation

• URL: http://arxiv.org/abs/2505.19569v1
• Date: Mon, 26 May 2025 06:33:48 GMT
• Title: What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation
• Authors: Jianghang Lin, Yue Hu, Jiangtao Shen, Yunhang Shen, Liujuan Cao, Shengchuan Zhang, Rongrong Ji,
• Abstract summary: Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time.<n>Existing paradigms typically perform class-agnostic region segmentation followed by category matching, leading to poor alignment between region segmentation and target concepts.<n>We propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process.
• Score: 65.80512502962071
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and target concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer Module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on target concepts. (3) A Cognition-Inspired Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching $27.2$ PQ, $17.0$ mAP, and $35.3$ mIoU on A-150. It further attains $56.2$, $28.2$, $15.4$, $59.2$, $18.7$, and $95.8$ mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free segmentation, offering enhanced flexibility in recognizing unseen categories. Code will be public.

Related papers

• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentation [9.759008308251127]
  Zero-shot referring image segmentation aims to locate and segment the target region based on a referring expression.<n>Previous works address this challenge by utilizing Vision-Language Models and mask proposal networks for region-text matching.<n>We present LGD (Leveraging Generative Descriptions), a framework that utilizes the advanced language generation capabilities of Multi-Modal Large Language Models.
  arXiv  Detail & Related papers  (2025-04-20T02:51:11Z)
• Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation [39.17707407384492]
  Generalized zero-shot semantic segmentation (GZS3) aims to achieve the human-level capability of segmenting seen and unseen classes.<n>We propose a Decoupled Vision-Language Matching (DeVLMatch) framework, composed of spatial-part (SPMatch) and channel-state (CSMatch) matching modules.
  arXiv  Detail & Related papers  (2025-03-31T07:39:14Z)
• Visual In-Context Prompting [100.93587329049848]
  In this paper, we introduce a universal visual in-context prompting framework for both vision tasks like open-set segmentation and detection. We build on top of an encoder-decoder architecture, and develop a versatile prompt encoder to support a variety of prompts like strokes, boxes, and points. Our extensive explorations show that the proposed visual in-context prompting elicits extraordinary referring and generic segmentation capabilities.
  arXiv  Detail & Related papers  (2023-11-22T18:59:48Z)
• CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection [78.0010542552784]
  CoDet is a novel approach to learn object-level vision-language representations for open-vocabulary object detection. By grouping images that mention a shared concept in their captions, objects corresponding to the shared concept shall exhibit high co-occurrence. CoDet has superior performances and compelling scalability in open-vocabulary detection.
  arXiv  Detail & Related papers  (2023-10-25T14:31:02Z)
• Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation [100.81837601210597]
  We propose Concept Curation (CoCu) to bridge the gap between visual and textual semantics in pre-training data. CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin.
  arXiv  Detail & Related papers  (2023-09-24T00:05:39Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.