Vision-Language Grounding as Bidirectional Concept Correspondence

• URL: http://arxiv.org/abs/2608.07886v1
• Date: Sat, 08 Aug 2026 03:26:38 GMT
• Title: Vision-Language Grounding as Bidirectional Concept Correspondence
• Abstract summary: Vision-language grounding connects language to visual content.<n>We formulate grounding as $textitbidirectional concept correspondence$ over an image-text pair.<n>We introduce $textbfConCor-1$, a grounding model built on top of a pretrained vision-language model.
• Score: 84.8022126995569
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.

Related papers

• SemImage: Semantic Image Representation for Text, a Novel Framework for Embedding Disentangled Linguistic Features [0.0]
  SemImage is a novel method for representing a text document as a two-dimensional semantic image to be processed by convolutional neural networks (CNNs)<n>In a SemImage, each word is represented as a pixel in a 2D image: rows correspond to sentences and an additional boundary row is inserted between sentences to mark semantic transitions.
  arXiv  Detail & Related papers  (2025-11-26T12:58:21Z)
• UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation [51.31795451147935]
  We present a unified generative model that supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework.<n>Our goal is to achieve unification along three axes: the model, the tasks, and the representations.<n> Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment.
  arXiv  Detail & Related papers  (2025-11-21T03:02:10Z)
• Text Augmented Spatial-aware Zero-shot Referring Image Segmentation [60.84423786769453]
  We introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial for mask post-processing. The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.
  arXiv  Detail & Related papers  (2023-10-27T10:52:50Z)
• Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency [47.3163261953469]
  Current vision-language generative models rely on expansive corpora of paired image-text data to attain optimal performance and generalization capabilities. We introduce ITIT: an innovative training paradigm grounded in the concept of cycle consistency which allows vision-language training on unpaired image and text data. ITIT is comprised of a joint image-text encoder with disjoint image and text decoders that enable bidirectional image-to-text and text-to-image generation in a single framework.
  arXiv  Detail & Related papers  (2023-10-05T17:55:19Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval [12.057465578064345]
  The goal of Text-to-Image Person Retrieval (TIPR) is to retrieve specific person images according to the given textual descriptions.<n>We propose a novel TIPR framework to build fine-grained interactions and alignment between person images and the corresponding texts.
  arXiv  Detail & Related papers  (2023-07-18T08:23:46Z)
• Text-based Person Search without Parallel Image-Text Data [52.63433741872629]
  Text-based person search (TBPS) aims to retrieve the images of the target person from a large image gallery based on a given natural language description. Existing methods are dominated by training models with parallel image-text pairs, which are very costly to collect. In this paper, we make the first attempt to explore TBPS without parallel image-text data.
  arXiv  Detail & Related papers  (2023-05-22T12:13:08Z)
• Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA [15.74007067413724]
  We propose a novel framework for Scene Text Visual Question Answering (STVQA) It requires models to read scene text in images for question answering.
  arXiv  Detail & Related papers  (2023-04-04T07:46:40Z)
• Learning Object-Language Alignments for Open-Vocabulary Object Detection [83.09560814244524]
  We propose a novel open-vocabulary object detection framework directly learning from image-text pair data. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way.
  arXiv  Detail & Related papers  (2022-11-27T14:47:31Z)
• Visual Grounding Strategies for Text-Only Natural Language Processing [1.2183405753834562]
  multimodal extensions of BERT allow a joint modeling of texts and images that lead to state-of-the-art results on multimodal tasks such as Visual Question Answering. Here, we leverage multimodal modeling for purely textual tasks with the expectation that the multimodal pretraining provides a grounding that can improve text processing accuracy. A first type of strategy, referred to as it transferred grounding consists in applying multimodal models to text-only tasks using a placeholder to replace image input. The second one, which we call it associative grounding, harnesses image retrieval to match texts with related images during both
  arXiv  Detail & Related papers  (2021-03-25T16:03:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.