Investigating Mechanisms for In-Context Vision Language Binding

• URL: http://arxiv.org/abs/2505.22200v1
• Date: Wed, 28 May 2025 10:25:43 GMT
• Title: Investigating Mechanisms for In-Context Vision Language Binding
• Abstract summary: Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities.<n>For instance, given an 'image of a red toy car', the model should associate this image to phrases like 'car','red toy','red object', etc.
• Score: 17.440725558041258
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: To understand a prompt, Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities. For instance, given an 'image of a red toy car', the model should associate this image to phrases like 'car', 'red toy', 'red object', etc. Feng and Steinhardt propose the Binding ID mechanism in LLMs, suggesting that the entity and its corresponding attribute tokens share a Binding ID in the model activations. We investigate this for image-text binding in VLMs using a synthetic dataset and task that requires models to associate 3D objects in an image with their descriptions in the text. Our experiments demonstrate that VLMs assign a distinct Binding ID to an object's image tokens and its textual references, enabling in-context association.

Related papers

• Fine-Grained Zero-Shot Composed Image Retrieval with Complementary Visual-Semantic Integration [64.12127577975696]
  Zero-shot composed image retrieval (ZS-CIR) is a rapidly growing area with significant practical applications.<n>Existing ZS-CIR methods often struggle to capture fine-grained changes and integrate visual and semantic information effectively.<n>We propose a novel Fine-Grained Zero-Shot Composed Image Retrieval method with Complementary Visual-Semantic Integration.
  arXiv  Detail & Related papers  (2026-01-20T15:17:14Z)
• Personalizing Retrieval using Joint Embeddings or "the Return of Fluffy" [55.07411490538404]
  We design a mapping network that can "translate" from a local image embedding (of the object instance) to a text token.<n>We show that our approach of using a trainable mapping network, termed pi-map, together with frozen CLIP text and image encoders, improves the state of the art on two benchmarks designed to assess personalized retrieval.
  arXiv  Detail & Related papers  (2025-10-06T22:08:30Z)
• Visual symbolic mechanisms: Emergent symbol processing in vision language models [1.0923877073891446]
  We identify a set of emergent symbolic mechanisms that support binding in vision language models (VLMs)<n>We find that binding errors can be traced directly to failures in these mechanisms.<n> Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for addressing the persistent binding failures exhibited by these models.
  arXiv  Detail & Related papers  (2025-06-18T20:35:44Z)
• How Can Objects Help Video-Language Understanding? [16.63183488540909]
  We introduce ObjectML, a framework capable of leveraging arbitrary computer vision algorithm to extract and structured visual representation.<n>Through extensive evaluations on six video question benchmarks, we confirm that explicit integration of object-centric representation remains necessary.<n>Surprisingly, we observe that the simple approach quantizing the continuous, structured object information and representing them as plain text performs the best.
  arXiv  Detail & Related papers  (2025-04-10T04:59:28Z)
• Composing Object Relations and Attributes for Image-Text Matching [70.47747937665987]
  This work introduces a dual-encoder image-text matching model, leveraging a scene graph to represent captions with nodes for objects and attributes interconnected by relational edges. Our model efficiently encodes object-attribute and object-object semantic relations, resulting in a robust and fast-performing system.
  arXiv  Detail & Related papers  (2024-06-17T17:56:01Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval [53.89454443114146]
  We study the zero-shot Composed Image Retrieval (ZS-CIR) task, which is to retrieve the target image given a reference image and a description without training on the triplet datasets. Previous works generate pseudo-word tokens by projecting the reference image features to the text embedding space. We propose a Knowledge-Enhanced Dual-stream zero-shot composed image retrieval framework (KEDs) KEDs implicitly models the attributes of the reference images by incorporating a database.
  arXiv  Detail & Related papers  (2024-03-24T04:23:56Z)
• How do Language Models Bind Entities in Context? [31.529253469951062]
  Language models (LMs) must bind entities to their attributes to correctly use in-context information. We identify the binding ID mechanism: a general mechanism for solving the binding problem. Our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context.
  arXiv  Detail & Related papers  (2023-10-26T07:10:31Z)
• Learning Comprehensive Representations with Richer Self for Text-to-Image Person Re-Identification [34.289949134802086]
  Text-to-image person re-identification (TIReID) retrieves pedestrian images of the same identity based on a query text. Existing methods for TIReID typically treat it as a one-to-one image-text matching problem, only focusing on the relationship between image-text pairs within a view. We propose a framework, called LCR$2$S, for modeling many-to-many correspondences of the same identity by learning representations for both modalities from a novel perspective.
  arXiv  Detail & Related papers  (2023-10-17T12:39:16Z)
• Beyond One-to-One: Rethinking the Referring Image Segmentation [117.53010476628029]
  Referring image segmentation aims to segment the target object referred by a natural language expression. We propose a Dual Multi-Modal Interaction (DMMI) Network, which contains two decoder branches. In the text-to-image decoder, text embedding is utilized to query the visual feature and localize the corresponding target. Meanwhile, the image-to-text decoder is implemented to reconstruct the erased entity-phrase conditioned on the visual feature.
  arXiv  Detail & Related papers  (2023-08-26T11:39:22Z)
• Focus! Relevant and Sufficient Context Selection for News Image Captioning [69.36678144800936]
  News Image Captioning requires describing an image by leveraging additional context from a news article. We propose to use the pre-trained vision and language retrieval model CLIP to localize the visually grounded entities in the news article. Our experiments demonstrate that by simply selecting a better context from the article, we can significantly improve the performance of existing models.
  arXiv  Detail & Related papers  (2022-12-01T20:00:27Z)
• Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching [102.62343739435289]
  Existing image-text matching approaches infer the similarity of an image-text pair by capturing and aggregating the affinities between the text and each independent object of the image. We propose a Dual Path Recurrent Neural Network (DP-RNN) which processes images and sentences symmetrically by recurrent neural networks (RNN) Our model achieves the state-of-the-art performance on Flickr30K dataset and competitive performance on MS-COCO dataset.
  arXiv  Detail & Related papers  (2020-02-20T00:51:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.