Visual symbolic mechanisms: Emergent symbol processing in vision language models

• URL: http://arxiv.org/abs/2506.15871v1
• Date: Wed, 18 Jun 2025 20:35:44 GMT
• Title: Visual symbolic mechanisms: Emergent symbol processing in vision language models
• Abstract summary: We identify a set of emergent symbolic mechanisms that support binding in vision language models (VLMs)<n>We find that binding errors can be traced directly to failures in these mechanisms.<n> Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for addressing the persistent binding failures exhibited by these models.
• Score: 1.0923877073891446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this 'binding problem' via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by vision language models (VLMs). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a set of emergent symbolic mechanisms that support binding in VLMs via a content-independent, spatial indexing scheme. Moreover, we find that binding errors can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for addressing the persistent binding failures exhibited by these models.

Related papers

• Linguistic Context Recodes Visual Representations in Vision-Language Models [20.8506186745408]
  We show that vision-language models (VLMs) can recode visual representations when presented with goal-directed language.<n>First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural language prompt.<n>Second, we demonstrate language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects.
  arXiv  Detail & Related papers  (2026-07-21T06:06:20Z)
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models [16.759528230678722]
  We identify a retrieval conditioned rebinding mechanism that encodes swap relevant binding information and reinstates it at readout.<n>Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families.
  arXiv  Detail & Related papers  (2026-06-07T14:16:44Z)
• The Geometry of Representational Failures in Vision Language Models [5.7337123720860435]
  Vision-Language Models (VLMs) exhibit puzzling failures in multi-object visual tasks.<n>These errors mirror human cognitive constraints, such as the "Binding Problem"<n>We propose a mechanistic insight by analyzing the representational geometry of open-weight VLMs.
  arXiv  Detail & Related papers  (2026-02-02T12:20:04Z)
• VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoning [69.64660280965971]
  VideoAnchor is a plug-and-play module that leverages subspace affinities to reinforce visual cues across frames without retraining.<n>We show consistent performance gains on benchmarks with InternVL2-8B and Q2.5VL-72B.<n>Our codes will be made public at https://github.com/feufhd/VideoAnchor.
  arXiv  Detail & Related papers  (2025-09-29T17:54:04Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability [2.1432646818975014]
  Vision-Language Models (VLMs) have shown remarkable performance in integrating visual and textual information for tasks such as image captioning and visual question answering.<n>These models struggle with compositional generalization and object binding.<n>Our work explores the root causes of these failures using mechanistic interpretability techniques.
  arXiv  Detail & Related papers  (2025-08-20T01:15:28Z)
• Investigating Mechanisms for In-Context Vision Language Binding [17.440725558041258]
  Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities.<n>For instance, given an 'image of a red toy car', the model should associate this image to phrases like 'car','red toy','red object', etc.
  arXiv  Detail & Related papers  (2025-05-28T10:25:43Z)
• CTRL-O: Language-Controllable Object-Centric Visual Representation Learning [30.218743514199016]
  Object-centric representation learning aims to decompose visual scenes into fixed-size vectors called "slots" or "object files"<n>Current object-centric models learn representations based on their preconceived understanding of objects, without allowing user input to guide which objects are represented.<n>We propose a novel approach for user-directed control over slot representations by conditioning slots on language descriptions.
  arXiv  Detail & Related papers  (2025-03-27T17:53:50Z)
• FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers [67.81792382350275]
  High-resolution visual input equips multimodal large language models (MLLMs) with enhanced visual perception capabilities for real-world tasks.<n>Most existing high-resolution MLLMs rely on a cropping-based approach to process images, which leads to fragmented visual encoding and a sharp increase in redundant tokens.<n>FALCON introduces a novel visual register technique to simultaneously eliminate redundant tokens at the stage of visual encoding.<n>FALCON demonstrates superior performance with a remarkable 9-fold reduction in visual tokens.
  arXiv  Detail & Related papers  (2025-01-27T18:36:10Z)
• ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives [109.11714588441511]
  The Ego-Exo object correspondence task aims to understand object relations across ego-exo perspectives through segmentation.<n> PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task.<n>We propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion and SSL-based Cross-View Object Alignment.
  arXiv  Detail & Related papers  (2024-11-28T12:01:03Z)
• Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem [37.27516441519387]
  We show that state-of-the-art vision language models exhibit surprising failures on basic multi-object reasoning tasks that humans perform with near perfect accuracy.<n>We find that many of the puzzling failures of state-of-the-art VLMs can be explained as arising due to the binding problem, and that these failure modes are strikingly similar to the limitations exhibited by rapid, feedforward processing in the human brain.
  arXiv  Detail & Related papers  (2024-10-31T22:24:47Z)
• Vision-Language Models Create Cross-Modal Task Representations [58.19152818504624]
  We find that vision-language models (VLMs) can align conceptually equivalent inputs into a shared task vector.<n>We measure this alignment via cross-modal transfer on a range of tasks and model architectures.<n>We show that task vectors can be transferred from a base language model to its fine-tuned vision-language counterpart.
  arXiv  Detail & Related papers  (2024-10-29T17:59:45Z)
• Interpreting the structure of multi-object representations in vision encoders [1.8749305679160366]
  We evaluate vision encoders pre-trained on classification, large vision-language models, and self-supervised methods.<n>We examine how object-wise representations are distributed across tokens and layers within these vision encoders.<n>Our findings highlight significant differences in the representation of objects depending on their relevance to the pre-training objective.
  arXiv  Detail & Related papers  (2024-06-13T12:54:20Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning [61.21923643289266]
  Chain of Manipulations is a mechanism that enables Vision-Language Models to solve problems step-by-step with evidence.<n>After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) actively without involving external tools.<n>Our trained model, textbfCogCoM, achieves state-of-the-art performance across 9 benchmarks from 4 categories.
  arXiv  Detail & Related papers  (2024-02-06T18:43:48Z)
• Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models [41.91490484827197]
  Injecting a collection of symbolic data directly into the training of Large Language Models can be problematic. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. Extensive experiments on both symbol- and NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models.
  arXiv  Detail & Related papers  (2023-11-15T18:59:56Z)
• How do Language Models Bind Entities in Context? [31.529253469951062]
  Language models (LMs) must bind entities to their attributes to correctly use in-context information. We identify the binding ID mechanism: a general mechanism for solving the binding problem. Our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context.
  arXiv  Detail & Related papers  (2023-10-26T07:10:31Z)
• Redundancy-aware Transformer for Video Question Answering [71.98116071679065]
  We propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions.
  arXiv  Detail & Related papers  (2023-08-07T03:16:24Z)
• Reusable Slotwise Mechanisms [45.48002329758044]
  We introduce Reusable Slotwise Mechanisms, or RSM, a framework that models object dynamics by leveraging communication among slots. RSM leverages the Central Contextual Information (CCI), enabling selected mechanisms to access the remaining slots. Experimental results demonstrate the superior performance of RSM compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-21T08:07:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.