Representational Analysis of Binding in Language Models

• URL: http://arxiv.org/abs/2409.05448v3
• Date: Fri, 25 Oct 2024 03:14:01 GMT
• Title: Representational Analysis of Binding in Language Models
• Abstract summary: Language models (LMs) must bind an entity to its attribute to recall attribute for a given entity. We show that there exists a low-rank subspace in the activations of LMs, that primarily encodes the order (i.e., OI) of entity and attribute. We also discover the causal effect of OI on binding that when editing representations along the OI encoding direction, LMs tend to bind a given entity to other attributes accordingly.
• Score: 21.567939573751254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, existing research introduces a Binding ID mechanism and states that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not captured the Ordering ID (OI) from entity activations that directly determines the binding behaviour. In this work, we provide a novel view of the BI mechanism by localizing OI and proving the causality between OI and binding behaviour. Specifically, by leveraging dimension reduction methods (e.g., PCA), we discover that there exists a low-rank subspace in the activations of LMs, that primarily encodes the order (i.e., OI) of entity and attribute. Moreover, we also discover the causal effect of OI on binding that when editing representations along the OI encoding direction, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the OI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.

Related papers

• A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models [16.759528230678722]
  We identify a retrieval conditioned rebinding mechanism that encodes swap relevant binding information and reinstates it at readout.<n>Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families.
  arXiv  Detail & Related papers  (2026-06-07T14:16:44Z)
• SAM-NER: Semantic Archetype Mediation for Zero-Shot Named Entity Recognition [50.82878172248818]
  emphSemantic Archetype Mediation stabilizes cross-domain transfer through an intermediate, domain-invariant archetype space.<n> SAM-NER consistently outperforms strong prior ZS-NER baselines in cross-domain settings.
  arXiv  Detail & Related papers  (2026-05-05T12:54:17Z)
• On the Representations of Entities in Auto-regressive Large Language Models [10.08529318994409]
  We introduce entity mention reconstruction as a novel framework for studying how Large Language Models (LLMs) encode and manipulate entities.<n>We investigate whether entity mentions can be generated from internal representations, how multi-token entities are encoded beyond last-token embeddings, and whether these representations capture relational knowledge.
  arXiv  Detail & Related papers  (2025-10-10T14:23:44Z)
• Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context [33.223631694438]
  A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval.<n>We show that this mechanism generalizes poorly to more complex settings.<n>We develop a causal model combining all three mechanisms that estimates next token distributions with 95% agreement.
  arXiv  Detail & Related papers  (2025-10-07T17:44:30Z)
• Align Your Query: Representation Alignment for Multimodality Medical Object Detection [55.86070915426998]
  We propose a detector-agnostic framework to align representations with modality context.<n>We integrate modality tokens into the detection process via Multimodality Context Attention.<n>The proposed approach consistently improves AP with minimal overhead and no architectural modifications.
  arXiv  Detail & Related papers  (2025-10-03T07:49:21Z)
• Investigating Mechanisms for In-Context Vision Language Binding [17.440725558041258]
  Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities.<n>For instance, given an 'image of a red toy car', the model should associate this image to phrases like 'car','red toy','red object', etc.
  arXiv  Detail & Related papers  (2025-05-28T10:25:43Z)
• Language Models use Lookbacks to Track Beliefs [51.92689607136242]
  We analyze Llama-3-70B-Instruct's ability to reason about characters' beliefs using causal mediation and abstraction.<n>Our work provides insights into the LM's belief tracking mechanisms, taking a step toward reverse-engineering ToM reasoning in LMs.
  arXiv  Detail & Related papers  (2025-05-20T17:59:45Z)
• Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis [98.21700880115938]
  Text-to-image (T2I) models often fail to accurately bind semantically related objects or attributes in the input prompts. We introduce a novel method called Token Merging (ToMe), which enhances semantic binding by aggregating relevant tokens into a single composite token.
  arXiv  Detail & Related papers  (2024-11-11T17:05:15Z)
• Editing Arbitrary Propositions in LLMs without Subject Labels [88.67755930096966]
  We introduce a simple and fast localization method called Gradient Tracing (GT) GT allows editing arbitrary propositions instead of just binary ones, and does so without the need for subject labels. We show that our method -- without access to subject labels -- performs close to state-of-the-art L&E methods which has access subject labels.
  arXiv  Detail & Related papers  (2024-01-15T08:08:24Z)
• How do Language Models Bind Entities in Context? [31.529253469951062]
  Language models (LMs) must bind entities to their attributes to correctly use in-context information. We identify the binding ID mechanism: a general mechanism for solving the binding problem. Our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context.
  arXiv  Detail & Related papers  (2023-10-26T07:10:31Z)
• Object-Centric Multiple Object Tracking [124.30650395969126]
  This paper proposes a video object-centric model for multiple-object tracking pipelines. It consists of an index-merge module that adapts the object-centric slots into detection outputs and an object memory module. Benefited from object-centric learning, we only require sparse detection labels for object localization and feature binding.
  arXiv  Detail & Related papers  (2023-09-01T03:34:12Z)
• From Alignment to Entailment: A Unified Textual Entailment Framework for Entity Alignment [17.70562397382911]
  Existing methods usually encode the triples of entities as embeddings and learn to align the embeddings. We transform both triples into unified textual sequences, and model the EA task as a bi-directional textual entailment task. Our approach captures the unified correlation pattern of two kinds of information between entities, and explicitly models the fine-grained interaction between original entity information.
  arXiv  Detail & Related papers  (2023-05-19T08:06:50Z)
• Entity Cloze By Date: What LMs Know About Unseen Entities [79.34707800653597]
  Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity.
  arXiv  Detail & Related papers  (2022-05-05T17:59:31Z)
• UniRE: A Unified Label Space for Entity Relation Extraction [67.53850477281058]
  Joint entity relation extraction models setup two separated label spaces for the two sub-tasks. We argue that this setting may hinder the information interaction between entities and relations. In this work, we propose to eliminate the different treatment on the two sub-tasks' label spaces.
  arXiv  Detail & Related papers  (2021-07-09T08:09:37Z)
• From Bag of Sentences to Document: Distantly Supervised Relation Extraction via Machine Reading Comprehension [22.39362905658063]
  We propose a new DS paradigm--document-based distant supervision. We show that our method achieves new state-of-the-art DS performance.
  arXiv  Detail & Related papers  (2020-12-08T10:16:27Z)
• Autoregressive Entity Retrieval [55.38027440347138]
  Entities are at the center of how we represent and aggregate knowledge. The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. We propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion.
  arXiv  Detail & Related papers  (2020-10-02T10:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.