A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models

• URL: http://arxiv.org/abs/2606.08644v1
• Date: Sun, 07 Jun 2026 14:16:44 GMT
• Title: A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models
• Abstract summary: We identify a retrieval conditioned rebinding mechanism that encodes swap relevant binding information and reinstates it at readout.<n>Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families.
• Score: 16.759528230678722
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To interpret context correctly and retrieve relevant information, large language models must bind entities to their attributes and update these bindings as state changes. We analyze how LLMs implement this binding process in a dynamic state tracking. Using causal interventions, we identify a retrieval conditioned rebinding mechanism, a compact attention head circuit that encodes swap relevant binding information and reinstates it at readout. Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families. In Gemma models, the binding signature is clearly expressed in the query/key subspaces of the relevant attention heads, whereas in Llama models, the binding information is carried primarily in key vectors. Overall, our results reveal an interpretable mechanism for context dependent state tracking in LLMs.

Related papers

• Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts [16.43008975197273]
  We investigate how vision-language models handle context-memory conflicts.<n>Models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images.
  arXiv  Detail & Related papers  (2026-08-31T19:37:07Z)
• ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships [53.24877477312131]
  ReBind is a systematic framework that introduces semantic instructions with embedded reference tokens.<n>We develop ReBind-Instruct, a specialized MLLM that learns to establish explicit bindings between visual attributes and their reference sources.<n>We also develop ReBind-Edit, which enables lightweight adaptation of text-to-video models to coordinate multiple references.
  arXiv  Detail & Related papers  (2026-07-16T07:43:27Z)
• Weight Patching: Toward Source-Level Mechanistic Localization in LLMs [12.558998651017374]
  We propose Weight Patching, a parameter-space intervention method for source-oriented analysis in paired same-architecture models.<n>We introduce a framework centered on a vector-anchor behavioral interface that provides a shared internal criterion for whether a task-relevant control state has been formed.
  arXiv  Detail & Related papers  (2026-04-15T10:21:38Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation [2.1521364454860525]
  We introduce a lightweight, interpretable control framework that embeds a model-based feedback loop directly within decoding.<n>We show that a PID controller dynamically modulates attention heads to maintain factual consistency without retraining or multi-pass decoding.
  arXiv  Detail & Related papers  (2025-11-05T05:30:28Z)
• Visual symbolic mechanisms: Emergent symbol processing in vision language models [1.0923877073891446]
  We identify a set of emergent symbolic mechanisms that support binding in vision language models (VLMs)<n>We find that binding errors can be traced directly to failures in these mechanisms.<n> Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for addressing the persistent binding failures exhibited by these models.
  arXiv  Detail & Related papers  (2025-06-18T20:35:44Z)
• Large Language Models are Good Relational Learners [55.40941576497973]
  We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for large language models (LLMs)<n>Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to process and reason over complex entity relationships.
  arXiv  Detail & Related papers  (2025-06-06T04:07:55Z)
• Core Context Aware Transformers for Long Context Language Modeling [50.774702091154204]
  We propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling.<n>Our method automatically focuses and strengthens core context while diminishing redundancy during the learning process.<n>Our method is able to replace the self-attention module in existing Large Language Models with minimal fine-tuning cost.
  arXiv  Detail & Related papers  (2024-12-17T01:54:08Z)
• Talking Heads: Understanding Inter-layer Communication in Transformer Language Models [32.2976613483151]
  We analyze a mechanism used in two LMs to selectively inhibit items in a context in one task.<n>We find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers.
  arXiv  Detail & Related papers  (2024-06-13T18:12:01Z)
• Relation Rectification in Diffusion Model [64.84686527988809]
  We introduce a novel task termed Relation Rectification, aiming to refine the model to accurately represent a given relationship it initially fails to generate. We propose an innovative solution utilizing a Heterogeneous Graph Convolutional Network (HGCN) The lightweight HGCN adjusts the text embeddings generated by the text encoder, ensuring the accurate reflection of the textual relation in the embedding space.
  arXiv  Detail & Related papers  (2024-03-29T15:54:36Z)
• How do Language Models Bind Entities in Context? [31.529253469951062]
  Language models (LMs) must bind entities to their attributes to correctly use in-context information. We identify the binding ID mechanism: a general mechanism for solving the binding problem. Our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context.
  arXiv  Detail & Related papers  (2023-10-26T07:10:31Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.