Cell-Based Representation of Relational Binding in Language Models

• URL: http://arxiv.org/abs/2604.19052v1
• Date: Tue, 21 Apr 2026 03:58:47 GMT
• Title: Cell-Based Representation of Relational Binding in Language Models
• Abstract summary: We study discourse-level relational binding and show that Large Language Models encode it via a Cell-based Binding Representation (CBR)<n>We identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression.<n>We find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer.
• Score: 26.59650841520019
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear. We study discourse-level relational binding and show that LLMs encode it via a Cell-based Binding Representation (CBR): a low-dimensional linear subspace in which each ``cell'' corresponds to an entity--relation index pair, and bound attributes are retrieved from the corresponding cell during inference. Using controlled multi-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression. Across domains and two model families, the indices are linearly decodable and form a grid-like geometry in the projected space. We further find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding.

Related papers

• Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy [4.794327378667608]
  Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations.<n>They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus.<n>Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately.
  arXiv  Detail & Related papers  (2026-08-28T01:17:18Z)
• Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation [54.74045834035952]
  Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs.<n>We propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control.<n>RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.
  arXiv  Detail & Related papers  (2026-06-28T15:41:31Z)
• Generating Logically Consistent Synthetic Supply Chain Data with LLM-Driven Knowledge Graph Reasoning [51.78065387034195]
  This paper introduces textbftextitTabKG, a knowledge-graph-guided framework for logically consistent synthetic supply chain data generation.
  arXiv  Detail & Related papers  (2026-05-26T10:41:37Z)
• Geometric Factual Recall in Transformers [57.48371649045765]
  A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring memorization counts that scale linearly with the number of facts.<n>We develop a theoretical and empirical account of an alternative, emphgeometric form of memorization in which learned embeddings encode structure directly.<n>In a controlled setting where a single-layer transformer must memorize random vectors from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices.<n>We extend these results to the multi-hop setting, providing constructions with and without chain-of-thought that exhibit a provable
  arXiv  Detail & Related papers  (2026-05-12T17:22:22Z)
• Empowering Heterogeneous Graph Foundation Models via Decoupled Relation Alignment [20.128308877213044]
  Decoupled relation Subspace Alignment (DRSA) is a novel, plug-and-play relation-driven framework.<n>It fundamentally shifts the paradigm by decoupling feature semantics from relation structures.<n>It can be seamlessly integrated as a universal preprocessing module.
  arXiv  Detail & Related papers  (2026-05-01T15:31:36Z)
• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction [6.478238734128006]
  Relation extraction (RE) aims to identify semantic relations between entities in unstructured text.<n>underlineRetrieval underlineOver underlineClassification (ROC) is a novel framework that reformulates multimodal RE as a retrieval task driven by relation semantics.<n>ROC integrates entity type and positional information through a multimodal encoder, expands relation labels into natural language descriptions using a large language model, and aligns entity-relation pairs via semantic similarity-based contrastive learning.
  arXiv  Detail & Related papers  (2025-09-25T13:38:38Z)
• Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck [0.0]
  We develop a principled topic identification method grounded in the Deterministic Information Bottleneck (DIB) for geometric clustering.<n>Our key contribution is to transform the DIB method into a practical algorithm for high-dimensional data by substituting its intractable KL divergence term with a computationally efficient upper bound.
  arXiv  Detail & Related papers  (2025-08-26T20:00:51Z)
• Large Language Models are Good Relational Learners [55.40941576497973]
  We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for large language models (LLMs)<n>Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to process and reason over complex entity relationships.
  arXiv  Detail & Related papers  (2025-06-06T04:07:55Z)
• Induced Covariance for Causal Discovery in Linear Sparse Structures [55.2480439325792]
  Causal models seek to unravel the cause-effect relationships among variables from observed data. This paper introduces a novel causal discovery algorithm designed for settings in which variables exhibit linearly sparse relationships.
  arXiv  Detail & Related papers  (2024-10-02T04:01:38Z)
• Linearity of Relation Decoding in Transformer Language Models [82.47019600662874]
  Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations. We show that, for a subset of relations, this computation is well-approximated by a single linear transformation on the subject representation.
  arXiv  Detail & Related papers  (2023-08-17T17:59:19Z)
• The Low-Dimensional Linear Geometry of Contextualized Word Representations [27.50785941238007]
  We study the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features are encoded in low-dimensional subspaces.
  arXiv  Detail & Related papers  (2021-05-15T00:58:08Z)
• TransINT: Embedding Implication Rules in Knowledge Graphs with Isomorphic Intersections of Linear Subspaces [10.79392871079383]
  We propose TransINT, a novel embedding method for Knowledge Graphs. TransINT maps entities (tied by a relation) to continuous sets of vectors that are inclusion-ordered isomorphically to relation implications. On a benchmark dataset, we outperform the best existing state-of-the-art rule integration embedding methods with significant margins in link Prediction and triple Classification.
  arXiv  Detail & Related papers  (2020-07-01T06:45:27Z)
• New advances in enumerative biclustering algorithms with online partitioning [80.22629846165306]
  This paper further extends RIn-Close_CVC, a biclustering algorithm capable of performing an efficient, complete, correct and non-redundant enumeration of maximal biclusters with constant values on columns in numerical datasets. The improved algorithm is called RIn-Close_CVC3, keeps those attractive properties of RIn-Close_CVC, and is characterized by: a drastic reduction in memory usage; a consistent gain in runtime.
  arXiv  Detail & Related papers  (2020-03-07T14:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.