Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy

• URL: http://arxiv.org/abs/2608.27813v1
• Date: Fri, 28 Aug 2026 01:17:18 GMT
• Title: Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy
• Abstract summary: Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations.<n>They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus.<n>Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately.
• Score: 4.794327378667608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.

Related papers

• See More, Match Better: Multi-Source Feature Fusion for Two-View Correspondence Learning [52.52121389921707]
  Two-view correspondence learning aims to distinguish true correspondences (inliers) from false ones (outliers) in image pairs.<n>Existing methods mainly rely on coordinate-based geometric consistency.<n>We propose TriMatch, a multi-source feature fusion framework for two-view correspondence learning.
  arXiv  Detail & Related papers  (2026-06-08T09:33:01Z)
• Barycentric alignment for instance-level comparison of neural representations [2.1920579994942164]
  We introduce a barycentric alignment framework that quotients out nuisance symmetries to construct a universal embedding space across many models.<n>We identify systematic input properties that predict representational convergence versus divergence across vision and language model families.<n>We also apply the same barycentric alignment framework to purely unimodal vision and language models and find that post-hoc alignment into a shared space yields image text similarity scores.
  arXiv  Detail & Related papers  (2026-02-09T21:49:44Z)
• Sparsification and Reconstruction from the Perspective of Representation Geometry [10.834177456685538]
  Sparse Autoencoders (SAEs) have emerged as a predominant tool in mechanistic interpretability.<n>This study explains the principles of sparsity from the perspective of representational geometry.<n>Specifically emphasizes the necessity of understanding representations and incorporating representational constraints.
  arXiv  Detail & Related papers  (2025-05-28T15:54:33Z)
• Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings [0.0]
  This paper proposes a framework for uncovering the specific dimensions of vector embeddings that encode distinct linguistic properties (LPs)<n>We introduce the Linguistically Distinct Sentence Pairs dataset, which isolates ten key linguistic features such as synonymy, negation, tense, and quantity.<n>Using this dataset, we analyze BERT embeddings with various methods to identify the most influential dimensions for each LP.<n>Our findings show that certain properties, such as negation and polarity, are robustly encoded in specific dimensions, while others, like synonymy, exhibit more complex patterns.
  arXiv  Detail & Related papers  (2025-04-20T23:38:16Z)
• Structure-Aware Correspondence Learning for Relative Pose Estimation [65.44234975976451]
  Relative pose estimation provides a promising way for achieving object-agnostic pose estimation.<n>Existing 3D correspondence-based methods suffer from small overlaps in visible regions and unreliable feature estimation for invisible regions.<n>We propose a novel Structure-Aware Correspondence Learning method for Relative Pose Estimation, which consists of two key modules.
  arXiv  Detail & Related papers  (2025-03-24T13:43:44Z)
• Supervised Pattern Recognition Involving Skewed Feature Densities [49.48516314472825]
  The classification potential of the Euclidean distance and a dissimilarity index based on the coincidence similarity index are compared. The accuracy of classifying the intersection point between the densities of two adjacent groups is taken into account.
  arXiv  Detail & Related papers  (2024-09-02T12:45:18Z)
• Quantifying Behavioural Distance Between Mathematical Expressions [0.0]
  This paper proposes and implements a measure of a behavioral distance, BED, that clusters together expressions with similar errors. Our findings also reveal that BED significantly improves the smoothness of the error landscape in the search space for symbolic regression.
  arXiv  Detail & Related papers  (2024-08-21T10:48:04Z)
• The Shape of Word Embeddings: Quantifying Non-Isometry With Topological Data Analysis [10.242373477945376]
  We use persistent homology from topological data analysis to measure the distances between language pairs from the shape of their unlabeled embeddings. To distinguish whether these differences are random training errors or capture real information about the languages, we use the computed distance matrices to construct language phylogenetic trees over 81 Indo-European languages.
  arXiv  Detail & Related papers  (2024-03-30T23:51:25Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• Syntactic Substitutability as Unsupervised Dependency Syntax [31.488677474152794]
  We model a more general property implicit in the definition of dependency relations, syntactic substitutability. This property captures the fact that words at either end of a dependency can be substituted with words from the same category. We show that increasing the number of substitutions used improves parsing accuracy on natural data.
  arXiv  Detail & Related papers  (2022-11-29T09:01:37Z)
• Image Synthesis via Semantic Composition [74.68191130898805]
  We present a novel approach to synthesize realistic images based on their semantic layouts. It hypothesizes that for objects with similar appearance, they share similar representation. Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
  arXiv  Detail & Related papers  (2021-09-15T02:26:07Z)
• Discrete representations in neural models of spoken language [56.29049879393466]
  We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We find that the different evaluation metrics can give inconsistent results.
  arXiv  Detail & Related papers  (2021-05-12T11:02:02Z)
• Prototypical Representation Learning for Relation Extraction [56.501332067073065]
  This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data. We learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art relational models.
  arXiv  Detail & Related papers  (2021-03-22T08:11:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.