StructLens: A Structural Lens for Language Models via Maximum Spanning Trees

• URL: http://arxiv.org/abs/2603.03328v1
• Date: Tue, 10 Feb 2026 11:30:32 GMT
• Title: StructLens: A Structural Lens for Language Models via Maximum Spanning Trees
• Abstract summary: StructLens is an analytical framework designed to reveal how internal structures relate holistically.<n>Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity.
• Score: 52.040177523973334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest internal structures as well. While interpretability research has investigated the components of language models, existing approaches focus on local inter-token relationships within layers or modules (e.g., Multi-Head Attention), leaving global inter-layer relationships largely overlooked. To address this gap, we introduce StructLens, an analytical framework designed to reveal how internal structures relate holistically through their inter-token connection within a layer. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, analogous to dependency parsing, and leverages the tree properties to quantify inter-layer distance (or similarity) from a structural perspective. Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity. Moreover, this structure-aware similarity proves to be beneficial for practical tasks, such as layer pruning, highlighting the effectiveness of structural analysis for understanding and optimizing language models. Our code is available at https://github.com/naist-nlp/structlens.

Related papers

• When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs [20.39151103511549]
  Large language models exhibit strong multilingual capabilities, but their internal representations are difficult to interpret.<n>Recent work has shown that causal-geometric structure can explain how certain concepts are encoded as approximately linear and separable directions.<n>We apply causal-geometric analysis to multilingual LLMs, studying 28 bilingual contrasts across three models.
  arXiv  Detail & Related papers  (2026-06-12T11:00:59Z)
• Tracking the emergence of linguistic structure in self-supervised models learning from speech [5.443419622061083]
  We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch.<n>We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories.
  arXiv  Detail & Related papers  (2026-04-02T13:48:22Z)
• From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders [12.294297654425334]
  Hierarchical Sparse Autoencoder (HSAE) learns a series of SAEs and the parent-child relationships between their features.<n>Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in language representations.
  arXiv  Detail & Related papers  (2026-02-12T12:30:23Z)
• From Static Structures to Ensembles: Studying and Harnessing Protein Structure Tokenization [15.864659611818661]
  Protein structure tokenization converts 3D structures into discrete or vectorized representations.<n>Despite many recent works on structure tokenization, the properties of the underlying discrete representations are not well understood.<n>We show that the successful utilization of structural tokens in a language model for structure prediction depends on using rich, pre-trained sequence embeddings.
  arXiv  Detail & Related papers  (2025-11-13T07:58:24Z)
• Counting trees: A treebank-driven exploration of syntactic variation in speech and writing across languages [0.0]
  We define syntactic structures as delexicalized dependency (sub)trees and extract them from spoken and written Universal Dependencies treebanks.<n>For each corpus, we analyze the size, diversity, and distribution of syntactic inventories, their overlap across modalities, and the structures most characteristic of speech.<n>Results show that, across both languages, spoken corpora contain fewer and less diverse syntactic structures than their written counterparts.
  arXiv  Detail & Related papers  (2025-05-28T18:43:26Z)
• Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures [49.19753720526998]
  We derive theoretical scaling laws for neural network performance on synthetic datasets.<n>We validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance.<n>This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
  arXiv  Detail & Related papers  (2025-05-11T17:44:14Z)
• Character-Level Chinese Dependency Parsing via Modeling Latent Intra-Word Structure [11.184330703168893]
  This paper proposes modeling latent internal structures within words in Chinese. A constrained Eisner algorithm is implemented to ensure the compatibility of character-level trees. A detailed analysis reveals that a coarse-to-fine parsing strategy empowers the model to predict more linguistically plausible intra-word structures.
  arXiv  Detail & Related papers  (2024-06-06T06:23:02Z)
• Grokking of Hierarchical Structure in Vanilla Transformers [72.45375959893218]
  We show that transformer language models can learn to generalize hierarchically after training for extremely long periods. intermediate-depth models generalize better than both very deep and very shallow transformers.
  arXiv  Detail & Related papers  (2023-05-30T04:34:13Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.<n>We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.<n>We demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysis [31.05169054736711]
  Cross-lingual structured sentiment analysis task aims to transfer the knowledge from source language to target one. We propose a Knowledge-Enhanced Adversarial Model (textttKEAM) with both implicit distributed and explicit structural knowledge. We conduct experiments on five datasets and compare textttKEAM with both the supervised and unsupervised methods.
  arXiv  Detail & Related papers  (2022-05-31T03:07:51Z)
• Capturing Structural Locality in Non-parametric Language Models [85.94669097485992]
  We propose a simple yet effective approach for adding locality information into non-parametric language models. Experiments on two different domains, Java source code and Wikipedia text, demonstrate that locality features improve model efficacy.
  arXiv  Detail & Related papers  (2021-10-06T15:53:38Z)
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach [78.77265671634454]
  We make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called "syntactic distances" Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.
  arXiv  Detail & Related papers  (2020-05-12T15:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.