Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

• URL: http://arxiv.org/abs/2601.03779v1
• Date: Wed, 07 Jan 2026 10:16:59 GMT
• Title: Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations
• Abstract summary: We ask if different ID profiles across LLM layers differentially characterize formal and functional complexity.<n>We find the formal contrast between sentences with multiple coordinated or subordinated clauses to be reflected in ID differences.<n>The functional contrasts between sentences characterized by right branching vs. center embedding or unambiguous vs. ambiguous relative clause attachment are also picked up by ID.
• Score: 8.540350401981277
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore the intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity, asking if different ID profiles across LLM layers differentially characterize formal and functional complexity. We find the formal contrast between sentences with multiple coordinated or subordinated clauses to be reflected in ID differences whose onset aligns with a phase of more abstract linguistic processing independently identified in earlier work. The functional contrasts between sentences characterized by right branching vs. center embedding or unambiguous vs. ambiguous relative clause attachment are also picked up by ID, but in a less marked way, and they do not correlate with the same processing phase. Further experiments using representational similarity and layer ablation confirm the same trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it allows to differentiate between different types of complexity, and that it points to similar stages of linguistic processing across disparate LLMs.

Related papers

• Local and Global Regimes of Geometric Complexity in Language Model Representations [5.972083596296453]
  This paper focuses on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset.<n>We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses.
  arXiv  Detail & Related papers  (2026-08-14T15:01:24Z)
• Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring [39.19569497931068]
  We propose a scalable, multifactorial experimental framework that probes LLM sensitivity to subtle semantic changes in pairwise document comparison.<n>Our analysis reveals several striking findings, including that LLMs exhibit a within-document positional bias.<n>The proposed framework offers a practical, LLM-agnostic toolkit for auditing and comparing scoring behavior across current and future models.
  arXiv  Detail & Related papers  (2026-04-20T20:59:25Z)
• Explainable Semantic Textual Similarity via Dissimilar Span Detection [53.32175252285023]
  We introduce the task of Dissimilar Span Detection (DSD), which aims to identify semantically differing spans between pairs of texts.<n>This can help users understand which particular words or tokens negatively affect the similarity score, or be used to improve performance in STS-dependent downstream tasks.
  arXiv  Detail & Related papers  (2026-03-22T11:32:31Z)
• Differential syntactic and semantic encoding in LLMs [49.300174325011426]
  We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs)<n>We find that the cross-layer encoding profiles of syntax and semantics are different, and that the two signals can to some extent be decoupled.
  arXiv  Detail & Related papers  (2026-01-08T09:33:29Z)
• QUDsim: Quantifying Discourse Similarities in LLM-Generated Text [70.22275200293964]
  We introduce an abstraction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression.<n>We then use this framework to build $textbfQUDsim$, a similarity metric that can detect discursive parallels between documents.<n>Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs.
  arXiv  Detail & Related papers  (2025-04-12T23:46:09Z)
• Linguistic Blind Spots of Large Language Models [14.755831733659699]
  We study the performance of recent large language models (LLMs) on linguistic annotation tasks.<n>We find that recent LLMs show limited efficacy in addressing linguistic queries and often struggle with linguistically complex inputs.<n>Our results provide insights to inform future advancements in LLM design and development.
  arXiv  Detail & Related papers  (2025-03-25T01:47:13Z)
• Computation Mechanism Behind LLM Position Generalization [59.013857707250814]
  Large language models (LLMs) exhibit flexibility in handling textual positions.<n>They can understand texts with position perturbations and generalize to longer texts.<n>This work connects the linguistic phenomenon with LLMs' computational mechanisms.
  arXiv  Detail & Related papers  (2025-03-17T15:47:37Z)
• Complex Ontology Matching with Large Language Model Embeddings [0.21847754147782888]
  This paper proposes to integrate large language models (LLMs) into an approach for generating expressive correspondences.<n>The generation of correspondences is performed by matching similar surroundings of instance sub-graphs.
  arXiv  Detail & Related papers  (2025-02-19T10:56:27Z)
• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow [58.56731133392544]
  We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE)<n>LLMs-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine to generate feedback-akin to textual gradients.<n>It consistently outperforms existing textual gradient baselines in both accuracy and training cost.
  arXiv  Detail & Related papers  (2025-01-28T03:18:48Z)
• A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension [16.671316494925346]
  This study investigates the effects of supervised fine-tuning and in-context learning on the hidden representations of Large Language Models (LLMs)<n>We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL.<n>We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT.
  arXiv  Detail & Related papers  (2024-12-09T06:37:35Z)
• Geometric Signatures of Compositionality Across a Language Model's Lifetime [47.25475802128033]
  We study whether contemporary language models reflect intrinsic simplicity of language enabled by compositionality.<n>We find that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training.<n>Our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition.
  arXiv  Detail & Related papers  (2024-10-02T11:54:06Z)
• Probing LLMs for Joint Encoding of Linguistic Categories [10.988109020181563]
  We propose a framework for testing the joint encoding of linguistic categories in Large Language Models (LLMs) We find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy.
  arXiv  Detail & Related papers  (2023-10-28T12:46:40Z)
• mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning [54.523172171533645]
  Cross-lingual named entity recognition (CrossNER) faces challenges stemming from uneven performance due to the scarcity of multilingual corpora. We propose Multi-view Contrastive Learning for Cross-lingual Named Entity Recognition (mCL-NER) Our experiments on the XTREME benchmark, spanning 40 languages, demonstrate the superiority of mCL-NER over prior data-driven and model-based approaches.
  arXiv  Detail & Related papers  (2023-08-17T16:02:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.