Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

• URL: http://arxiv.org/abs/2606.02765v1
• Date: Mon, 01 Jun 2026 18:28:56 GMT
• Title: Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
• Abstract summary: We develop a framework for estimating how many near-orthogonal directions a model can support.<n>Applying this metric across dozens of open-source models reveals two classes: models with high $varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $varepsilon$ that maintain it.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored. Grounded in the Linear Representation and Superposition Hypotheses - which propose that models encode features as near-orthogonal directions in latent space - we develop a framework for estimating how many such directions a model can support. We first establish the embedding matrix as a measurable proxy for near-orthogonality constraints across the latent space: the boundary between meaningful token relationships and incidental similarity in the pairwise cosine similarity distribution gives a concrete estimate of the model's accepted deviation $\varepsilon$ from perfect orthogonality. Applying this metric across dozens of open-source models reveals two classes: models with high $\varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $\varepsilon$ that maintain it. We then show that the standard Johnson-Lindenstrauss lemma greatly underestimates the packing efficiency of trained representations, and derive an adjusted capacity formula in which the number of near-orthogonal directions depends on the ratio of vectors to dimensions ($k/d$) rather than the raw count - a single modification that cuts prediction error by two orders of magnitude with no extra parameters. Combining these results, we define representational capacity as an upper bound on the number of distinguishable directions available for features and embeddings in a model's latent space. Capacity is exponentially sensitive to $\varepsilon$, and larger models favor tighter orthogonality constraints over maximizing raw capacity - a pattern compatible with several explanations (a stability-capacity trade-off, a ceiling on usable concepts, or confounds with model scale) that we leave to future work.

Related papers

• A Censored Transformed Model for Proportional Outcomes with Boundary Mass and an Application to Loss Given Default Modeling [6.789370732159177]
  We introduce the zero-one censored transformed normal (ZOCTN) model for proportional responses with potential probability mass at the boundaries 0 and 1.<n>We characterize the transformation parameters establish large-sample properties, and relate the affine-logit specification to broader classes of interior distributions.
  arXiv  Detail & Related papers  (2026-06-19T15:10:41Z)
• A Spectral Framework for Closed-Form Relative Density Estimation [0.554780083433538]
  We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models.<n>We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features.
  arXiv  Detail & Related papers  (2026-05-11T14:51:17Z)
• Factor Graph-Based Shape Estimation for Continuum Robots via Magnus Expansion [19.359841144204115]
  This letter combines the strength of both paradigms by estimating the coefficients of a low-dimensional Geometric Variable Strain (GVS) parameterization within a factor graph framework.<n>A novel kinematic factor, derived from the Magnus expansion of the strain field, encodes the closed-form rod geometry as a prior constraint linking the GVS strain coefficients to the backbone pose variables.<n>The resulting formulation yields a compact state vector amenable directly to model-based control, while retaining the continuum, probabilistic treatment and computational efficiency of factor graph inference.
  arXiv  Detail & Related papers  (2026-04-17T01:52:07Z)
• Latent Semantic Manifolds in Large Language Models [0.0]
  We develop a mathematical framework that interprets Large Language Models hidden states as points on a latent semantic manifold.<n>We prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap.<n>We discuss implications for architecture design, model compression, decoding strategies, and scaling laws.
  arXiv  Detail & Related papers  (2026-03-17T13:05:56Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  We present a unifying perspective on recent results on ridge regression.<n>We use the basic tools of random matrix theory and free probability, aimed at readers with backgrounds in physics and deep learning.<n>Our results extend and provide a unifying perspective on earlier models of scaling laws.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• High-dimensional limit theorems for SGD: Effective dynamics and critical scaling [6.950316788263433]
  We prove limit theorems for the trajectories of summary statistics of gradient descent (SGD) We show a critical scaling regime for the step-size, below which the effective ballistic dynamics matches gradient flow for the population loss. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate.
  arXiv  Detail & Related papers  (2022-06-08T17:42:18Z)
• Low-Rank Constraints for Fast Inference in Structured Models [110.38427965904266]
  This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models. Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces.
  arXiv  Detail & Related papers  (2022-01-08T00:47:50Z)
• A Picture's Worth a Thousand Words: Visualizing n-dimensional Overlap in Logistic Regression Models with Empirical Likelihood [0.0]
  We introduce a sensitivity testing point of view for the maximum likelihood estimate for multidimensional predictor. The well known condition of Silvapulle is translated to be an empirical likelihood which, with existing R code, mechanizes the process of assessing overlap status. The code is applied to reveal the character of overlap by examining minimal overlapping structures and cataloging them in dimensions fewer than four.
  arXiv  Detail & Related papers  (2020-11-15T19:39:56Z)
• Exponentially Weighted l_2 Regularization Strategy in Constructing Reinforced Second-order Fuzzy Rule-based Model [72.57056258027336]
  In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules. We introduce an exponential weight approach inspired by the weight function theory encountered in harmonic analysis.
  arXiv  Detail & Related papers  (2020-07-02T15:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.