Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models

• URL: http://arxiv.org/abs/2603.20642v1
• Date: Sat, 21 Mar 2026 04:38:30 GMT
• Title: Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models
• Abstract summary: We show that representational geometry is consistently log-compressive across three magnitude domains.<n>One model produces a human-range Weber fraction (WF = 0.20) while the other does not, and both models perform at chance on temporal and spatial discrimination.<n> causal intervention reveals a layer dissociation: early layers are functionally implicated in magnitude processing (4.1x specificity) while later layers where geometry is strongest are not causally engaged.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We apply the formal tools of psychophysics to resolve this. Using four converging paradigms (representational similarity analysis, behavioural discrimination, precision gradients, causal intervention) across three magnitude domains in three 7-9B instruction-tuned models spanning three architecture families (Llama, Mistral, Qwen), we report three findings. First, representational geometry is consistently log-compressive: RSA correlations with a Weber-law dissimilarity matrix ranged from .68 to .96 across all 96 model-domain-layer cells, with linear geometry never preferred. Second, this geometry is dissociated from behaviour: one model produces a human-range Weber fraction (WF = 0.20) while the other does not, and both models perform at chance on temporal and spatial discrimination despite possessing logarithmic geometry. Third, causal intervention reveals a layer dissociation: early layers are functionally implicated in magnitude processing (4.1x specificity) while later layers where geometry is strongest are not causally engaged (1.2x). Corpus analysis confirms the efficient coding precondition (alpha = 0.77). These results suggest that training data statistics alone are sufficient to produce log-compressive magnitude geometry, but geometry alone does not guarantee behavioural competence.

Related papers

• The information geometry of large language models is shared, learned, and controllable [0.0]
  We show that behaviour determines geometry up to output-preserving symmetries.<n>Across transformer, state-space and recurrent models, output geometries agree more strongly than activation geometries.<n> Controlled language assignments show that geometry follows the language law across architectures.
  arXiv  Detail & Related papers  (2026-09-10T04:09:49Z)
• Hierarchical Grading in Large Language Models [0.0]
  We introduce Graded Large Language Models (GLLMs)<n>GLLMs are a framework that equips the representation space of a transformer with a grading.<n>The construction extends the theory of graded neural networks and graded transformers to autoregressive language models.
  arXiv  Detail & Related papers  (2026-07-23T20:14:48Z)
• Riemannian geometry meets fMRI: the advantages of modeling correlation manifolds and eigenvector subspaces [0.0]
  Correlation matrices are fundamental summaries of functional brain networks.<n>Existing geometric methods frequently lack closed-form operations or depend on arbitrary region ordering.<n>We introduce a scalable geometric framework with two components: the Off-log metric and Grassmannian subspace discrimination.
  arXiv  Detail & Related papers  (2026-05-21T11:22:31Z)
• Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation [0.7162422068114824]
  We investigate whether independently trained language models converge to geometrically compatible latent representations.<n>We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model.<n>We report a near-zero correlation between geometric alignment quality and behavioral correction rate.
  arXiv  Detail & Related papers  (2026-03-20T18:26:23Z)
• Learning and extrapolating scale-invariant processes [3.331543293568139]
  We tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches?<n>We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model.
  arXiv  Detail & Related papers  (2026-01-21T09:35:44Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Unveiling the Sampling Density in Non-Uniform Geometric Graphs [69.93864101024639]
  We consider graphs as geometric graphs: nodes are randomly sampled from an underlying metric space, and any pair of nodes is connected if their distance is less than a specified neighborhood radius. In a social network communities can be modeled as densely sampled areas, and hubs as nodes with larger neighborhood radius. We develop methods to estimate the unknown sampling density in a self-supervised fashion.
  arXiv  Detail & Related papers  (2022-10-15T08:01:08Z)
• Learning Physical Dynamics with Subequivariant Graph Neural Networks [99.41677381754678]
  Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. Physical laws abide by symmetry, which is a vital inductive bias accounting for model generalization. Our model achieves on average over 3% enhancement in contact prediction accuracy across 8 scenarios on Physion and 2X lower rollout MSE on RigidFall.
  arXiv  Detail & Related papers  (2022-10-13T10:00:30Z)
• Curved Geometric Networks for Visual Anomaly Recognition [39.91252195360767]
  Learning a latent embedding to understand the underlying nature of data distribution is often formulated in Euclidean spaces with zero curvature. In this work, we investigate benefits of the curved space for analyzing anomalies or out-of-distribution objects in data.
  arXiv  Detail & Related papers  (2022-08-02T01:15:39Z)
• NeuroMorph: Unsupervised Shape Interpolation and Correspondence in One Go [109.88509362837475]
  We present NeuroMorph, a new neural network architecture that takes as input two 3D shapes. NeuroMorph produces smooth and point-to-point correspondences between them. It works well for a large variety of input shapes, including non-isometric pairs from different object categories.
  arXiv  Detail & Related papers  (2021-06-17T12:25:44Z)
• GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles [60.12186997181117]
  Prediction of a molecule's 3D conformer ensemble from the molecular graph holds a key role in areas of cheminformatics and drug discovery. Existing generative models have several drawbacks including lack of modeling important molecular geometry elements. We propose GeoMol, an end-to-end, non-autoregressive and SE(3)-invariant machine learning approach to generate 3D conformers.
  arXiv  Detail & Related papers  (2021-06-08T14:17:59Z)
• Self-supervised Geometric Perception [96.89966337518854]
  Self-supervised geometric perception is a framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels. We show that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.
  arXiv  Detail & Related papers  (2021-03-04T15:34:43Z)
• Identifying the latent space geometry of network models through analysis of curvature [7.644165047073435]
  We present a method to consistently estimate the manifold type, dimension, and curvature from an empirically relevant class of latent spaces. Our core insight comes by representing the graph as a noisy distance matrix based on the ties between cliques.
  arXiv  Detail & Related papers  (2020-12-19T00:35:29Z)
• Computational Analysis of Deformable Manifolds: from Geometric Modelling to Deep Learning [0.0]
  We will show that the diversity of non-flat spaces provides a rich area of study. We will explore geometric methods for shape processing and data analysis.
  arXiv  Detail & Related papers  (2020-09-03T16:50:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.