On the Geometric Structure of Layer Updates in Deep Language Models

• URL: http://arxiv.org/abs/2604.02459v1
• Date: Thu, 02 Apr 2026 18:44:34 GMT
• Title: On the Geometric Structure of Layer Updates in Deep Language Models
• Abstract summary: We study the geometric structure of layer updates in deep language models.<n>We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the geometric structure of layer updates in deep language models. Rather than analyzing what information is encoded in intermediate representations, we ask how representations change from one layer to the next. We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes. Across multiple architectures, including Transformers and state-space models, we find that the full layer update is almost perfectly aligned with the tokenwise component, while the residual exhibits substantially weaker alignment, larger angular deviation, and significantly lower projection onto the dominant tokenwise subspace. This indicates that the residual is not merely a small correction, but a geometrically distinct component of the transformation. This geometric separation has functional consequences: approximation error under the restricted tokenwise model is strongly associated with output perturbation, with Spearman correlations often exceeding 0.7 and reaching up to 0.95 in larger models. Together, these results suggest that most layerwise updates behave like structured reparameterizations along a dominant direction, while functionally significant computation is concentrated in a geometrically distinct residual component. Our framework provides a simple, architecture-agnostic method for probing the geometric and functional structure of layer updates in modern language models.

Related papers

• Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling [55.36837305276515]
  We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling.<n>We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost.
  arXiv  Detail & Related papers  (2026-08-29T11:43:25Z)
• Topological Simplification in Predictive Coding Networks [0.8399542059465145]
  We study the topology of learned in predictive coding networks (PCNs) using a quantitative-wise persistent homology analysis.<n>Results suggest that persistent homology offers a useful lens on the compression--reconstruction and layers in PCNs.
  arXiv  Detail & Related papers  (2026-08-03T19:12:11Z)
• Variable-Width Transformers [57.07167443557886]
  We investigate nonuniform capacity allocation across network depth by proposing a $times$-shaped > former architecture.<n>Our design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism.<n>By reducing the average layer width, this architecture also requires fewer overall FLOPs and smaller KV cache memory and I/O cost.
  arXiv  Detail & Related papers  (2026-06-16T17:59:03Z)
• BA-T: An Iterative Transformer for Two-View Bundle Adjustment [53.16990349592508]
  We propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space.<n> experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations.<n>BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture.
  arXiv  Detail & Related papers  (2026-06-02T07:51:14Z)
• How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks [3.449933410012168]
  Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures.<n>We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings.<n>We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations.
  arXiv  Detail & Related papers  (2026-05-26T20:25:44Z)
• Geometry-Preserving Neural Architectures on Manifolds with Boundary [3.6352834408416412]
  We propose a class of geometry-aware architectures that interleave geometric updates between layers.<n>We establish universal approximation results for constrained neural ODEs.<n>Experiments on dynamics over S2 and SO(3), and diffusion on Sd-1-valued features demonstrate exact feasibility for analytic updates.
  arXiv  Detail & Related papers  (2026-02-03T04:09:39Z)
• Deep Delta Learning [91.75868893250662]
  We introduce Deep Delta Learning (DDL), a novel architecture that generalizes the standard residual connection.<n>We provide a spectral analysis of this operator, demonstrating that the gate $(mathbfX)$ enables dynamic between identity mapping, projection, and geometric reflection.<n>This unification empowers the network to explicitly control the spectrum of its layer-wise transition operator, enabling the modeling of complex, non-monotonic dynamics.
  arXiv  Detail & Related papers  (2026-01-01T18:11:38Z)
• Reconstructing Multi-Scale Physical Fields from Extremely Sparse Measurements with an Autoencoder-Diffusion Cascade [38.28865883904372]
  Cascaded Sensing (Cas-Sensing) is a hierarchical reconstruction framework that integrates an autoencoder-diffusion cascade.<n>A conditional diffusion model, trained with a mask-cascade strategy, generates fine-scale details conditioned on large-scale structures.<n>Experiments on both simulation and real-world datasets demonstrate that Cas-Sensing generalizes well across varying sensor configurations and geometric boundaries.
  arXiv  Detail & Related papers  (2025-12-01T11:46:14Z)
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure [8.201374511929538]
  This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design.<n>We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling.
  arXiv  Detail & Related papers  (2025-10-29T02:24:27Z)
• Geometric Properties of Neural Multivariate Regression [3.259067345005505]
  Collapsed models exhibit ID_H ID_Y, leading to over-compression and poor generalization.<n>We identify two regimes that determine when expanding or reducing feature dimensionality improves performance.
  arXiv  Detail & Related papers  (2025-10-01T16:50:57Z)
• Hierarchical-embedding autoencoder with a predictor (HEAP) as efficient architecture for learning long-term evolution of complex multi-scale physical systems [41.94295877935867]
  Structures of various scales that dynamically emerge in the system interact with each other only locally.<n>The hierarchical fully-convolutional autoencoder transforms the state of a physical system into a series of embedding layers.<n> Interactions between features of various scales are modeled using a combination of convolutional operators.
  arXiv  Detail & Related papers  (2025-05-24T20:27:16Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• The geometry of hidden representations of large transformer models [43.16765170255552]
  Large transformers are powerful architectures used for self-supervised data analysis across various data types. We show that the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets.
  arXiv  Detail & Related papers  (2023-02-01T07:50:26Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.