Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs

• URL: http://arxiv.org/abs/2605.12714v1
• Date: Tue, 12 May 2026 20:22:45 GMT
• Title: Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs
• Abstract summary: We propose Layer-wise Representation Dynamics (LRD), a framework with three layer-wise measurement families.<n>Applying LRD to 31 models reveals architectural and task-level differences that are not apparent from final-layer representations alone.
• Score: 8.425980139753646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hidden states change substantially across the layers of modern language models, but most layer-wise analyses focus on one aspect of that change. We propose Layer-wise Representation Dynamics (LRD), a framework with three layer-wise measurement families: Frenet (Grassmann speed and curvature) for global subspace motion, Neighborhood Retention Score (NRS) for local nearest-neighbor retention, and Graph Filtration Mutual Information (GFMI) for alignment with the final layer. Applying LRD to 31 models (encoder-based and decoder-based embedders, plus base LLMs) on 30 MTEB tasks reveals architectural and task-level differences that are not apparent from final-layer representations alone. We then use LRD for two applications: label-free model selection and inference-time layer pruning. For selection, all three model-level scores correlate positively with downstream MTEB performance, with end-to-end subspace displacement (d_{0,L}) the strongest, and the same direction holds on a smaller base-LLM MMLU panel. For pruning, GFMI is the only measurement-guided rule that beats Random at the 15% and 20% budgets and has the best median change at every budget. Frenet is effective only at the lightest budget, while NRS does not transfer from model selection to pruning. These results show that layer-wise structure provides signal for both interpretation and deployment decisions.

Related papers

• Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning [30.408921338315093]
  We propose LA-ReduNet, a lightweight adaptive architecture that refines the layer-wise update rule.<n>We show that LA-ReduNet requires significantly fewer layers for the MCR$2$ objective to reach a stable value.<n>Remarkably, under the considered experimental settings, LA-ReduNet requires only approximately $1/29$ of the parameter storage of the unfolded ReduNet module.
  arXiv  Detail & Related papers  (2026-08-21T01:58:51Z)
• Variable-Width Transformers [57.07167443557886]
  We investigate nonuniform capacity allocation across network depth by proposing a $times$-shaped > former architecture.<n>Our design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism.<n>By reducing the average layer width, this architecture also requires fewer overall FLOPs and smaller KV cache memory and I/O cost.
  arXiv  Detail & Related papers  (2026-06-16T17:59:03Z)
• Last-Layer-Centric Feature Recombination: Unleashing 3D Geometric Knowledge in DINOv3 for Monocular Depth Estimation [4.654162664140336]
  Recent vision foundation models (VFMs) have significantly improved accuracy and generalization for dense prediction.<n>In this study, we present a systematic layer-wise analysis of DINOv3, revealing that 3D information is distributed non-uniformly.<n>Motivated by this, we introduce a Last-Layer-Centric Feature Recombination (LFR) module to enhance geometric expressiveness.
  arXiv  Detail & Related papers  (2026-04-29T09:11:36Z)
• What matters for Representation Alignment: Global Information or Spatial Structure? [64.67092609921816]
  Representation alignment (REPA) guides generative training by distilling representations from a strong, pretrained vision encoder to intermediate diffusion features.<n>We investigate a fundamental question: what aspect of the target representation matters for generation, its textitglobal revisionsemantic information.<n>We replace the standard projection layer in REPA with a simple convolution layer and introduce a spatial normalization layer for the external representation.
  arXiv  Detail & Related papers  (2025-12-11T16:39:53Z)
• Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models [4.935224714809964]
  We introduce Hierarchical Alignment, a novel method that applies targeted DPO to distinct functional blocks of a model's layers.<n>Specifically, aligning the local layers (Local-Align) enhances grammatical fluency.<n> aligning the global layers (Global-Align) improves factual consistency as hypothesized but also proves to be the most effective strategy for enhancing logical coherence.
  arXiv  Detail & Related papers  (2025-10-14T00:58:34Z)
• Hierarchical LoRA MoE for Efficient CTR Model Scaling [56.608809143548946]
  HiLoMoE is a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner.<n>Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel.
  arXiv  Detail & Related papers  (2025-10-12T03:54:11Z)
• Multimodal Language Models See Better When They Look Shallower [54.5303326937134]
  Multimodal large language models (MLLMs) typically extract visual features from the final layers of a pretrained Vision Transformer (ViT)<n>We present the first comprehensive study of visual layer selection for MLLMs, analyzing representation similarity across ViT layers.<n>We find that while deep layers excel in semantic-rich tasks like OCR, shallow and middle layers significantly outperform them on fine-grained visual tasks.
  arXiv  Detail & Related papers  (2025-04-30T09:07:10Z)
• Maximum Redundancy Pruning: A Principle-Driven Layerwise Sparsity Allocation for LLMs [25.66881879984798]
  Large language models (LLMs) have demonstrated impressive capabilities, but their enormous size poses significant challenges for deployment in real-world applications.<n>Recent sparsity allocation methods is often based on iteration or search that can easily lead to suboptimal performance.<n>We propose Redundancy Pruning (MRP), an iterative pruning algorithm that prunes in the most redundant layers.
  arXiv  Detail & Related papers  (2025-03-24T06:17:30Z)
• How to Make LLMs Strong Node Classifiers? [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect [38.148626520751385]
  We show that many layers of Large Language Models (LLMs) exhibit high similarity, and some layers play a negligible role in network functionality. We propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning.
  arXiv  Detail & Related papers  (2024-03-06T17:04:18Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Layer-adaptive sparsity for the Magnitude-based Pruning [88.37510230946478]
  We propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score. LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
  arXiv  Detail & Related papers  (2020-10-15T09:14:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.