NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

• URL: http://arxiv.org/abs/2603.06922v1
• Date: Fri, 06 Mar 2026 22:50:43 GMT
• Title: NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks
• Abstract summary: We introduce NerVE, a unified espectral framework for understanding how feedforward networks (FFNs) regulate information flow in high-dimensional latent space.<n>NerVE addresses this gap through lightweight, memory-efficient tracking of eigen dynamics via four complementary geometries.
• Score: 2.8232103900765693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-error.

Related papers

• Fourier fractal dimension to predict the generalization of deep neural networks [0.0]
  We propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations.<n>By analyzing the characteristic function of the Lévy-driven differential equations in the frequency domain, we extract a metric that robustly captures the geometric complexity of the learning process.<n>Our method achieves state-of-the-art Kendall rank correlation, outperforming a wide array of existing norm-based, margin-based, and PAC-Bayesian measures.
  arXiv  Detail & Related papers  (2026-06-06T19:40:54Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning [10.647088281181222]
  SpecMuon is a spectral-aware, multi-mode gradient flow for physics-informed learning.<n>It regulates step sizes according to the global loss energy while preserving Muon's scale-balancing properties.<n>It achieves faster convergence and improved stability compared with Adam AdamW.
  arXiv  Detail & Related papers  (2026-02-18T03:56:20Z)
• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• Online Bayesian Experimental Design for Partially Observed Dynamical Systems [10.774974720491565]
  We develop a principled framework for optimizing data collection in dynamical systems with partial observability.<n>Our framework successfully handles both partial observability and online inference.
  arXiv  Detail & Related papers  (2025-11-06T14:29:05Z)
• Balancing Sparse RNNs with Hyperparameterization Benefiting Meta-Learning [0.0]
  This paper develops alternative hyper parameters for specifying sparse Recurrent Neural Networks (RNNs)<n>These hyper parameters allow for varying sparsity within the trainable weight matrices of the model while improving overall performance.<n>This architecture enables the definition of a novel metric, hidden proportion, which seeks to balance the distribution of unknowns within the model and provides significant explanatory power of model performance.
  arXiv  Detail & Related papers  (2025-09-18T15:20:13Z)
• Weight Spectra Induced Efficient Model Adaptation [54.8615621415845]
  Fine-tuning large-scale foundation models incurs prohibitive computational costs.<n>We show that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact.<n>We propose a novel method that leverages learnable rescaling of top singular directions.
  arXiv  Detail & Related papers  (2025-05-29T05:03:29Z)
• Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems [49.819436680336786]
  We propose an efficient transformed Gaussian process state-space model (ETGPSSM) for scalable and flexible modeling of high-dimensional, non-stationary dynamical systems.<n>Specifically, our ETGPSSM integrates a single shared GP with input-dependent normalizing flows, yielding an expressive implicit process prior that captures complex, non-stationary transition dynamics.<n>Our ETGPSSM outperforms existing GPSSMs and neural network-based SSMs in terms of computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-03-24T03:19:45Z)
• EqNIO: Subequivariant Neural Inertial Odometry [33.96552018734359]
  We show that IMU data transforms equivariantly, when rotated around the gravity vector and reflected with respect to arbitrary planes parallel to gravity. We then map the IMU data into this frame, thereby achieving an invariant canonicalization that can be directly used with off-the-shelf inertial odometry networks.
  arXiv  Detail & Related papers  (2024-08-12T17:42:46Z)
• Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference [47.460898983429374]
  We introduce an ensemble Kalman filter (EnKF) into the non-mean-field (NMF) variational inference framework to approximate the posterior distribution of the latent states. This novel marriage between EnKF and GPSSM not only eliminates the need for extensive parameterization in learning variational distributions, but also enables an interpretable, closed-form approximation of the evidence lower bound (ELBO) We demonstrate that the resulting EnKF-aided online algorithm embodies a principled objective function by ensuring data-fitting accuracy while incorporating model regularizations to mitigate overfitting.
  arXiv  Detail & Related papers  (2023-12-10T15:22:30Z)
• FedNAR: Federated Optimization with Normalized Annealing Regularization [54.42032094044368]
  We explore the choices of weight decay and identify that weight decay value appreciably influences the convergence of existing FL algorithms. We develop Federated optimization with Normalized Annealing Regularization (FedNAR), a plug-in that can be seamlessly integrated into any existing FL algorithms.
  arXiv  Detail & Related papers  (2023-10-04T21:11:40Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks [0.0]
  We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that the network learns eigenfunctions of an integral operator $T_Kinfty$ determined by the Neural Tangent Kernel (NTK) We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.
  arXiv  Detail & Related papers  (2022-01-12T23:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.