Geometry-Guided Layerwise FFN Width Allocation in Transformers

• URL: http://arxiv.org/abs/2608.02064v1
• Date: Mon, 03 Aug 2026 11:08:58 GMT
• Title: Geometry-Guided Layerwise FFN Width Allocation in Transformers
• Abstract summary: Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth.<n>We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior.<n>We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics.
• Score: 4.109803241412197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior. We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics. A layerwise approximation surrogate yields an exact fixed-budget optimizer. Across seven pretrained language models, raw Euclidean work largely tracks residual-norm growth, whereas normalized work is predominantly front-loaded. Gromov-Wasserstein work is more consistently associated with perturbation-based layer sensitivity than the finite-sample topological estimate. In paired 128M and 256M training runs, several normalized-work schedules reduce mean validation loss relative to both uniform width and a hand-designed cosine taper. With the amplified paired differences at 440M, the best geometry-based allocations improve over uniform substantially larger than the cosine taper, while the anti-topological raw control is worse than uniform.

Related papers

• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Hierarchical Transformer Preconditioning for Interactive Physics Simulation [28.137076331332413]
  Hierarchical Transformer Preconditioner, a neural preconditioner anchored to a weak-admissibility H-matrix partition.<n>Network models the inverse through low-rank far-field factors.<n>Uses highway connections to propagate context across depth sparse.
  arXiv  Detail & Related papers  (2026-05-13T11:02:27Z)
• QuadNorm: Resolution-Robust Normalization for Neural Operators [41.429437803093485]
  We introduce a quadrature normalization family that replaces existing uniform averaging in normalization layers with numerical quadrature: QuadNorm and BlendQuadNorm.<n>On endpoint-inclusive uniform grids, the proposed quadrature moments are $O(h2)$-consistent across discretizations.<n>A transfer-error bound then predicts how normalization-induced mismatch scales with both the resolution gap and network depth.
  arXiv  Detail & Related papers  (2026-05-08T07:30:43Z)
• Stream separation improves Bregman conditioning in transformers [4.7718339202518685]
  Linear methods for steering transformer representations implicitly assume the geometry of representation space is Euclidean.<n>Park et al. showed that softmax induces a curved Bregman geometry whose metric tensor is the Hessian of the log-normalizer, $H() = Cov[ | ]$.<n>We measure this Hessian at intermediate layers in a controlled 2x2 design crossing stream separation with per-layer supervision.
  arXiv  Detail & Related papers  (2026-03-22T16:55:57Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• SPINAL -- Scaling-law and Preference Integration in Neural Alignment Layers [16.976750197698063]
  We introduce SPINAL, a diagnostic that measures how alignment reshapes representations across depth.<n>Across model families, DPO produces a layerwise calibration effect concentrated in the final decoder blocks.<n>Aligned checkpoints show a late-layer ramp-up in contraction and a smooth reduction in transport, consistent with tightened and stabilized policy mass.
  arXiv  Detail & Related papers  (2026-01-08T17:47:12Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• Precise Bayesian Neural Networks [0.0]
  We develop a lightweight, implementation-ready variational unit that fits modern normalized architectures and improves calibration without sacrificing accuracy.<n>In short, by aligning the variational posterior with the network's intrinsic geometry, BNNs can be simultaneously principled, practical, and precise.
  arXiv  Detail & Related papers  (2025-06-24T15:42:00Z)
• Orthogonalizing Convolutional Layers with the Cayley Transform [83.73855414030646]
  We propose and evaluate an alternative approach to parameterize convolutional layers that are constrained to be orthogonal. We show that our method indeed preserves orthogonality to a high degree even for large convolutions.
  arXiv  Detail & Related papers  (2021-04-14T23:54:55Z)
• TFill: Image Completion via a Transformer-Based Architecture [69.62228639870114]
  We propose treating image completion as a directionless sequence-to-sequence prediction task. We employ a restrictive CNN with small and non-overlapping RF for token representation. In a second phase, to improve appearance consistency between visible and generated regions, a novel attention-aware layer (AAL) is introduced.
  arXiv  Detail & Related papers  (2021-04-02T01:42:01Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.