Feed-Forward Steering in Transformer Residual Dynamics

• URL: http://arxiv.org/abs/2608.02071v1
• Date: Mon, 03 Aug 2026 11:15:31 GMT
• Title: Feed-Forward Steering in Transformer Residual Dynamics
• Abstract summary: We extend attention-only Transformer residual directions by incorporating the feed-forward network (FFN) term as a local steering field on each token state.<n>The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space.<n>As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss.
• Score: 4.109803241412197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FFN) term as a local steering field acting on each token state. The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space, that critical residual directions correspond to nonlinear projective equilibria, and that a commutator defect determines when a finite attention--FFN block can be accurately approximated by a parallel, additive flow. Across GPT-2, Pythia, Mistral, and Llama models, the extended theory improves one-step angular prediction relative to an attention-only baseline, with the contribution of the FFN increasing from GPT-2 to Llama-3-8B. Intervention experiments show that retaining only the tangential FFN component preserves most model quality, whereas retaining only the radial component causes performance to collapse. The tangential component also preserves output diversity under aggregation pressure. As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss, whereas layers with large defects degrade rapidly. These findings support the interpretation of FFN layers as directional steering fields that shape Transformer residual geometry and govern the feasibility of block-level interventions.

Related papers

• FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation [11.04143971119602]
  We present FAN-LoRA, a novel frequency-decoupled fine-tuning architecture.<n>We show that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines.<n>Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors.
  arXiv  Detail & Related papers  (2026-08-27T02:08:57Z)
• Making Every Step Count: Spatio-Temporal Information Allocation for Imaging Inverse Problems [63.904793414062816]
  We introduce two complementary and training-free components, i.e., Spectrum-based Scheduling (SAS) and Measurement-Prioritized Attention (MPA)<n>SAS distributes the available NFEs over flow time according to the spectrum and logSNR geometry, thus better balancing semantic exploration and detail refinement.<n>MPA guides information toward weakly constrained regions, thereby enhancing semantic and structural fidelity.
  arXiv  Detail & Related papers  (2026-08-12T07:39:22Z)
• An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers [1.4180331276028662]
  We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers.<n>We state boundary advantages under independently checkable energy, correlation, and local-channel bounds.
  arXiv  Detail & Related papers  (2026-07-20T08:44:31Z)
• A lift for input-convex neural network training [3.142113135607563]
  In Inputdimensional neural networks (ICNNs) are used for logconcave density estimation, convexpotential normalizing flows, optimal transport, and transportmap inversion for posteriors.<n>The standard projected gradient descent (PGD) onto the non-negative cone applies a hard, nonsmooth projection.<n>The differentiable alternative, softplus reparametrization, attenuates the gradient exponentially in the weight magnitude, stalling training with dead interlayer weights loss.<n>We show that the lift reaches a lower test loss than both PGD and direct softplus, and turns a plateau-bounded training into a
  arXiv  Detail & Related papers  (2026-05-22T22:59:11Z)
• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization [57.37943479039033]
  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent.<n>We show that locality and weight sharing fundamentally change this picture.
  arXiv  Detail & Related papers  (2026-03-05T04:50:51Z)
• Data-Free PINNs for Compressible Flows: Mitigating Spectral Bias and Gradient Pathologies via Mach-Guided Scaling and Hybrid Convolutions [0.0]
  This paper presents a fully data-free Physics-Informed Neural Network (PINN) capable of solving compressible inviscid flows.<n>The proposed framework successfully captures the detached bow shock without referential data.
  arXiv  Detail & Related papers  (2026-03-01T09:04:18Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• A scalable flow-based approach to mitigate topological freezing [34.54607280864912]
  We present a flow-based strategy to remove topological artifacts from Markov Chain Monte Carlo simulations.<n>The strategy is based on a Normalizing Flow (SNF) that alternates non-equilibrium Monte Carlo updates with localized, stout-equivariant defect layers.<n>We show that defect SNFs achieve better performances than reproducing non-equilibrium methods at comparable cost.
  arXiv  Detail & Related papers  (2026-01-28T15:40:46Z)
• On Exact Editing of Flow-Based Diffusion Models [97.0633397035926]
  We propose Conditioned Velocity Correction (CVC) to reformulate flow-based editing as a distribution transformation problem driven by a known source prior.<n>CVC rethinks the role of velocity in inter-distribution transformation by introducing a dual-perspective velocity conversion mechanism.<n>We show that CVC consistently achieves superior fidelity, better semantic alignment, and more reliable editing behavior across diverse tasks.
  arXiv  Detail & Related papers  (2025-12-30T06:29:20Z)
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability [53.21677928601684]
  Layer-wise relevance propagation is one of the most promising approaches to explainability in deep learning.<n>We propose specialized theoretically-grounded LRP rules designed to propagate attributions across various positional encoding methods.<n>Our method significantly outperforms the state-of-the-art in both vision and NLP explainability tasks.
  arXiv  Detail & Related papers  (2025-06-02T18:07:55Z)
• Robust topological feature against non-Hermiticity in Jaynes-Cummings Model [0.0]
  We analytically analyze the topological feature manifested by the Jaynes-Cummings Model (JCM) The non-Hermiticity tilts the spin winding plane and induces out-of-plane component, while the topological feature is maintained.
  arXiv  Detail & Related papers  (2024-02-09T12:26:59Z)
• Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond [84.56978780892783]
  We propose CoupledTPS, which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. In light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. Experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art solutions for rotation correction.
  arXiv  Detail & Related papers  (2024-01-24T13:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.