Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

• URL: http://arxiv.org/abs/2607.17822v1
• Date: Mon, 20 Jul 2026 11:10:51 GMT
• Title: Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
• Abstract summary: We introduce Mean Root Square Normalization (MRSNorm)<n>MRSNorm mathematically inverts the traditional scaling paradigm.<n>We analytically demonstrate that this constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity.
• Score: 3.0459410574367314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.

Related papers

• Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization [7.792020418343022]
  Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning.<n>We show that the two methods are locally consistent under parameter-space perturbations.<n>We also show that divergence-based regularization is defined through transformations of the input.
  arXiv  Detail & Related papers  (2026-09-08T19:01:05Z)
• Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI [0.7766379063545538]
  We introduce the Statistically Meaningful Geometry (SMG) framework, an information-geometric paradigm lifting parametric models into infinite-dimensional minimization non-asymptotic statistics.<n>We prove that under connection-filtered pre-training, out-of-distribution predictive variance is strictly upper-bounded by the finite diameter of the identifiable base manifold $mathcalB$, establishing a hard geometric containment of generative hallucinations.
  arXiv  Detail & Related papers  (2026-07-03T13:46:31Z)
• Variance Reduction for Stochastic Gradient Generalized Non-reversible Langevin Monte Carlo Algorithms [7.336637417508747]
  We study the leading-order fluctuation of gradient-Maruyama estimators for non-reversible Langevin dynamics.<n>We prove that the empirical average over a horizon of order the inverse squared stepsize satisfies a central limit theorem in the vanishing-stepsize regime.
  arXiv  Detail & Related papers  (2026-06-27T08:25:48Z)
• On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry [0.0]
  In curved models, including mixtures, curved exponential families, latent-variable models, and manifold-root parameter spaces, finite-sample behavior can deviate systematically from predictions.<n>We develop a coordinate-invariant, curvature-aware refinement by viewing a regular parametric family as a ((,g)) with Fisher-constrained-Rao metric.<n>We describe the resolved metric, the role of the real log canonical threshold in learning rates and posterior mean-squared error, and a curvature-based covariance expansion on the resolved space that recovers the regular theory as a special case.
  arXiv  Detail & Related papers  (2026-04-14T13:40:13Z)
• Muon Dynamics as a Spectral Wasserstein Flow [21.689846521201588]
  We introduce a family of Wasserstein norms on positive semidefinite matrices.<n>We prove the equivalence between the static and dynamic Benamou-Brenier formulations.<n>We identify exact gradientity associated with flow models.
  arXiv  Detail & Related papers  (2026-04-06T17:41:12Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds [0.5729426778193398]
  We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
  arXiv  Detail & Related papers  (2026-01-16T23:11:02Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Graph-based Clustering Revisited: A Relaxation of Kernel $k$-Means Perspective [73.18641268511318]
  We propose a graph-based clustering algorithm that only relaxes the orthonormal constraint to derive clustering results.<n>To ensure a doubly constraint into a gradient, we transform the non-negative constraint into a class probability parameter.
  arXiv  Detail & Related papers  (2025-09-23T09:14:39Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Square Root Marginalization for Sliding-Window Bundle Adjustment [53.34552451834707]
  We propose a novel square root sliding-window bundle adjustment suitable for real-time odometry applications. We show that the proposed square root marginalization is algebraically equivalent to the conventional use of Schur complement (SC) on the Hessian. Our evaluation of visual and visual-inertial odometry on real-world datasets demonstrates that the proposed estimator is 36% faster than the baseline.
  arXiv  Detail & Related papers  (2021-09-05T23:22:38Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.