Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections

• URL: http://arxiv.org/abs/2603.20896v1
• Date: Sat, 21 Mar 2026 18:03:03 GMT
• Title: Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections
• Abstract summary: Unconstrained mixing disrupts the identity mapping property to the residual connection, causing unstable training.<n>We propose Spectral-Sphere-trained Hyper-Connections (sHC)<n>sHC allows negative entries, unlocking subtractive interactions for selective feature diversification.
• Score: 10.82991070567946
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyper-Connections (HC) generalize residual connections into multiple streams, employing residual matrices for cross-stream feature mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variant restrict these matrices to the Birkhoff polytope (doubly stochastic matrices) via Sinkhorn iterations or permutation-based parameterizations. We reveal three limitations of this polytope constraint: (1) identity degeneration, where learned matrices collapse around the identity and diminish cross-stream interactions, (2) an expressivity bottleneck, as the non-negativity constraint prevents subtractive feature disentanglement, and (3) parameterization inefficiencies, manifesting as unstable Sinkhorn iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere, sHC allows negative entries, unlocking subtractive interactions for selective feature diversification. This shift eliminates unstable Sinkhorn projections and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Related papers

• On the global well-posedness for the nonlocal Fokas-Lenells equation with the weighted Sobolev initial data on the line [2.2877784723541907]
  We establish the global well-posedness of the Cauchy problem for the reverse space-time nonlocal Fokas-Lenells equation.<n>A spectral uniformization transform is introduced to resolve the singular behavior inherent in the KN-type negative flow spectral problem.
  arXiv  Detail & Related papers  (2026-07-22T01:15:11Z)
• Nonconvex Composite Functional Constraints via First-Order Augmented Lagrangian Methods under Local Regularity [10.643433352418915]
  We show uniformly augmented condition bounds of primal non-balancing non-concave minimax reformulation.<n>With dual regularization, a global local argument together with a bias.<n>$KKT.
  arXiv  Detail & Related papers  (2026-07-09T21:28:29Z)
• TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes [0.0]
  Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams.<n>We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP)<n>Our approach avoids iterative normalization and explosion while preserving full expressivity of the Birkhoff polytope.
  arXiv  Detail & Related papers  (2026-05-20T20:31:10Z)
• JPmHC Dynamical Isometry via Orthogonal Hyper-Connections [2.4311915994390403]
  JPmHC is a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams.<n>It prevents gradient pathologies and enhances stability.<n>It achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines.
  arXiv  Detail & Related papers  (2026-02-20T16:06:01Z)
• The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks [1.7523718031184992]
  We identify a critical spectral degradation phenomenon inherent to structure-preserving deep architectures.<n>We show that maximum-entropy bias drives the mixing operator towards the uniform barycenter, suppressing the subdominant singular value .<n>We derive a spectral bound linking to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow receptive field.
  arXiv  Detail & Related papers  (2026-01-05T13:09:42Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Spectral Graph Clustering under Differential Privacy: Balancing Privacy, Accuracy, and Efficiency [53.98433419539793]
  We study the problem of spectral graph clustering under edge differential privacy (DP)<n>Specifically, we develop three mechanisms: (i) graph perturbation via randomized edge flipping combined with adjacency matrix shuffling, which enforces edge privacy; (ii) private graph projection with additive Gaussian noise in a lower-dimensional space to reduce dimensionality and computational complexity; and (iii) a noisy power iteration method that distributes Gaussian noise across iterations to ensure edge DP while maintaining convergence.
  arXiv  Detail & Related papers  (2025-10-08T15:30:27Z)
• Graph-based Clustering Revisited: A Relaxation of Kernel $k$-Means Perspective [73.18641268511318]
  We propose a graph-based clustering algorithm that only relaxes the orthonormal constraint to derive clustering results.<n>To ensure a doubly constraint into a gradient, we transform the non-negative constraint into a class probability parameter.
  arXiv  Detail & Related papers  (2025-09-23T09:14:39Z)
• Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Models [66.0716790920952]
  We provide the first finite-time error bounds and convergence rate analysis for discrete diffusion models using absorbing rate matrices.<n>We establish the first convergence guarantees for both the $tau$-leaping and uniformization samplers under absorbing rate matrices.<n>Under suitable assumptions, we provide convergence guarantees without early stopping.
  arXiv  Detail & Related papers  (2025-06-02T23:14:35Z)
• Spiral flow of quantum quartic oscillator with energy cutoff [0.0]
  The cutoff dependence of the corrected matrices is found to be described by a spiral motion of a three-dimensional vector. This foundational combination of a limit-cycle and a floating fixed-point behaviors warrants further study.
  arXiv  Detail & Related papers  (2024-04-26T14:34:05Z)
• Orthogonal Nonnegative Matrix Factorization with Sparsity Constraints [0.0]
  This article presents a novel approach to solving the sparsity-constrained Orthogonal Nonnegative Matrix Factorization (SCONMF) problem.<n>By reformulating SCONMF as a capacity-constrained facility-location problem, the proposed method naturally integrates non-negativity, orthogonality, and sparsity constraints.<n>Specifically, our approach integrates control-barrier function (CBF) based framework used for dynamic optimal control design problems with maximum-entropy-principle-based framework used for facility location problems to enforce these constraints while ensuring robust factorization.
  arXiv  Detail & Related papers  (2022-10-06T04:30:59Z)
• Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation [64.49871502193477]
  We propose a novel semi-supervised subspace clustering method, which is able to simultaneously augment the initial supervisory information and construct a discriminative affinity matrix. Comprehensive experimental results on six commonly-used benchmark datasets demonstrate the superiority of our method over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-05-21T01:47:17Z)
• Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation [71.27989298860481]
  We address the non- optimisation problem of finding a matrix on the Stiefel manifold that maximises a quadratic objective function. We propose a simple yet effective sparsity-promoting algorithm for finding the dominant eigenspace matrix.
  arXiv  Detail & Related papers  (2021-09-30T19:17:35Z)
• Surveying the Multicomponent Scattering Matrix: Unitarity and Symmetries [0.0]
  We derive a robust theoretical procedure, which is fundamental in quantum-transport problems for unitarity preservation. We predict the interplay for the state-vector transfer matrix, together with the large values of its condition number, as a novel complementary tools for a more accurate definition of the threshold for tunnelling channels.
  arXiv  Detail & Related papers  (2020-10-29T20:37:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.