TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

• URL: http://arxiv.org/abs/2605.21724v1
• Date: Wed, 20 May 2026 20:31:10 GMT
• Title: TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
• Abstract summary: Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams.<n>We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP)<n>Our approach avoids iterative normalization and explosion while preserving full expressivity of the Birkhoff polytope.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with $(n-1)^2$ degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training' demonstrate competitive performance with improved stability and scalability.

Related papers

• oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions [28.273021522179423]
  Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n times n$ residual matrix.<n>Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training.<n>We propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction.
  arXiv  Detail & Related papers  (2026-09-02T14:45:42Z)
• go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices [0.0]
  We introduce a novel exact parameterization grounded in the theory of generalized orthostochastic matrices.<n>We instantiate this parameterization in go-$m$HC.<n>Our method composes naturally with Kronecker-factorized methods, substantially recovering expressivity at similar FLOP costs.
  arXiv  Detail & Related papers  (2026-04-02T17:51:19Z)
• Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections [10.82991070567946]
  Unconstrained mixing disrupts the identity mapping property to the residual connection, causing unstable training.<n>We propose Spectral-Sphere-trained Hyper-Connections (sHC)<n>sHC allows negative entries, unlocking subtractive interactions for selective feature diversification.
  arXiv  Detail & Related papers  (2026-03-21T18:03:03Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• JPmHC Dynamical Isometry via Orthogonal Hyper-Connections [2.4311915994390403]
  JPmHC is a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams.<n>It prevents gradient pathologies and enhances stability.<n>It achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines.
  arXiv  Detail & Related papers  (2026-02-20T16:06:01Z)
• Stabilizing Fixed-Point Iteration for Markov Chain Poisson Equations [49.702772230127465]
  We study finite-state Markov chains with $n$ states and transition matrix $P$.<n>We show that all non-decaying modes are captured by a real peripheral invariant subspace $mathcalK(P)$, and that the induced operator on the quotient space $mathbbRn/mathcalK(P) is strictly contractive, yielding a unique quotient solution.
  arXiv  Detail & Related papers  (2026-01-31T02:57:01Z)
• KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices [6.968486021891596]
  We propose textbfKromHC, which uses the underlineKronecker products of smaller residual matrices to parametrize the residual matrix in underlinemHC.<n>Experiments demonstrate that KromHC matches or even outperforms state-of-the-art mHC variants, while requiring significantly fewer trainable parameters.
  arXiv  Detail & Related papers  (2026-01-29T11:43:05Z)
• mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations [5.518733929171501]
  Unconstrained residual matrices can compromise training stability.<n>DeepSeek's Manifold-Constrained Hyper-Connections (mHC) approximately projects these matrices onto the Birkhoff polytope via iterative Sinkhorn--Knopp (SK) normalization.
  arXiv  Detail & Related papers  (2026-01-09T11:19:14Z)
• Graph-based Clustering Revisited: A Relaxation of Kernel $k$-Means Perspective [73.18641268511318]
  We propose a graph-based clustering algorithm that only relaxes the orthonormal constraint to derive clustering results.<n>To ensure a doubly constraint into a gradient, we transform the non-negative constraint into a class probability parameter.
  arXiv  Detail & Related papers  (2025-09-23T09:14:39Z)
• Fast Semisupervised Unmixing Using Nonconvex Optimization [80.11512905623417]
  We introduce a novel convex convex model for semi/library-based unmixing. We demonstrate the efficacy of Alternating Methods of sparse unsupervised unmixing.
  arXiv  Detail & Related papers  (2024-01-23T10:07:41Z)
• Controlling the Complexity and Lipschitz Constant improves polynomial nets [55.121200972539114]
  We derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets. We propose a principled regularization scheme that we evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-02-10T14:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.