oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

• URL: http://arxiv.org/abs/2609.02672v1
• Date: Wed, 02 Sep 2026 14:45:42 GMT
• Title: oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions
• Abstract summary: Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n times n$ residual matrix.<n>Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training.<n>We propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction.
• Score: 28.273021522179423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n \times n$ residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We prove that inside this set the mixing step can reduce the norm of the residual streams only by shrinking the differences between the streams, while their mean is left unchanged; and since the reduction accumulates over layers, the streams grow more alike and their diversity is spent with depth. We therefore propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction, which keeps training stable and no longer forces the differences between the streams to contract. Specifically, at the four streams used by recent HC models we parameterize the group in closed form by a pair of unit quaternions, which adds no parameters, replaces the iterative projection with a fixed pattern of signed additions, and can be constructed faster than mHC. We evaluate oHC across a comprehensive set of downstream tasks, where it outperforms the single-stream residual baseline, mHC and iHC, which fixes the residual matrix to the identity.

Related papers

• Nonlocality-induced critical-length hierarchy from non-Hermitian competition [48.9823527611878]
  In short-range systems formed by two coupled chains, competition leads to the critical-length law $N_csimln D$.<n>Here we show that long-range hoppings fundamentally reorganizes this critical behavior, producing a hierarchy of distinct scaling laws.<n>Our results offer a platform-independent framework testable in programmable topoelectrical circuits, photonic lattices and digital quantum simulators.
  arXiv  Detail & Related papers  (2026-08-03T18:00:18Z)
• CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning [47.406985340436925]
  CoCurve is a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly.<n>It reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.
  arXiv  Detail & Related papers  (2026-07-20T05:27:57Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation [48.87937677659571]
  Hyper-Connections replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices.<n>We study how this symmetry is resolved in practice, whether streams specialize in a balanced way or exhibit dominant-stream usage.
  arXiv  Detail & Related papers  (2026-06-02T11:00:49Z)
• TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes [0.0]
  Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams.<n>We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP)<n>Our approach avoids iterative normalization and explosion while preserving full expressivity of the Birkhoff polytope.
  arXiv  Detail & Related papers  (2026-05-20T20:31:10Z)
• scHelix: Asymmetric Dual-Stream Integration via Explicit Gene-Level Disentanglement [84.88065404629079]
  scHelix is a dataset-adaptive framework that fundamentally changes how features are processed.<n> scHelix utilizes a dual-stream sparse diffusion encoder equipped with stop-gradient graph caching.<n> scHelix outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-18T15:55:13Z)
• Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections [10.82991070567946]
  Unconstrained mixing disrupts the identity mapping property to the residual connection, causing unstable training.<n>We propose Spectral-Sphere-trained Hyper-Connections (sHC)<n>sHC allows negative entries, unlocking subtractive interactions for selective feature diversification.
  arXiv  Detail & Related papers  (2026-03-21T18:03:03Z)
• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices [6.968486021891596]
  We propose textbfKromHC, which uses the underlineKronecker products of smaller residual matrices to parametrize the residual matrix in underlinemHC.<n>Experiments demonstrate that KromHC matches or even outperforms state-of-the-art mHC variants, while requiring significantly fewer trainable parameters.
  arXiv  Detail & Related papers  (2026-01-29T11:43:05Z)
• mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations [5.518733929171501]
  Unconstrained residual matrices can compromise training stability.<n>DeepSeek's Manifold-Constrained Hyper-Connections (mHC) approximately projects these matrices onto the Birkhoff polytope via iterative Sinkhorn--Knopp (SK) normalization.
  arXiv  Detail & Related papers  (2026-01-09T11:19:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.