Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation

• URL: http://arxiv.org/abs/2606.03483v1
• Date: Tue, 02 Jun 2026 11:00:49 GMT
• Title: Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
• Abstract summary: Hyper-Connections replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices.<n>We study how this symmetry is resolved in practice, whether streams specialize in a balanced way or exhibit dominant-stream usage.
• Score: 48.87937677659571
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-based language models, we trace how multi-stream representations are actually used. We find that after an early seeding stage, residual mixing often remains close to identity, limiting a core HC mechanism for exchanging information between streams. Moreover, both signal and interpretable features concentrate in a dominant stream, and the nominally multi-stream residual connection can underutilize its capacity, behaving closer to a single-stream residual pathway. Finally, we show that breaking symmetry at stream initialization reduces dominant behavior and improves performance across \textit{m}HC variants. Our code is publicly available.

Related papers

• How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing [27.28406965127085]
  Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n.<n>We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity.
  arXiv  Detail & Related papers  (2026-09-04T16:02:20Z)
• oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions [28.273021522179423]
  Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n times n$ residual matrix.<n>Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training.<n>We propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction.
  arXiv  Detail & Related papers  (2026-09-02T14:45:42Z)
• TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing [7.909872430050534]
  Residual connections rely on a static residual pathway, and are essential for training deep neural networks.<n>We propose underlinetextbfTensorized underlinetextbfEfficient underlinetextbfManifold-constrained underlinetextbfParameterization for underlinetextbfExpressive Residual underlinetextbfRouting.
  arXiv  Detail & Related papers  (2026-08-08T01:45:27Z)
• Evaluating Transformer and LSTM Frameworks for Prediction in Ungauged Basins [4.077787659104315]
  In ungauged basins, the absence of direct observations increases uncertainty and limits the ability to anticipate extreme events.<n>This study evaluates whether an encoder-only Transformer provides an advantage over an LSTM for upstream streamflow inference under limited hydrologic information.
  arXiv  Detail & Related papers  (2026-06-01T18:57:20Z)
• scHelix: Asymmetric Dual-Stream Integration via Explicit Gene-Level Disentanglement [84.88065404629079]
  scHelix is a dataset-adaptive framework that fundamentally changes how features are processed.<n> scHelix utilizes a dual-stream sparse diffusion encoder equipped with stop-gradient graph caching.<n> scHelix outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-18T15:55:13Z)
• Flow Matching with Optimized Subclass Priors for Medical Image Augmentation [10.504309161945065]
  We propose an offline strategy that introduces informative priors at two levels.<n>We partition each coarse label into coherent submodes via Gaussian mixture modeling.<n>We learn subclass-conditioned source distributions that re-center and re-scale the starting distribution per submode.
  arXiv  Detail & Related papers  (2026-05-15T12:05:15Z)
• Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections [5.356431232110434]
  We present the first open-source mHC language model to probe how parallel streams encode and utilize information.<n>We introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference.
  arXiv  Detail & Related papers  (2026-03-16T05:24:44Z)
• JPmHC Dynamical Isometry via Orthogonal Hyper-Connections [2.4311915994390403]
  JPmHC is a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams.<n>It prevents gradient pathologies and enhances stability.<n>It achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines.
  arXiv  Detail & Related papers  (2026-02-20T16:06:01Z)
• Trajectory Stitching for Solving Inverse Problems with Flow-Based Models [68.36374645801901]
  Flow-based generative models have emerged as powerful priors for solving inverse problems.<n>We propose MS-Flow, which represents the trajectory as a sequence of intermediate latent states rather than a single initial code.<n>We demonstrate the effectiveness of MS-Flow over existing methods on image recovery and inverse problems, including inpainting, super-resolution, and computed tomography.
  arXiv  Detail & Related papers  (2026-02-09T11:36:41Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems [51.99765487172328]
  Posterior sampling for inverse problem solving can be effectively achieved using flows.<n>Flow-Driven Posterior Sampling (FlowDPS) outperforms state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2025-03-11T07:56:14Z)
• Flowformer: Linearizing Transformers with Conservation Flows [77.25101425464773]
  We linearize Transformers free from specific inductive biases based on the flow network theory. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions.
  arXiv  Detail & Related papers  (2022-02-13T08:44:10Z)
• Dimensionless Anomaly Detection on Multivariate Streams with Variance Norm and Path Signature [3.1248842798600607]
  We propose a dimensionless anomaly detection method for multivariate streams. We first propose the variance norm, a generalisation of Mahalanobis distance to handle infinite-dimensional feature space. We then combine the variance norm with the path signature, an infinite collection of iterated integrals that provide global features of streams. We show that SigMahaKNN is invariant to stream reparametrisation, stream concatenation and has a graded discrimination power depending on the truncation level of the path signature.
  arXiv  Detail & Related papers  (2020-06-05T14:42:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.