How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

• URL: http://arxiv.org/abs/2609.05309v1
• Date: Fri, 04 Sep 2026 16:02:20 GMT
• Title: How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
• Abstract summary: Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n.<n>We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity.
• Score: 27.28406965127085
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.

Related papers

• From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts [61.695980784491475]
  Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds.<n>We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity.
  arXiv  Detail & Related papers  (2026-09-06T05:43:22Z)
• oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions [28.273021522179423]
  Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n times n$ residual matrix.<n>Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training.<n>We propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction.
  arXiv  Detail & Related papers  (2026-09-02T14:45:42Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Spectral-Embedded Operator Learning for Three-Phase Interfacial Flow: A Ternary Cahn-Hilliard-Navier-Stokes Benchmark [0.6008132390640294]
  We introduce a three-phase interfacial-flow benchmark to examine whether the trunk coordinate representation matters for a multi-channel, interface-dominated target.<n>From 1,024 Sobol-sampled simulations spanning a nine-dimensional parameter space, we learn the mapping from physical parameters to five-channel space-time fields.<n>Results indicate that the Chebyshev representation is particularly effective for the strongly non-periodic wall-normal and temporal structure of this three-phase flow.
  arXiv  Detail & Related papers  (2026-08-29T05:45:42Z)
• PTQ4SNN: Membrane-Aware Post-Training Quantization for Spiking Neural Networks [51.21235018272886]
  Spiking neural networks (SNNs) enable sparse and event-driven computation, but their low-bit deployment remains incomplete.<n>We propose PTQ4SNN, a membrane-aware post-training quantization framework that jointly quantizes weights and recurrent membrane states using only a small calibration set.<n>Experiments on static and event-based classification and semantic segmentation show that PTQ4SNN effectively preserves model accuracy under W4 quantization and approximately 4-bit membrane precision.
  arXiv  Detail & Related papers  (2026-08-07T10:16:25Z)
• WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers [1.2973070941583096]
  WAV v1 is a lightweight residual routing method for decoder-only Transformers.<n>It reduces loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8.
  arXiv  Detail & Related papers  (2026-06-04T16:15:27Z)
• Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation [48.87937677659571]
  Hyper-Connections replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices.<n>We study how this symmetry is resolved in practice, whether streams specialize in a balanced way or exhibit dominant-stream usage.
  arXiv  Detail & Related papers  (2026-06-02T11:00:49Z)
• scHelix: Asymmetric Dual-Stream Integration via Explicit Gene-Level Disentanglement [84.88065404629079]
  scHelix is a dataset-adaptive framework that fundamentally changes how features are processed.<n> scHelix utilizes a dual-stream sparse diffusion encoder equipped with stop-gradient graph caching.<n> scHelix outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-18T15:55:13Z)
• QYOLO: Lightweight Object Detection via Quantum Inspired Shared Channel Mixing [2.571702746660217]
  This work introduces QYOLO, a quantum-inspired channel mixing framework that achieves genuine architectural compression.<n>The proposed block performs global channel recalibration through a sinusoidal mixing mechanism with shared learnable parameters.<n>QYOLOv8s achieves 21.8% reduction with 0.1 pp degradation, and full accuracy parity is recovered at no cost to compression.
  arXiv  Detail & Related papers  (2026-04-29T08:47:27Z)
• Structure-Informed Estimation for Pilot-Limited MIMO Channels via Tensor Decomposition [51.56484100374058]
  This paper formulates pilot-limited channel estimation as low-rank tensor completion from sparse observations.<n>Experiments on synthetic channels demonstrate 10-20,dB normalized mean-square error (NMSE) improvement over least-squares (LS)<n> evaluations on DeepMIMO ray-tracing channels show 24-44% additional NMSE reduction over pure tensor-based methods.
  arXiv  Detail & Related papers  (2026-02-03T23:38:05Z)
• ProReflow: Progressive Reflow with Decomposed Velocity [52.249464542399636]
  Flow matching aims to reflow the diffusion process of diffusion models into a straight line for a few-step and even one-step generation.<n>We introduce progressive reflow, which progressively reflows the diffusion models in local timesteps until the whole diffusion progresses.<n>We also introduce aligned v-prediction, which highlights the importance of direction matching in flow matching over magnitude matching.
  arXiv  Detail & Related papers  (2025-03-05T04:50:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.