Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

• URL: http://arxiv.org/abs/2603.14833v1
• Date: Mon, 16 Mar 2026 05:24:44 GMT
• Title: Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
• Abstract summary: We present the first open-source mHC language model to probe how parallel streams encode and utilize information.<n>We introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference.
• Score: 5.356431232110434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.

Related papers

• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation [48.87937677659571]
  Hyper-Connections replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices.<n>We study how this symmetry is resolved in practice, whether streams specialize in a balanced way or exhibit dominant-stream usage.
  arXiv  Detail & Related papers  (2026-06-02T11:00:49Z)
• TopoEvo: A Topology-Aware Self-Evolving Multi-Agent Framework for Root Cause Analysis in Microservices [10.602349579405832]
  Root cause analysis (RCA) intricmetrics is challenging due to noisy and heterogeneous multimodal observability.<n>Recent LLM-based RCA agents can generate tool-grounded explanations, yet they often remain topology-agnostic.<n>We propose textbfTopoEvo, a topology-aware self-evolving multi-agent that couples graph representation learning with structured, topology-constrained reasoning.
  arXiv  Detail & Related papers  (2026-05-15T04:45:44Z)
• Spontaneous symmetry breaking and Goldstone modes for deep information propagation [43.66941556185932]
  In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes.<n>In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry.<n>We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations.
  arXiv  Detail & Related papers  (2026-05-14T10:59:28Z)
• A Learning-Based Superposition Operator for Non-Renewal Arrival Processes in Queueing Networks [0.0]
  Superposition of arrival processes is a fundamental yet analytically intractable operation in queueing networks.<n>We propose a scalable data-driven operator that maps low-order moments and autocorrelation descriptors of multiple arrival streams to those of their merged process.
  arXiv  Detail & Related papers  (2026-03-11T13:46:32Z)
• Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA [61.12136997430116]
  Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.<n> directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) catastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) textitinefficient communication and convergence during model aggregation process,
  arXiv  Detail & Related papers  (2026-02-24T02:45:32Z)
• Distributed physics-informed neural networks via domain decomposition for fast flow reconstruction [8.614942690565782]
  PINs offer a powerful paradigm for flow reconstruction, seamlessly integrating velocity measurements with the governing Navier-N equations to recover complete velocity and latent pressure fields.<n>A critical challenge in such distributed PINs is pressure indeterminacy, where independent sub-networks drift into inconsistent local pressure baselines.<n>By enforcing a unidirectional flow from designated master ranks, our approach eliminates uniqueness and guarantees global pressure while preserving temporal continuity.
  arXiv  Detail & Related papers  (2026-02-05T16:41:55Z)
• Deep Learning for Contextualized NetFlow-Based Network Intrusion Detection: Methods, Data, Evaluation and Deployment [5.402853794565817]
  This paper synthesizes recent research on context-aware deep learning for flow-based intrusion detection.<n>We organize existing methods into a four-dimensional taxonomy covering temporal context, graph or relational context, multimodal context, and multi-resolution context.<n>We review common failure modes that can inflate reported results, including temporal leakage, data splitting, dataset design flaws, limited dataset diversity, and weak cross-dataset generalization.
  arXiv  Detail & Related papers  (2026-02-05T12:25:18Z)
• ReGAIN: Retrieval-Grounded AI Framework for Network Traffic Analysis [5.887997322139195]
  ReGAIN is a framework that combines traffic summarization, retrieval-augmented generation (RAG), and Large Language Model (LLM) reasoning for transparent and accurate network traffic analysis.<n> evaluated on ICMP ping flood and TCP SYN flood traces from the real-world traffic dataset.
  arXiv  Detail & Related papers  (2025-12-23T00:16:14Z)
• Unleashing Degradation-Carrying Features in Symmetric U-Net: Simpler and Stronger Baselines for All-in-One Image Restoration [52.82397287366076]
  All-in-one image restoration aims to handle diverse degradations (e.g., noise, blur, adverse weather) within a unified framework.<n>In this work, we reveal a critical insight: well-crafted feature extraction inherently encodes degradation-carrying information.<n>Our symmetric design preserves intrinsic degradation signals robustly, rendering simple additive fusion in skip connections.
  arXiv  Detail & Related papers  (2025-12-11T12:20:31Z)
• StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA [60.86024022291499]
  We introduce StreamingCoT, the first dataset explicitly designed for temporally evolving reasoning in streaming video streams.<n>Our framework generates per-second dense descriptions and constructs temporally-dependent semantic segments through similarity fusion.<n>This dataset establishes a foundation for advancing research in streaming video understanding, complex temporal reasoning, and multimodal inference.
  arXiv  Detail & Related papers  (2025-10-29T09:47:38Z)
• DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection [52.74152717667157]
  We propose a lightweight module called Dual Attention Module (DAM) for capturing cross-dimension interaction relationships in-temporal skeletal data. It employs the frame attention mechanism to identify the most significant frames and the skeleton attention mechanism to capture broader relationships across fixed partitions with minimal parameters and flops.
  arXiv  Detail & Related papers  (2024-06-05T06:18:03Z)
• SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion [59.96233305733875]
  Time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations. This paper presents an efficient-based model, the Series-cOre Fused Time Series forecaster (SOFTS)
  arXiv  Detail & Related papers  (2024-04-22T14:06:35Z)
• Collaborative Feedback Discriminative Propagation for Video Super-Resolution [66.61201445650323]
  Key success of video super-resolution (VSR) methods stems mainly from exploring spatial and temporal information. Inaccurate alignment usually leads to aligned features with significant artifacts. propagation modules only propagate the same timestep features forward or backward.
  arXiv  Detail & Related papers  (2024-04-06T22:08:20Z)
• Forecast Network-Wide Traffic States for Multiple Steps Ahead: A Deep Learning Approach Considering Dynamic Non-Local Spatial Correlation and Non-Stationary Temporal Dependency [6.019104024723682]
  This research studies two particular problems in traffic forecasting: (1) capture the dynamic and non-local spatial correlation between traffic links and (2) model the dynamics of temporal dependency for accurate multiple steps ahead predictions. We propose a deep learning framework named Spatial-Temporal Sequence to Sequence model (STSeq2Seq) to address these issues. This model builds on sequence to sequence (seq2seq) architecture to capture temporal feature and relies on graph convolution for aggregating spatial information.
  arXiv  Detail & Related papers  (2020-04-06T03:40:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.