JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

• URL: http://arxiv.org/abs/2602.18308v1
• Date: Fri, 20 Feb 2026 16:06:01 GMT
• Title: JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
• Abstract summary: JPmHC is a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams.<n>It prevents gradient pathologies and enhances stability.<n>It achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines.
• Score: 2.4311915994390403
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivity patterns. While these innovations yield significant performance gains, they compromise the identity mapping property of residual connections, leading to training instability, limited scalability, and increased memory overhead. To address these challenges, we propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections), a framework that replaces identity skips with a trainable linear mixer acting on n parallel streams while explicitly controlling gradient conditioning. By constraining the mixer M on operator-norm-bounded manifolds (e.g., bistochastic, Stiefel, Grassmann), JPmHC prevents gradient pathologies and enhances stability. JPmHC introduces three key contributions: (i) a free-probability analysis that predicts Jacobian spectra for structured skips, providing actionable design rules for mixer selection; (ii) memory-efficient implicit differentiation for fixed-point projections, reducing activation memory and synchronization overhead; and (iii) a Stiefel-constrained mixer via Cayley transforms, ensuring orthogonality without post-hoc normalization. Empirical evaluations on ARC-AGI demonstrate that JPmHC achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines. As a flexible and scalable extension of HC, JPmHC advances spectrum-aware, stable, and efficient deep learning, offering insights into topological architecture design and foundational model evolution.

Related papers

• Post-training Quantization for Hybrid Iterative Generative Models [52.791973512723324]
  Iterative Generative Models (IGMs) span autoregressive and diffusion paradigms, and hybrid variants that couple them can achieve remarkable image-generation fidelity.<n>Post-training Quantization (PTQ) is appealing for acceleration, while directly applying vanilla PTQ to hybrid IGMs can trigger model collapse.<n>We introduce HyGenQ, a PTQ framework for hybrid IGMs.<n>It successfully quantizes representative hybrid IGMs to 8-bit precision (W8A8), significantly outperforming existing baselines and validating its robustness across different model families.
  arXiv  Detail & Related papers  (2026-08-14T04:07:14Z)
• A Quantum-Classical Surrogate Model for the Collision Operator of the Lattice Boltzmann Method [1.0554048699217669]
  We introduce a hybrid approach utilising a quantum machine learning surrogate model to approximate the non-linear collision dynamics of the LBM.<n>Our results demonstrate that the hybrid model achieves high accuracy and generalisability while closely replicating classical solutions.
  arXiv  Detail & Related papers  (2026-06-30T08:47:36Z)
• On the Complementarity of Quantum and Classical Features: Adaptive Hybrid Quantum-Classical Feature Fusion for Breast Cancer Classification [1.4856472820492366]
  The integration of quantum machine learning with classical deep learning offers promising avenues for medical image analysis.<n>We propose a novel hybrid quantum-classical architecture for breast cancer diagnosis based on a dual-branch feature-extraction pipeline.<n>Our framework extracts and unifies complementary representations from classical models and quantum circuits.
  arXiv  Detail & Related papers  (2026-04-24T14:54:53Z)
• A Deep Equilibrium Network for Hyperspectral Unmixing [15.1730669896454]
  We propose DEQ-Unmix, which reformulates abundance estimation as a deep equilibrium model.<n>It replaces the operator of the data reconstruction term with a trainable convolutional network to capture spectral-spatial information.<n>Experiments on synthetic and two real-world datasets demonstrate that DEQ-Unmix achieves superior unmixing performance while maintaining constant memory cost.
  arXiv  Detail & Related papers  (2026-04-13T10:38:28Z)
• Extended Hybrid Timed Petri Nets with Semi-Supervised Anomaly Detection for Switched Systems, Modelling and Fault Detection [0.0]
  This paper proposes a unified fault detection framework for hybrid dynamical systems.<n>It integrates an Extended Timed Continuous Petri Net model with semi-supervised anomaly detection.<n>Results demonstrate high detection accuracy, fast convergence, and robust performance.
  arXiv  Detail & Related papers  (2026-04-05T10:54:15Z)
• SKANet: A Cognitive Dual-Stream Framework with Adaptive Modality Fusion for Robust Compound GNSS Interference Classification [47.20483076887704]
  Global Navigation Satellite Systems (GNSS) face growing threats from sophisticated jamming interference.<n>We propose a cognitive deep learning framework built upon a dual-stream architecture that integrates Time-Frequency Images (TFIs) and Power Spectral Density (PSD)<n>We show that SKANet achieves an overall accuracy of 96.99%, exhibiting superior robustness for compound jamming classification.
  arXiv  Detail & Related papers  (2026-01-19T07:42:45Z)
• mHC: Manifold-Constrained Hyper-Connections [43.69451283828811]
  Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm by expanding the residual stream width and diversifying connectivity patterns.<n>We propose Manifold-Constrained Hyper-Connections (mHC) to restore the identity mapping property intrinsic to the residual connection.<n>mHC is effective for training at scale, offering tangible performance improvements and superior scalability.
  arXiv  Detail & Related papers  (2025-12-31T14:16:26Z)
• HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation [72.69742127579508]
  Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models)<n>In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors.<n> Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge.
  arXiv  Detail & Related papers  (2025-11-25T17:23:38Z)
• Adapformer: Adaptive Channel Management for Multivariate Time Series Forecasting [49.40321003932633]
  Adapformer is an advanced Transformer-based framework that merges the benefits of CI and CD methodologies through effective channel management.<n>Adapformer achieves superior performance over existing models, enhancing both predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-11-18T16:24:05Z)
• Bifidelity Karhunen-Loève Expansion Surrogate with Active Learning for Random Fields [0.4899818550820576]
  We present a bifidelity Karhunen-Loeve expansion (KLE) surrogate model for field-valued quantities of interest (QoIs) under uncertain inputs.<n>We form an active learning strategy that adaptively selects new HF evaluations based on the surrogate's generalization error.<n>New HF samples are then acquired by maximizing an expected improvement criterion, targeting regions of high surrogate error.
  arXiv  Detail & Related papers  (2025-11-05T04:14:44Z)
• Flow-Matching Guided Deep Unfolding for Hyperspectral Image Reconstruction [53.26903617819014]
  Flow-Matching-guided Unfolding network (FMU) is first to integrate flow matching into HSI reconstruction.<n>To further strengthen the learned dynamics, we introduce a mean velocity loss.<n>Experiments on both simulated and real datasets show that FMU significantly outperforms existing approaches in reconstruction quality.
  arXiv  Detail & Related papers  (2025-10-02T11:32:00Z)
• Advanced Hybrid Transformer LSTM Technique with Attention and TS Mixer for Drilling Rate of Penetration Prediction [0.9282594860064428]
  This study presents a new deep learning Hybrid LSTM-Trans-Mixer-Att framework for rate of Penetration prediction.<n>The proposed framework combines sequential memory, static feature interactions, global context learning, and dynamic feature weighting.<n> Experimental validation on real-world drilling datasets demonstrates superior performance, achieving an Rsquare of 0.9991 and a MAPE of 1.447%.
  arXiv  Detail & Related papers  (2025-08-07T09:45:56Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• Nonparametric learning of covariate-based Markov jump processes using RKHS techniques [3.3005714301829148]
  We propose a novel nonparametric approach for linking co variables to Continuous Time Markov Chains (CTMCs)<n>CTMCs provide a robust framework for modeling transitions across clinical or behavioral states.<n>We use a generalized Representer Theorem to enable tractable inference in functional space.
  arXiv  Detail & Related papers  (2025-05-06T02:26:02Z)
• Quantized and Asynchronous Federated Learning [22.40154714677385]
  We develop a novel scheme, Quantized Federated AsynchronousQAL, to deal with the communication bottleneck. We prove that QAL achieves $mathtcalqr$dic convergence without requiring uniform client arrivals. We validate our theoretical findings by using standard benchmarks.
  arXiv  Detail & Related papers  (2024-09-30T21:22:41Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.