InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search

• URL: http://arxiv.org/abs/2606.18676v1
• Date: Wed, 17 Jun 2026 04:22:00 GMT
• Title: InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search
• Abstract summary: We operationalize intrinsic trainability through analysis of neural information processing.<n>InTrain ranks correlations on par with state-of-the-art ensemble-based proxies and outperforms other single-metric methods.
• Score: 18.006142809073044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training-free neural architecture search promises efficient discovery of high-performance networks without costly training. However, existing zero-cost proxies rely on fragmented heuristics that fail to capture the fundamental question: what makes an architecture trainable? This paper introduces Intrinsic Trainability (InTrain), a unified theoretical proxy that formalizes trainability as an architectural invariant emerging from two synergistic components: geometric capacity and optimization resilience. We operationalize intrinsic trainability through analysis of neural information processing. Geometric capacity is quantified via the participation ratio of activation covariance eigenspectrum, capturing the effective dimensionality of representation manifolds. Optimization resilience is measured through cumulative gradient health, assessing the robustness of backpropagation across network depth. InTrain synthesizes these dimensions through a scale-invariant multiplicative coupling, which we hypothesize is essential for capturing their synergistic, non-additive relationship. Extensive experiments on standard NAS benchmarks and search spaces demonstrate that InTrain achieves ranking correlations on par with state-of-the-art ensemble-based proxies and outperforms other single-metric methods.

Related papers

• T-GINEE: A Tensor-Based Multilayer Graph Representation Learning [50.294562905744215]
  T-GINEE is a statistical regularization framework combining tensor-based generalized estimating equations with task-specific loss to model cross-network correlations explicitly.<n>Our theoretical analysis establishes consistency and normality under mild conditions.<n>Extensive experiments on synthetic and real-world datasets validate T-GINEE's effectiveness for multilayer network analysis.
  arXiv  Detail & Related papers  (2026-05-27T10:51:29Z)
• Globally Optimal Training of Spiking Neural Networks via Parameter Reconstruction [21.873941320853763]
  Spiking Neural Networks (SNNs) have been proposed as biologically plausible and energy-efficient alternatives to conventional Artificial Neural Networks (ANNs)<n>We propose a parameter reconstruction algorithm for SNN training that demonstrates consistent and significant advantages across various tasks.
  arXiv  Detail & Related papers  (2026-05-08T17:10:08Z)
• ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling [57.91760520589592]
  Scaling network depth has been a central driver behind the success of modern foundation models.<n>This paper revisits the default mechanism for deepening neural networks, namely residual connections.<n>We introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data.
  arXiv  Detail & Related papers  (2026-02-09T18:54:18Z)
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning [58.533203990515034]
  Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL)<n>We show that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements.<n>We finally distill these insights into Module-Specific Training (MST), a practical framework that exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.
  arXiv  Detail & Related papers  (2025-10-14T03:03:08Z)
• Neural Networks with Orthogonal Jacobian [10.111901389604423]
  Very deep neural networks achieve state-of-the-art performance by extracting rich, hierarchical features.<n>Training them via backpropagation is often hindered by vanishing or exploding gradients.<n>We introduce a unified mathematical framework that describes a broad class of nonlinear feedforward and residual networks.
  arXiv  Detail & Related papers  (2025-08-04T20:24:49Z)
• Ray-Tracing for Conditionally Activated Neural Networks [4.9844734080376725]
  We introduce a novel architecture for conditionally activated neural networks with a sampling mechanism that converges to an optimized configuration of expert activation.<n> Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines.
  arXiv  Detail & Related papers  (2025-02-20T18:09:03Z)
• Non-asymptotic Convergence of Training Transformers for Next-token Prediction [48.9399496805422]
  Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer. We show that the trained transformer presents non-token prediction ability with dataset shift.
  arXiv  Detail & Related papers  (2024-09-25T20:22:06Z)
• Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training [0.0]
  We develop a dynamic learning rate algorithm that integrates exponential decay and advanced anti-overfitting strategies. We prove that the superlevel sets of the loss function, as influenced by our adaptive learning rate, are always connected.
  arXiv  Detail & Related papers  (2024-09-25T09:27:17Z)
• Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
  We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs) Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators. Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
  arXiv  Detail & Related papers  (2024-05-22T17:23:15Z)
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths [12.068608358926317]
  First-order optimization algorithms are known to efficiently locate favorable minima in deep neural networks. We focus on the fundamental geometric properties of sampled quantities of optimization on two key paths. Our findings suggest that not only do optimization trajectories never encounter significant obstacles, but they also maintain stable dynamics during the majority of training.
  arXiv  Detail & Related papers  (2023-06-20T22:10:40Z)
• Defensive Tensorization [113.96183766922393]
  We propose tensor defensiveization, an adversarial defence technique that leverages a latent high-order factorization of the network. We empirically demonstrate the effectiveness of our approach on standard image classification benchmarks. We validate the versatility of our approach across domains and low-precision architectures by considering an audio task and binary networks.
  arXiv  Detail & Related papers  (2021-10-26T17:00:16Z)
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [126.49572353148262]
  We study two factors in neural network training: data parallelism and sparsity. Despite their promising benefits, understanding of their effects on neural network training remains elusive.
  arXiv  Detail & Related papers  (2020-03-25T10:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.