Designing Compact Neural Architectures via Neuron Gating and Mixed Activation

• URL: http://arxiv.org/abs/2608.14443v1
• Date: Fri, 14 Aug 2026 16:28:32 GMT
• Title: Designing Compact Neural Architectures via Neuron Gating and Mixed Activation
• Abstract summary: We develop a general bilevel optimization framework for Neural Architecture Search (NAS)<n>We propose three scalable formulations that replace discrete neuron- and activation-level decisions with continuous relaxations.<n>Experiments show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance.
• Score: 0.18425326702824343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level optimizes the architecture using validation performance and the lower-level trains network parameters using training loss. However, NAS is computationally expensive due to discrete architectural decisions, exponentially growing search spaces, and the high cost of training candidate architectures. This work develops a general bilevel optimization framework for NAS across diverse architectures, including MLPs, CNNs, RNNs, and Transformers, to identify compact architectures with strong predictive performance. We propose three scalable formulations that replace discrete neuron- and activation-level decisions with continuous relaxations, enabling differentiable optimization over otherwise combinatorial architecture spaces. These formulations give rise to three NAS methods: NAS based on Neuron Gating (NAS-NG), NAS based on Mixed Activation (NAS-MA), and NAS based on Neuron Gating and Mixed Activation (NAS-NGMA). Experiments on MLPs and CNNs using MNIST and CIFAR-10 show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance. On MNIST, NAS-NGMA achieves 98.68% test accuracy with 7.69M MLP parameters, while NAS-NG achieves 99.63% accuracy with only 0.26M CNN parameters. On CIFAR-10, the proposed methods consistently outperform vanilla DARTS. Further experiments demonstrate that NAS-NG can optimize substantially over-parameterized and literature-optimal architectures, improving accuracy while reducing parameters. These results establish relaxed bilevel optimization as a scalable alternative to discrete NAS and provide a general framework for efficient neuron- and activation-level architecture optimization.

Related papers

• LP-NAS: Linear Programming-based Neural Architecture Search [0.18425326702824343]
  We propose Linear Programming-based NAS (LP-NAS), a mathematical programming-based framework for differentiable NAS.<n>We introduce two computationally efficient variants of LP-NAS, namely S-LP-NAS and R-LP-NAS.<n>Both variants achieve faster convergence and significantly higher validation performance during the early search iterations than the standard DARTS algorithm.
  arXiv  Detail & Related papers  (2026-08-14T16:53:11Z)
• OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search [9.616990304130203]
  We propose an optimization proxy to streamline the Neural Architecture Search (NAS) problem.<n>Using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth.<n>Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains fully demonstrate the superior search results and efficiency.
  arXiv  Detail & Related papers  (2025-09-06T09:26:02Z)
• A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [56.09418231453024]
  Neural architecture search (NAS) enables researchers to automatically explore vast search spaces and find efficient neural networks.<n>NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process.<n>We propose the SMEM-NAS, a pairwise comparison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
  arXiv  Detail & Related papers  (2024-07-22T12:46:22Z)
• DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNs [53.82853297675979]
  1-bit convolutional neural networks (CNNs) with binary weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS. We introduce Discrepant Child-Parent Neural Architecture Search (DCP-NAS) to efficiently search 1-bit CNNs.
  arXiv  Detail & Related papers  (2023-06-27T11:28:29Z)
• DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
  We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
  arXiv  Detail & Related papers  (2023-05-26T13:58:18Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework [6.895590095853327]
  Differentiable architecture search has gradually become the mainstream research topic in the field of Neural Architecture Search (NAS) Recent differentiable NAS also aims at further improving the search performance and reducing the GPU-memory consumption. We propose the TND-NAS, which is with the merits of the high efficiency in differentiable NAS framework and the compatibility among non-differentiable metrics in Multi-objective NAS.
  arXiv  Detail & Related papers  (2021-11-06T14:19:36Z)
• Effective, Efficient and Robust Neural Architecture Search [4.273005643715522]
  Recent advances in adversarial attacks show the vulnerability of deep neural networks searched by Neural Architecture Search (NAS) We propose an Effective, Efficient, and Robust Neural Architecture Search (E2RNAS) method to search a neural network architecture by taking the performance, robustness, and resource constraint into consideration. Experiments on benchmark datasets show that the proposed E2RNAS method can find adversarially robust architectures with optimized model size and comparable classification accuracy.
  arXiv  Detail & Related papers  (2020-11-19T13:46:23Z)
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery [54.60327265077322]
  We study two important problems in the automated design of neural networks through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In the second part of this paper, we establish a connection between NAS and structured sparse recovery.
  arXiv  Detail & Related papers  (2020-07-07T00:57:09Z)
• Geometry-Aware Gradient Algorithms for Neural Architecture Search [41.943045315986744]
  We argue for the study of single-level empirical risk minimization to understand NAS with weight-sharing. We present a geometry-aware framework that exploits the underlying structure of this optimization to return sparse architectural parameters. We achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision.
  arXiv  Detail & Related papers  (2020-04-16T17:46:39Z)
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture [62.587982139871976]
  We propose Broad Neural Architecture Search (BNAS) where we elaborately design broad scalable architecture dubbed Broad Convolutional Neural Network (BCNN) BNAS delivers 0.19 days which is 2.37x less expensive than ENAS who ranks the best in reinforcement learning-based NAS approaches.
  arXiv  Detail & Related papers  (2020-01-18T15:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.