Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns

• URL: http://arxiv.org/abs/2605.07378v1
• Date: Fri, 08 May 2026 07:33:34 GMT
• Title: Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns
• Abstract summary: We propose SWAP-Score, a novel and highly effective zero-shot metric.<n>SWAP-Score is broadly applicable across both architecture families and task domains.<n>It outperforms existing zero-shot metrics across computer vision and natural language processing tasks.
• Score: 44.380310717314735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot proxies, also known as training-free metrics, are widely adopted to reduce the computational overhead in neural network evaluation for scenarios such as Neural Architecture Search (NAS), as they do not require any training. Existing zero-shot metrics have several limitations, including weak correlation with the true performance and poor generalisation across different networks or downstream tasks. For example, most of these metrics apply only to either convolutional neural networks (CNNs) or Transformers, but not both. To address these limitations, we propose Sample-Wise Activation Patterns (SWAP), and its derivative, SWAP-Score, a novel and highly effective zero-shot metric. SWAP-Score is broadly applicable across both architecture families and task domains, demonstrating strong predictive performance in the majority of tasks. This metric measures the expressivity of neural networks over a mini-batch of samples, showing a high correlation with the neural networks' ground-truth performance. For both CNNs and Transformers, the SWAP-Score outperforms existing zero-shot metrics across computer vision and natural language processing tasks. For instance, Spearman's correlation coefficient between the SWAP-Score and CIFAR-10 validation accuracy for DARTS CNNs is 0.93, and 0.71 for FlexiBERT Transformers on GLUE tasks. Moreover, SWAP-Score is label-independent, hence can be applied at the pre-training stage of language models to estimate their performance for downstream tasks. When applied to NAS, SWAP-empowered NAS, SWAP-NAS can achieve competitive performance using only approximately 6 and 9 minutes of GPU time, on CIFAR-10 and ImageNet respectively. Our code is available at: https://github.com/pym1024/SWAP_Universal

Related papers

• Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature [8.219278958506592]
  We propose a zero-cost proxy that omits the requirement of labelled data for its computation.<n>Our approach enables accurate prediction of network performance on test data using only a single label-free data sample.
  arXiv  Detail & Related papers  (2025-08-18T14:52:14Z)
• A Pipeline of Augmentation and Sequence Embedding for Classification of Imbalanced Network Traffic [0.0]
  We propose a pipeline to balance the dataset and classify it using a robust and accurate embedding technique.<n>We demonstrate that the proposed augmentation pipeline, combined with FS-Embedding, increases convergence speed and leads to a significant reduction in the number of model parameters.
  arXiv  Detail & Related papers  (2025-02-26T07:55:24Z)
• SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS [35.041289296298565]
  Training-free metrics are widely used to avoid resource-intensive neural network training. We propose Sample-Wise Activation Patterns and its derivative, SWAP-Score, a novel high-performance training-free metric. The SWAP-Score is strongly correlated with ground-truth performance across various search spaces and tasks.
  arXiv  Detail & Related papers  (2024-03-07T02:40:42Z)
• Neural Architecture Search: Two Constant Shared Weights Initialisations [0.0]
  epsinas is a novel zero-cost NAS metric that assesses architecture potential using two constant shared weight initialisations and the statistics of their outputs.<n>We show that the dispersion of raw outputs, normalised by their average magnitude, strongly correlates with trained accuracy.<n>Our computation requires no data labels, operates on a single minibatch, and eliminates the need for gradient.
  arXiv  Detail & Related papers  (2023-02-09T02:25:38Z)
• Evolutionary Neural Cascade Search across Supernetworks [68.8204255655161]
  We introduce ENCAS - Evolutionary Neural Cascade Search. ENCAS can be used to search over multiple pretrained supernetworks. We test ENCAS on common computer vision benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T11:06:01Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Neural network relief: a pruning algorithm based on neural activity [47.57448823030151]
  We propose a simple importance-score metric that deactivates unimportant connections. We achieve comparable performance for LeNet architectures on MNIST. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations.
  arXiv  Detail & Related papers  (2021-09-22T15:33:49Z)
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift [128.32670289503025]
  Recently proposed neural architecture search (NAS) methods co-train billions of architectures in a supernet and estimate their potential accuracy. The ranking correlation between the architectures' predicted accuracy and their actual capability is incorrect, which causes the existing NAS methods' dilemma. We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift. We address these two shifts simultaneously using a nontrivial supernet-Pi model, called Pi-NAS.
  arXiv  Detail & Related papers  (2021-08-22T09:08:48Z)
• AQD: Towards Accurate Fully-Quantized Object Detection [94.06347866374927]
  We propose an Accurate Quantized object Detection solution, termed AQD, to get rid of floating-point computation. Our AQD achieves comparable or even better performance compared with the full-precision counterpart under extremely low-bit schemes.
  arXiv  Detail & Related papers  (2020-07-14T09:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.