Loss Functions for Predictor-based Neural Architecture Search

• URL: http://arxiv.org/abs/2506.05869v1
• Date: Fri, 06 Jun 2025 08:36:46 GMT
• Title: Loss Functions for Predictor-based Neural Architecture Search
• Authors: Han Ji, Yuqi Feng, Jiahao Fan, Yanan Sun,
• Abstract summary: We conduct the first comprehensive study on loss functions in performance predictors, categorizing them into three main types: regression, ranking, and weighted loss functions.<n>Our results reveal that specific categories of loss functions can be effectively combined to enhance predictor-based NAS.
• Score: 6.014777261874645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluation is a critical but costly procedure in neural architecture search (NAS). Performance predictors have been widely adopted to reduce evaluation costs by directly estimating architecture performance. The effectiveness of predictors is heavily influenced by the choice of loss functions. While traditional predictors employ regression loss functions to evaluate the absolute accuracy of architectures, recent approaches have explored various ranking-based loss functions, such as pairwise and listwise ranking losses, to focus on the ranking of architecture performance. Despite their success in NAS, the effectiveness and characteristics of these loss functions have not been thoroughly investigated. In this paper, we conduct the first comprehensive study on loss functions in performance predictors, categorizing them into three main types: regression, ranking, and weighted loss functions. Specifically, we assess eight loss functions using a range of NAS-relevant metrics on 13 tasks across five search spaces. Our results reveal that specific categories of loss functions can be effectively combined to enhance predictor-based NAS. Furthermore, our findings could provide practical guidance for selecting appropriate loss functions for various tasks. We hope this work provides meaningful insights to guide the development of loss functions for predictor-based methods in the NAS community.

Related papers

• Evaluating Loss Functions for Graph Neural Networks: Towards Pretraining and Generalization [1.2522462543913029]
  The study looked at both inductive and transductive settings.<n>We meticulously analyzed the top ten model-loss combinations for each metric based on their average rank.<n>The GIN architecture always showed the highest-level average performance, especially with Cross-Entropy loss.
  arXiv  Detail & Related papers  (2025-06-17T02:12:19Z)
• Loss Functions in Deep Learning: A Comprehensive Review [3.8001666556614446]
  Loss functions are at the heart of deep learning, shaping how models learn and perform across diverse tasks.<n>This paper presents a comprehensive review of loss functions, covering fundamental metrics like Mean Squared Error and Cross-Entropy to advanced functions such as Adversarial and Diffusion losses.
  arXiv  Detail & Related papers  (2025-04-05T18:07:20Z)
• A survey and taxonomy of loss functions in machine learning [51.35995529962554]
  We present a comprehensive overview of the most widely used loss functions across key applications, including regression, classification, generative modeling, ranking, and energy-based modeling. We introduce 43 distinct loss functions, structured within an intuitive taxonomy that clarifies their theoretical foundations, properties, and optimal application contexts.
  arXiv  Detail & Related papers  (2023-01-13T14:38:24Z)
• Evaluating the Impact of Loss Function Variation in Deep Learning for Classification [0.0]
  The loss function is arguably among the most important hyper parameters for a neural network. We consider deep neural networks in a supervised classification setting and analyze the impact the choice of loss function has onto the training result. While certain loss functions perform suboptimally, our work empirically shows that under-represented losses can outperform the State-of-the-Art choices significantly.
  arXiv  Detail & Related papers  (2022-10-28T09:10:10Z)
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks [78.27036391638802]
  AutoLoss-Zero is the first framework for searching loss functions from scratch for generic tasks. A loss-rejection protocol and a gradient-equivalence-check strategy are developed so as to improve the search efficiency. Experiments on various computer vision tasks demonstrate that our searched loss functions are on par with or superior to existing loss functions.
  arXiv  Detail & Related papers  (2021-03-25T17:59:09Z)
• Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search [101.73248560009124]
  We propose an effective convergence-simulation driven evolutionary search algorithm, CSE-Autoloss, for speeding up the search progress. We conduct extensive evaluations of loss function search on popular detectors and validate the good generalization capability of searched losses. Our experiments show that the best-discovered loss function combinations outperform default combinations by 1.1% and 0.8% in terms of mAP for two-stage and one-stage detectors.
  arXiv  Detail & Related papers  (2021-02-09T08:34:52Z)
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation [56.343646789922545]
  We propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metric. Experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently.
  arXiv  Detail & Related papers  (2020-10-15T17:59:08Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)
• Mixability of Integral Losses: a Key to Efficient Online Aggregation of Functional and Probabilistic Forecasts [72.32459441619388]
  We adapt basic mixable (and exponentially concave) loss functions to compare functional predictions and prove that these adaptations are also mixable (exp-concave)<n>As an application of our main result, we prove that various loss functions used for probabilistic forecasting are mixable (exp-concave)
  arXiv  Detail & Related papers  (2019-12-15T14:25:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.