$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

• URL: http://arxiv.org/abs/2211.10105v1
• Date: Fri, 18 Nov 2022 09:07:19 GMT
• Title: $\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling
• Authors: Bicheng Guo, Shuxuan Guo, Miaojing Shi, Peng Chen, Shibo He, Jiming Chen, Kaicheng Yu
• Abstract summary: Differentiable architecture search (DARTS) has been a mainstream direction in automatic machine learning. We propose to additionally inject semantic information by formulating a patch recovery approach. Our method surpasses all previous DARTS variants and achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet.
• Score: 25.75814720792934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable architecture search (DARTS) has been a mainstream direction in automatic machine learning. Since the discovery that original DARTS will inevitably converge to poor architectures, recent works alleviate this by either designing rule-based architecture selection techniques or incorporating complex regularization techniques, abandoning the simplicity of the original DARTS that selects architectures based on the largest parametric value, namely $\alpha$. Moreover, we find that all the previous attempts only rely on classification labels, hence learning only single modal information and limiting the representation power of the shared network. To this end, we propose to additionally inject semantic information by formulating a patch recovery approach. Specifically, we exploit the recent trending masked image modeling and do not abandon the guidance from the downstream tasks during the search phase. Our method surpasses all previous DARTS variants and achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet without complex manual-designed strategies.

Related papers

• Cross-Architecture Distillation Made Simple with Redundancy Suppression [8.844066299737845]
  We describe a simple method for cross-architecture knowledge distillation, where the knowledge transfer is cast into a redundant information suppression formulation.<n>We propose to extract the architecture-agnostic knowledge in heterogeneous representations by reducing the redundant architecture-exclusive information.<n>Our method is devoid of the architecture-specific designs and complex operations in the pioneering method of OFA.
  arXiv  Detail & Related papers  (2025-07-29T14:21:40Z)
• Exploring Synergistic Ensemble Learning: Uniting CNNs, MLP-Mixers, and Vision Transformers to Enhance Image Classification [2.907712261410302]
  We build upon and improve previous work exploring the complementarity between different architectures. We preserve the integrity of each architecture and combine them using ensemble techniques. A direct outcome of this work is the creation of an ensemble of classification networks that surpasses the accuracy of the previous state-of-the-art single classification network on ImageNet.
  arXiv  Detail & Related papers  (2025-04-12T04:32:52Z)
• A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends [67.43992456058541]
  Image restoration (IR) aims to recover high-quality images from inputs degraded by various factors such as noise, blur, compression, and adverse weather.<n>Traditional IR methods typically focus on specific types of degradation, which limits their effectiveness in real-world scenarios with complex distortions.<n>The all-in-one image restoration paradigm has recently emerged, offering a unified framework that adeptly addresses multiple degradation types.
  arXiv  Detail & Related papers  (2024-10-19T11:11:09Z)
• EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition [54.99121380536659]
  Eye movement biometrics have received increasing attention thanks to its high secure identification. Deep learning (DL) models have been recently successfully applied for eye movement recognition. DL architecture still is determined by human prior knowledge. We propose EM-DARTS, a hierarchical differentiable architecture search algorithm to automatically design the DL architecture for eye movement recognition.
  arXiv  Detail & Related papers  (2024-09-22T13:11:08Z)
• Relax DARTS: Relaxing the Constraints of Differentiable Architecture Search for Eye Movement Recognition [9.905155497581815]
  We introduce automated network search (NAS) algorithms to the field of eye movement recognition. Relax DARTS is an improvement of the Differentiable Architecture Search (DARTS) to realize more efficient network search and training. Relax DARTS exhibits adaptability to other multi-feature temporal classification tasks.
  arXiv  Detail & Related papers  (2024-09-18T02:37:04Z)
• DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation [99.88539409432916]
  We study the unsupervised domain adaptation (UDA) process. We propose a novel UDA method, DAFormer, based on the benchmark results. DAFormer significantly improves the state-of-the-art performance by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes.
  arXiv  Detail & Related papers  (2021-11-29T19:00:46Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• Rethinking Architecture Selection in Differentiable NAS [74.61723678821049]
  Differentiable Neural Architecture Search is one of the most popular NAS methods for its search efficiency and simplicity. We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet. We find that several failure modes of DARTS can be greatly alleviated with the proposed selection method.
  arXiv  Detail & Related papers  (2021-08-10T00:53:39Z)
• Neighborhood-Aware Neural Architecture Search [43.87465987957761]
  We propose a novel neural architecture search (NAS) method to identify flat-minima architectures in the search space. Our formulation takes the "flatness" of an architecture into account by aggregating the performance over the neighborhood of this architecture. Based on our formulation, we propose neighborhood-aware random search (NA-RS) and neighborhood-aware differentiable architecture search (NA-DARTS)
  arXiv  Detail & Related papers  (2021-05-13T15:56:52Z)
• Multi-Stage Progressive Image Restoration [167.6852235432918]
  We propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets.
  arXiv  Detail & Related papers  (2021-02-04T18:57:07Z)
• Towards Learning Convolutions from Scratch [34.71001535076825]
  Convolution is one of the most essential components of architectures used in computer vision. Current state-of-the-art architecture search algorithms use convolution as one of the existing modules rather than learning it from data. We propose $beta$-LASSO, a simple variant of LASSO algorithm that learns architectures with local connections.
  arXiv  Detail & Related papers  (2020-07-27T16:13:13Z)
• Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search [50.40004966087121]
  We introduce a new reinforcement learning based neural architecture search (NAS) methodology for generative adversarial network (GAN) architecture search. The key idea is to formulate the GAN architecture search problem as a Markov decision process (MDP) for smoother architecture sampling. We exploit an off-policy GAN architecture search algorithm that makes efficient use of the samples generated by previous policies.
  arXiv  Detail & Related papers  (2020-07-17T18:29:17Z)
• RC-DARTS: Resource Constrained Differentiable Architecture Search [162.7199952019152]
  We propose the resource constrained differentiable architecture search (RC-DARTS) method to learn architectures that are significantly smaller and faster. We show that the RC-DARTS method learns lightweight neural architectures which have smaller model size and lower computational complexity.
  arXiv  Detail & Related papers  (2019-12-30T05:02:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.