EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search.
Vasco Lopes,Saeid Alirezazadeh,Luís A. Alexandre +2 more
- 14 Sep 2021
- pp 552-563
TL;DR: In this paper, an efficient performance estimation strategy, EPE-NAS, is proposed, which mitigates the problem of evaluating networks, by scoring untrained networks and correlating them with their trained performance.
read more
Abstract: Neural Architecture Search (NAS) has shown excellent results in designing architectures for computer vision problems. NAS alleviates the need for human-defined settings by automating architecture design and engineering. However, NAS methods tend to be slow, as they require large amounts of GPU computation. This bottleneck is mainly due to the performance estimation strategy, which requires the evaluation of the generated architectures, mainly through training, to update the sampler method. In this paper, we propose EPE-NAS, an efficient performance estimation strategy, that mitigates the problem of evaluating networks, by scoring untrained networks and correlating them with their trained performance. We perform this process by looking at intra and inter-class correlations of an untrained network. We show that EPE-NAS can produce a robust correlation and that by incorporating it into a simple random sampling strategy, we are able to search for competitive networks, without requiring any training, in a matter of seconds using a single GPU. Moreover, EPE-NAS is agnostic to the search method, as it focuses on evaluating untrained networks, making it easy to integrate into almost any NAS method.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Arjun Krishnakumar,Colin White,Arber Zela,Renbo Tu,Mahmoud Safari,Frank Hutter +5 more
- 06 Oct 2022
TL;DR: This work creates NAS-Bench-Suite-Zero, a procedure to improve the performance of ZC proxies by reducing biases such as cell size, and shows that incorporating all 13 ZC proxy models into the surrogate models used by NAS algorithms can improve their predictive performance by up to 42%.
•Posted Content
Evaluating Efficient Performance Estimators of Neural Architectures.
TL;DR: In this article, the authors conduct an extensive and organized assessment of OSEs and ZSEs on three NAS benchmarks: NAS-Bench-101/201/301, and reveal that they have certain biases and variances.
21
EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization
20 Jul 2023
TL;DR: In this article , the authors proposed a diversity-prompting selection strategy and a compatibility screening protocol to avoid premature convergence and improve search efficiency for mixed-precision quantization.
Efficient guided evolution for neural architecture search
09 Jul 2022
TL;DR: G-EA as discussed by the authors guides the evolution by exploring the search space by generating and evaluating several architectures in each generation at initialisation stage using a zero-proxy estimator, where only the highest-scoring architecture is trained and kept for the next generation.
10
RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies
Peijie Dong,Xin-Yi Niu,Lujun Li,Zhiliang Tian,Xiaodong Wang,Zimian Wei,Hengyue Pan,Dongsheng Li +7 more
- 24 Jan 2023
TL;DR: Zhang et al. as discussed by the authors proposed Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge.
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
102.6K
ImageNet classification with deep convolutional neural networks
TL;DR: A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
•Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky,Ilya Sutskever,Geoffrey E. Hinton +2 more
- 03 Dec 2012
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Going deeper with convolutions
Christian Szegedy,Wei Liu,Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich +8 more
- 07 Jun 2015
TL;DR: Inception as mentioned in this paper is a deep convolutional neural network architecture that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).