Open AccessProceedings Article
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Chris Ying,Aaron Klein,Eric Christiansen,Esteban Real,Kevin Murphy,Frank Hutter +5 more
- 24 May 2019
- pp 7105-7114
TL;DR: This work introduces NAS-Bench-101, the first public architecture dataset for NAS research, which allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset.
read more
Abstract: Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Posted Content
An Analysis of Super-Net Heuristics in Weight-Sharing NAS
TL;DR: In this article, the authors disentangle super-net training from the search algorithm, isolate 14 frequently-used training heuristics, and evaluate them over three benchmark search spaces.
•Posted Content
Neural networks adapting to datasets: learning network size and topology.
TL;DR: A flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training is introduced, which has the structure of a graph tailored to the particular learning task and dataset.
•Posted Content
A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search.
TL;DR: In this paper, an evolutionary algorithm for neural architecture search is proposed, which organizes the topology in multiple hierarchical modules, while the design process exploits this representation, in order to explore the search space.
•Posted Content
EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
TL;DR: In this paper, an efficient performance estimation strategy, EPE-NAS, is proposed, which mitigates the problem of evaluating networks by scoring untrained networks and creating a correlation with their trained performance.
Learning a Unified Latent Space for NAS: Toward Leveraging Structural and Symbolic Information
TL;DR: How to build a proper representation of network architecture that preserves explicit or implicit information inside the architecture is discussed, and the effectiveness of the proposed method as compared with the state-of-the-art predictors is demonstrated.
References
•Posted Content
Deep Residual Learning for Image Recognition
TL;DR: This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
117.9K
ImageNet: A large-scale hierarchical image database
Jia Deng,Wei Dong,Richard Socher,Li-Jia Li,Kai Li,Li Fei-Fei +5 more
- 20 Jun 2009
TL;DR: A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
•Proceedings Article
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever,Oriol Vinyals,Quoc V. Le +2 more
- 08 Dec 2014
TL;DR: The authors used a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector.
•Posted Content
Decoupled Weight Decay Regularization
Ilya Loshchilov,Frank Hutter +1 more
TL;DR: This work proposes a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function, and provides empirical evidence that this modification substantially improves Adam's generalization performance.
14.4K
•Journal Article
Random search for hyper-parameter optimization
James Bergstra,Yoshua Bengio +1 more
TL;DR: This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid, and shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper- parameter optimization algorithms.
Related Papers (5)
Hanxiao Liu,Karen Simonyan,Yiming Yang +2 more
- 24 Jun 2018
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016