HyperSpace: Distributed Bayesian Hyperparameter Optimization
M. Todd Young,Jacob Hinkle,Arvind Ramanathan,Ramakrishnan Kannan +3 more
- 01 Sep 2018
- pp 339-347
20
TL;DR: HyperSpace is presented, a parallel implementation of Bayesian sequential model-based optimization that leverages high performance computing resources to better understand unknown, potentially non-convex hyperparameter search spaces and shows that it is possible to learn the dependencies between model hyperparameters through the optimization process.
read more
Abstract: As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Recent Advances in Bayesian Optimization
TL;DR: This paper attempts to provide a comprehensive and updated survey of recent advances in Bayesian optimization and identify interesting open problems and promising future research directions.
155
Efficient hyperparameter tuning for kernel ridge regression with Bayesian optimization
Annika Stuke,Patrick Rinke,Milica Todorović,Milica Todorović +3 more
- 14 Jun 2021
TL;DR: In this paper, Bayesian optimization is applied to the kernel ridge regression machine learning method for two different descriptors for the atomic structure of organic molecules, one of which introduces its own set of hyperparameters to the method.
63
•Posted Content
Efficient hyperparameter tuning for kernel ridge regression with Bayesian optimization
TL;DR: It is demonstrated that for an increasing number of hyperparameters, BO and random search become significantly more efficient in computational time than an exhaustive grid search, while delivering an equivalent or even better accuracy.
45
Distributed Bayesian optimization of deep reinforcement learning algorithms
TL;DR: An open source, distributed Bayesian model-based optimization algorithm, HyperSpace, is provided and it is shown that it consistently outperforms standard hyperparameter optimization techniques across three DRL algorithms.
38
Developing a deep learning model for the simulation of micro-pollutants in a watershed
TL;DR: In this paper, the authors used deep learning (DL) models to predict the concentrations of micro-pollutants (MPs) using deep learning and compared the results with simulation results obtained from the soil water assessment tool (SWAT) model.
23
References
•Journal Article
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +15 more
TL;DR: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems, focusing on bringing machine learning to non-specialists using a general-purpose high-level language.
•Posted Content
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Andreas Müller,Joel Nothman,Gilles Louppe,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +18 more
TL;DR: Scikit-learn as mentioned in this paper is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems.
28.9K
•Book
Machine Learning : A Probabilistic Perspective
Kevin P. Murphy
- 24 Aug 2012
TL;DR: This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach, and is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.
11.8K
•Proceedings Article
Practical Bayesian Optimization of Machine Learning Algorithms
Jasper Snoek,Hugo Larochelle,Ryan P. Adams +2 more
- 03 Dec 2012
TL;DR: This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms.
Stochastic gradient boosting
TL;DR: It is shown that both the approximation accuracy and execution speed of gradient boosting can be substantially improved by incorporating randomization into the procedure.
7.2K