Book Chapter10.1007/978-981-19-0332-8_38
A Comparative Study of Hyperparameter Optimization Techniques for Deep Learning
Hong Yu
- 01 Jan 2022
pp 509-521
10
TL;DR: In this article , the authors analyzed which algorithm takes the longest optimization time to optimize an architecture and whether the performance of HPO algorithms is consistent across different datasets and architectures, including Grid search (GS), Genetic algorithm (GA), Bayesian optimization (BO), Random Search (RS), Hyperband (HB), and Particle Swarm Optimization (PSO) algorithms.
read more
Abstract: Algorithms for deep learning (DL) have been widely employed in a variety of applications and fields. The hyperparameters of a deep learning model must be optimized to match different challenges. For deep learning models, choosing the optimum hyperparameter configuration has a direct influence on the model’s performance. It typically involves a thorough understanding of deep learning algorithms and their hyperparameter optimization (HPO) techniques. Although there are various automatic optimization approaches available, each has its own set of advantages and disadvantages when applied to different datasets and architectures. In this paper, we analyzed which algorithm takes the longest optimization time to optimize an architecture and whether the performance of HPO algorithms is consistent across different datasets and architectures. We selected VGG16 and ResNet50 architectures, CIFAR10 and Intel Image Classification Dataset, as well as Grid search (GS), Genetic algorithm (GA), Bayesian optimization (BO), Random search (RS), Hyperband (HB) and Particle swarm optimization (PSO) HPO algorithms for comparison. Due to the lack of pattern, it is challenging to determine which approach obtains the best performance on different datasets and architecture. The results show that all of the algorithms have similar results in terms of optimization time. This research is expected to aid DL users, developers, data analysts, and researchers in their attempts to use and adapt DL models utilizing appropriate HPO methodologies and frameworks. It will also help to better understand the challenges that currently exist in the HPO field, allowing future research into HPO and DL applications to move forward.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Development of a deep learning-based surveillance system for forest fire detection and monitoring using UAV
TL;DR: A deep learning-based surveillance system for forest fire detection and monitoring using UAV is developed. The system utilizes YOLOv8 and YOLOv5 object detection algorithms and a CNN-RCNN network for fire detection and classification. The system is embedded on an NVIDIA Jetson Nano and includes a ground station interface for data visualization. The system achieved high accuracy in fire detection and monitoring, with results of up to 96%.
15
A deep learning‐based multivariate decomposition and ensemble framework for container throughput forecasting
Anurag Kulshrestha,Abhishek Yadav,Himanshu Sharma,Shikha Suman +3 more
TL;DR: A deep learning-based multivariate decomposition and ensemble framework for container throughput forecasting accurately forecasts container throughput using economic indicators and advanced deep learning techniques.
2
Inception sh: a new cnn model based on inception module for classifying scene images
Sedat Metlek,Halit ÇETİNER +1 more
TL;DR: Inception SH model is a lightweight CNN model based on Inception V3 model that achieves higher performance values than the underlying model on scene image classification tasks.
1
Rapid Quantitative Analysis of Active Ingredients in Radix Gentianae Macrophyllae Extracts Based on Near-Infrared Spectroscopy and Deep Learning
Cheng Peng,Mengyu Zhang,Ming Kong,S. S. Zhang,Chang Li,Tingting Feng,Weilu Tian,Lie Nie,H. L. Zang +8 more
- 01 Jan 2024
TL;DR: Rapid and accurate analysis of active ingredients in Radix Gentianae Macrophyllae extracts based on near-infrared spectroscopy and deep learning.
Exploring Hyperspectral Imaging and 3D Convolutional Neural Network for Stress Classification in Plants
TL;DR: The potency of HSI combined with 3D-CNN in agriculture domain where plant health and vitality are paramount is explored, and the role of individual channels and channel groups within the color and near-infrared spectrum in predicting results for each stress-level group is investigated.
References
Particle swarm optimization
James Kennedy,Russell C. Eberhart +1 more
- 06 Aug 2002
TL;DR: A concept for the optimization of nonlinear functions using particle swarm methodology is introduced, and the evolution of several paradigms is outlined, and an implementation of one of the paradigm is discussed.
44.1K
Grey Wolf Optimizer
TL;DR: The results of the classical engineering design problems and real application prove that the proposed GWO algorithm is applicable to challenging problems with unknown search spaces.
15K
The Whale Optimization Algorithm
Seyedali Mirjalili,Andrew Lewis +1 more
TL;DR: Optimization results prove that the WOA algorithm is very competitive compared to the state-of-art meta-heuristic algorithms as well as conventional methods.
11.1K
Firefly algorithms for multimodal optimization
Xin-She Yang
- 26 Oct 2009
TL;DR: In this article, a new Firefly Algorithm (FA) was proposed for multimodal optimization applications. And the proposed FA was compared with other metaheuristic algorithms such as particle swarm optimization (PSO).
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Li Yang,Abdallah Shami +1 more
TL;DR: This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively and introducing several state-of-the-art optimization techniques.