/pdf/activation-functions-in-deep-learning-a-comprehensive-survey-11svx8wh.pdf

Activation functions in deep learning: A comprehensive survey and benchmark

The primary neural networks’ decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics. This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.

pdf/review-and-comparison-of-commonly-used-activation-functions-3l8alxnygr.pdf

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

Abstract Data scarcity is a major challenge when training deep learning (DL) models. DL demands a large amount of data to achieve exceptional performance. Unfortunately, many applications have small or inadequate data to train DL frameworks. Usually, manual labeling is needed to provide labeled data, which typically involves human annotators with a vast background of knowledge. This annotation process is costly, time-consuming, and error-prone. Usually, every DL framework is fed by a significant amount of labeled data to automatically learn representations. Ultimately, a larger amount of data would generate a better DL model and its performance is also application dependent. This issue is the main barrier for many applications dismissing the use of DL. Having sufficient data is the first step toward any successful and trustworthy DL application. This paper presents a holistic survey on state-of-the-art techniques to deal with training DL models to overcome three challenges including small, imbalanced datasets, and lack of generalization. This survey starts by listing the learning techniques. Next, the types of DL architectures are introduced. After that, state-of-the-art solutions to address the issue of lack of training data are listed, such as Transfer Learning (TL), Self-Supervised Learning (SSL), Generative Adversarial Networks (GANs), Model Architecture (MA), Physics-Informed Neural Network (PINN), and Deep Synthetic Minority Oversampling Technique (DeepSMOTE). Then, these solutions were followed by some related tips about data acquisition needed prior to training purposes, as well as recommendations for ensuring the trustworthiness of the training dataset. The survey ends with a list of applications that suffer from data scarcity, several alternatives are proposed in order to generate more data in each application including Electromagnetic Imaging (EMI), Civil Structural Health Monitoring, Medical imaging, Meteorology, Wireless Communications, Fluid Mechanics, Microelectromechanical system, and Cybersecurity. To the best of the authors’ knowledge, this is the first review that offers a comprehensive overview on strategies to tackle data scarcity in DL. 

/pdf/a-survey-on-deep-learning-tools-dealing-with-data-scarcity-3amzf9cp.pdf

A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications

This article reviews the applications of artificial neural networks (ANNs) in greenhouse technology, and also presents how this type of model can be developed in the coming years by adapting to new technologies such as the internet of things (IoT) and machine learning (ML). Almost all the analyzed works use the feedforward architecture, while the recurrent and hybrid networks are little exploited in the various tasks of the greenhouses. Throughout the document, different network training techniques are presented, where the feasibility of using optimization models for the learning process is exposed. The advantages and disadvantages of neural networks (NNs) are observed in the different applications in greenhouses, from microclimate prediction, energy expenditure, to more specific tasks such as the control of carbon dioxide. The most important findings in this work can be used as guidelines for developers of smart protected agriculture technology, in which systems involve technologies 4.0.

https://www.mdpi.com/2076-3417/10/11/3835/pdf

Applications of Artificial Neural Networks in Greenhouse Technology and Overview for Smart Agriculture Development

The integration of artificial intelligence (AI) into medical imaging has guided in an era of transformation in healthcare. This literature review explores the latest innovations and applications of AI in the field, highlighting its profound impact on medical diagnosis and patient care. The innovation segment explores cutting-edge developments in AI, such as deep learning algorithms, convolutional neural networks, and generative adversarial networks, which have significantly improved the accuracy and efficiency of medical image analysis. These innovations have enabled rapid and accurate detection of abnormalities, from identifying tumors during radiological examinations to detecting early signs of eye disease in retinal images. The article also highlights various applications of AI in medical imaging, including radiology, pathology, cardiology, and more. AI-based diagnostic tools not only speed up the interpretation of complex images but also improve early detection of disease, ultimately delivering better outcomes for patients. Additionally, AI-based image processing facilitates personalized treatment plans, thereby optimizing healthcare delivery. This literature review highlights the paradigm shift that AI has brought to medical imaging, highlighting its role in revolutionizing diagnosis and patient care. By combining cutting-edge AI techniques and their practical applications, it is clear that AI will continue shaping the future of healthcare in profound and positive ways.

How Artificial Intelligence Is Shaping Medical Imaging Technology: A Survey of Innovations and Applications

With the prevalence of digital cameras, the number of digital images increases quickly, which raises the demand for non-manual image quality assessment. While there are many methods considered useful for detecting blurriness, in this paper we propose and evaluate a new method that uses a deep convolutional neural network, which can determine whether an image is blurry or not. Experimental results demonstrate the effectiveness of the proposed scheme and are compared to deterministic methods using the confusion matrix.

pdf/convolutional-neural-network-for-blur-images-detection-as-an-3jne23ojg7.pdf

Convolutional Neural Network for Blur Images Detection as an Alternative for Laplacian Method

Rapid progress in machine learning and artificial intelligence (AI) has brought increased attention to the potential security and reliability of AI technologies. This paper identifies the threat of network incorrectly relying on counterfactual features that can stay undetectable during validation but cause serious issues in life application. Furthermore, we propose a method to counter this hazard. It combines well-known techniques: object detection tool and saliency map obtaining formula to compute metric indicating potentially faulty learning. We prove the effectiveness of the method, as well as discuss its shortcomings.

Automated Method for Evaluating Neural Network's Attention Focus.

The choice of the most appropriate activation functions for artificial neural networks has a significant effect on the training time and task performance. Nowadays the most widely-used activation function is the Rectified Linear Unit (ReLU). Despite its “dying ReLU problem” and many attempts to replace it with something better, it is still considered as default choice to begin with creation of network. Two years ago a new, promising function has been described formulated by Google Brain Team. The proposed function - named Swish - was obtained using a combination of exhaustive and reinforcement learning-based search. According to the authors, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. This paper describes an experiment on CIFAR-10 image set where Swish appears not to outperform ReLU.

Benchmarking Comparison of Swish vs. Other Activation Functions on CIFAR-10 Imageset.

Over the last years, Deep Neural Network models have been recognized as successful in solving many complex problems. However, these methods are mostly focused on the efficiency of final results and rarely provide sufficient evidence and details on factors that contribute to their outcomes This is why a growing demand for analysis techniques appeared. Thanks to visualisation techniques we can if network works as expected or even improve output of given model if possible. Moreover we can use these methods as optimization technique to boost network’s performance but pruning less important neurons. Finally, if we know how a given model works we can prepare a disruption to its work process. This paper shows how we can combine Class Activation Map with feature map to determine a few of the most contributing pixels for given input and modify them to perform an adversarial attack.

Using Convolutional Network Visualisation to Determine the Most Significant Pixels

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Tomasz Szandała

Author Tools

Chat about Author