In this technical article, to seek the optimal solution to time-dependent nonlinear optimization subject to linear inequality and equality constraints (TDNO-IEC), the gradient-based differential neural-solution, termed as GDN model, is proposed and researched. Notably, TDNO-IEC is first converted into the nonhomogeneous linear equation with the dynamic parameter. Second, differential neural-solution with the aid of gradient is designed. The contrastive theoretical analyses among the GDN model, gradient-based neural network (GNN), and the dual neural network (DNN) prove that the proposed GDN model has higher accuracy for eliminating the large solution error with exponential convergence. In addition, reasonable convergent time of the GDN model is guaranteed by activation functions with simple formulation. Last, an illustrative example and real-world applications, including robot motion planning and data dimension reduction and reconstruction, further validate the high availability of the proposed GDN model. 

Gradient-Based Differential Neural-Solution to Time-Dependent Nonlinear Optimization

In this technical article, to seek the optimal solution to time-dependent nonlinear optimization subject to linear inequality and equality constraints (TDNO-IEC), the gradient-based differential neural-solution, termed as GDN model, is proposed and researched. Notably, TDNO-IEC is first converted into the nonhomogeneous linear equation with the dynamic parameter. Second, differential neural-solution with the aid of gradient is designed. The contrastive theoretical analyses among the GDN model, gradient-based neural network (GNN), and the dual neural network (DNN) prove that the proposed GDN model has higher accuracy for eliminating the large solution error with exponential convergence. In addition, reasonable convergent time of the GDN model is guaranteed by activation functions with simple formulation. Last, an illustrative example and real-world applications, including robot motion planning and data dimension reduction and reconstruction, further validate the high availability of the proposed GDN model.

It is generally considered that controlling a robot precisely becomes tough on the condition of unknown structure information. Applying a data-driven approach to the robot control with the unknown structure implies a novel feasible research direction. Therefore, in this article, as a combination of the structural learning and robot control, an acceleration-level data-driven repetitive motion planning (DDRMP) scheme is proposed with the corresponding recurrent neural network (RNN) constructed. Then, theoretical analyses on the learning and control abilities are provided. Moreover, simulative experiments on employing the acceleration-level DDRMP scheme as well as the corresponding RNN to control a Sawyer robot and a Baxter robot with unknown structure information are performed. Accordingly, simulation results validate the feasibility of the proposed method and comparisons among the existing repetitive motion planning (RMP) schemes indicate the superiority of the proposed method. This work offers sufficient theoretical and simulative solutions for the acceleration-level redundancy problem of redundant robots with unknown structure and joint limits considered.

An Acceleration-Level Data-Driven Repetitive Motion Planning Scheme for Kinematic Control of Robots With Unknown Structure

Adversarial attacks and defenses in machine learning and deep neural network (DNN) have been gaining significant attention due to the rapidly growing applications of deep learning in communication networks. This survey provides a comprehensive overview of the recent advancements in the field of adversarial attack and defense techniques, with a focus on DNN-based classification models for communication applications. Specifically, we conduct a comprehensive classification of recent adversarial attack methods and state-of-the-art adversarial defense techniques based on attack principles, and present them in visually appealing tables and tree diagrams. This is based on a rigorous evaluation of the existing works, including an analysis of their strengths and limitations. We also categorize the methods into counter-attack detection and robustness enhancement, with a specific focus on regularization-based methods for enhancing robustness. New avenues of attack are also explored, including search-based, decision-based, drop-based, and physical-world attacks, and a hierarchical classification of the latest defense methods is provided, highlighting the challenges of balancing training costs with performance, maintaining clean accuracy, overcoming the effect of gradient masking, and ensuring method transferability. At last, the lessons learned and open challenges are summarized with future research opportunities recommended.

Adversarial Attacks and Defenses in Machine Learning-Empowered Communication Systems and Networks: A Contemporary Survey

Incremental random weight networks (IRWNs) face the issues of weak generalization and complicated network structure. There is an important reason: the learning parameters of IRWNs are determined in a random fashion without guidance, which may increase many redundant hidden nodes, and thereby leading to inferior performance. To resolve this issue, a novel IRWN with compact constraint that guides the assignment of random learning parameters (CCIRWN) is developed in this brief. Using the iteration method of Greville, a compact constraint that simultaneously assures the quality of generated hidden nodes and the convergence of the CCIRWN is built to perform learning parameter configuration. Meanwhile, the output weights of the CCIRWN are assessed analytically. Two types of learning methods for constructing the CCIRWN are proposed. Finally, the performance evaluation of the proposed CCIRWN is undertaken on the 1-D nonlinear function approximation, several real-world datasets, and data-driven estimation based on the industrial data. Numerical and industrial examples indicate that the proposed CCIRWN with compact structure can achieve favorable generalization ability.

A Compact Constraint Incremental Method for Random Weight Networks and Its Application.

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this article, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this article gives conditions that the GAF needs to meet and, on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this article proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

pdf/activated-gradients-for-deep-neural-networks-2ggeyho42s.pdf

Activated Gradients for Deep Neural Networks.

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at this https URL.

Activated Gradients for Deep Neural Networks

In deep learning, it is usually assumed that the shape of the loss surface is fixed. Differently, a novel concept of deformation operator is first proposed in this paper to deform the loss surface, thereby improving the optimization. Deformation function, as a type of deformation operator, can improve the generalization performance. Moreover, various deformation functions are designed, and their contributions to the loss surface are further provided. Then, the original stochastic gradient descent optimizer is theoretically proved to be a flat minima filter that owns the talent to filter out the sharp minima. Furthermore, the flatter minima could be obtained by exploiting the proposed deformation functions, which is verified on CIFAR-100, with visualizations of loss landscapes near the critical points obtained by both the original optimizer and optimizer enhanced by deformation functions. The experimental results show that deformation functions do find flatter regions. Moreover, on ImageNet, CIFAR-10, and CIFAR-100, popular convolutional neural networks enhanced by deformation functions are compared with the corresponding original models, where significant improvements are observed on all of the involved models equipped with deformation functions. For example, the top-1 test accuracy of ResNet-20 on CIFAR-100 increases by 1.46%, with insignificant additional computational overhead.

Deforming the Loss Surface

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Liangming Chen

Author Tools

Chat about Author

Papers

Activated Gradients for Deep Neural Networks.

Activated Gradients for Deep Neural Networks

Deforming the Loss Surface