Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
Lam M. Nguyen,Jie Liu,Katya Scheinberg,Martin Takáč +3 more
16
TL;DR: This paper analyzes the mini-batch SARAH algorithm for nonconvex optimization, providing sublinear and linear convergence rates for general and gradient-dominated functions, respectively, outperforming other stochastic gradient algorithms for nonconvex losses.
read more
Abstract: In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Anomaly Detection in Manufacturing Systems Using Structured Neural Networks
Jie Liu,Jianlin Guo,Philip Orlik,Shibata Masahiko,Nakahara Daiki,Satoshi Mii,Martin Takáč +6 more
- 04 Jul 2018
TL;DR: The proposed structured neural networks outperform the unstructured neural networks in terms of anomaly detection accuracy and can reduce test error by 20% and reduce anomaly detection misclassification error by as much as 64%.
55
Zeroth-Order Stochastic Alternating Direction Method of Multipliers for Nonconvex Nonsmooth Optimization
Feihu Huang,Shangqian Gao,Songcan Chen,Songcan Chen,Heng Huang +4 more
- 01 Aug 2019
TL;DR: A class of fast zeroth-order stochastic ADMM methods for solving nonconvex problems with multiple nonsmooth penalties, based on the coordinate smoothing gradient estimator, which not only reach the best convergence rate for the non Convex optimization, but also are able to effectively solve many complex machine learning problems withmultiple regularized penalties and constraints.
SCAFFOLD Stochastic Controlled Averaging for Federated Learning
Sai Praneeth Karimireddy,Satyen Kale,Mehryar Mohri,Sashank J. Reddi,Sebastian U. Stich,Ananda Theertha Suresh +5 more
TL;DR: This paper proposes SCAFFOLD, a federated learning algorithm that corrects for "client-drift" using control variates, achieving faster convergence and requiring fewer communication rounds, especially in heterogeneous data settings.
Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds
TL;DR: In this article, the Riemannian Stochastic path integrated differential estimator (R-SPIDER) was proposed to solve the finite-sum and online RiemANNian non-convex minimization problems.
References
Cubic regularization of Newton method and its global performance
Yurii Nesterov,Boris T. Polyak +1 more
TL;DR: This paper provides theoretical analysis for a cubic regularization of Newton method as applied to unconstrained minimization problem and proves general local convergence results for this scheme.
1.2K
Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
Saeed Ghadimi,Guanghui Lan +1 more
TL;DR: The randomized stochastic gradient (RSG) algorithm as mentioned in this paper is a type of approximation algorithm for non-convex nonlinear programming problems, and it has a nearly optimal rate of convergence if the problem is convex.
1.1K
Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems.
Aaron J. Defazio,Tibério S. Caetano,Justin Domke +2 more
TL;DR: Researchers introduce Finito, a faster incremental gradient method for big data problems, achieving four times faster convergence rate than existing methods, with further speed-ups through sampling without replacement, and demonstrating state-of-the-art performance in empirical results.
Introductory Lectures on Convex Optimization: A Basic Course
Yurii Nesterov
TL;DR: This book offers a modern, comprehensive introduction to convex optimization, a crucial field in applied mathematics, economics, finance, engineering, and computer science, with significant applications in data science and machine learning.
Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization
Shai Shalev-Shwartz,Tong Zhang +1 more
- 21 Jun 2014
TL;DR: The runtime of the framework is analyzed and rates that improve state-of-the-art results for various key machine learning optimization problems including SVM, logistic regression, ridge regression, Lasso, and multiclass SVM are obtained.