Frank–Wolfe algorithm

Abstract: c 2015 Dimitri P. Bertsekas All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

Abstract: This paper studies algorithms for solving the problem of recovering a low-rank matrix with a fraction of its entries arbitrarily corrupted. This problem can be viewed as a robust version of classical PCA, and arises in a number of application domains, including image processing, web data ranking, and bioinformatic data analysis. It was recently shown that under surprisingly broad conditions, it can be exactly solved via a convex programming surrogate that combines nuclear norm minimization and `1-norm minimization. This paper develops and compares two complementary approaches for solving this convex program. The first is an accelerated proximal gradient algorithm directly applied to the primal; while the second is a gradient algorithm applied to the dual problem. Both are several orders of magnitude faster than the previous state-of-the-art algorithm for this problem, which was based on iterative thresholding. Simulations demonstrate the performance improvement that can be obtained via these two algorithms, and clarify their relative merits.

Abstract: This chapter presents in a self-contained manner recent advances in the design and analysis of gradient-based schemes for specially structured smooth and nonsmooth minimization problems. We focus on the mathematical elements and ideas for building fast gradient-based methods and derive their complexity bounds. Throughout the chapter, the resulting schemes and results are illustrated and applied on a variety of problems arising in several specific key applications such as sparse approximation of signals, total variation-based image processing problems, and sensor location problems.

Abstract: In this paper, a distributed optimization problem with general differentiable convex objective functions is studied for continuous-time multi-agent systems with single-integrator dynamics. The objective is for multiple agents to cooperatively optimize a team objective function formed by a sum of local objective functions with only local interaction and information while explicitly taking into account nonuniform gradient gains, finite-time convergence, and a common convex constraint set. First, a distributed nonsmooth algorithm is introduced for a special class of convex objective functions that have a quadratic-like form. It is shown that all agents reach a consensus in finite time while minimizing the team objective function asymptotically. Second, a distributed algorithm is presented for general differentiable convex objective functions, in which the interaction gains of each agent can be self-adjusted based on local states. A corresponding condition is then given to guarantee that all agents reach a consensus in finite time while minimizing the team objective function asymptotically. Third, a distributed optimization algorithm with state-dependent gradient gains is given for general differentiable convex objective functions. It is shown that the distributed continuous-time optimization problem can be solved even though the gradient gains are not identical. Fourth, a distributed tracking algorithm combined with a distributed estimation algorithm is given for general differentiable convex objective functions. It is shown that all agents reach a consensus while minimizing the team objective function in finite time. Fifth, as an extension of the previous results, a distributed constrained optimization algorithm with nonuniform gradient gains and a distributed constrained finite-time optimization algorithm are given. It is shown that both algorithms can be used to solve a distributed continuous-time optimization problem with a common convex constraint set. Numerical examples are included to illustrate the obtained theoretical results.