Abstract: Many mechanics and physics problems have variational formulations making them appropriate for numerical treatment by finite element techniques and efficient iterative methods. This book describes the mathematical background and reviews the techniques for solving problems, including those that require large computations such as transonic flows for compressible fluids and the Navier-Stokes equations for incompressible viscous fluids. Finite element approximations and non-linear relaxation, augmented Lagrangians, and nonlinear least square methods are all covered in detail, as are many applications. "Numerical Methods for Nonlinear Variational Problems," originally published in the Springer Series in Computational Physics, is a classic in applied mathematics and computational physics and engineering. This long-awaited softcover re-edition is still a valuable resource for practitioners in industry and physics and for advanced students.

Abstract: In this paper, we describe the embedded conic solver (ECOS), an interior-point solver for second-order cone programming (SOCP) designed specifically for embedded applications. ECOS is written in low footprint, single-threaded, library-free ANSI-C and so runs on most embedded platforms. The main interior-point algorithm is a standard primal-dual Mehrotra predictor-corrector method with Nesterov-Todd scaling and self-dual embedding, with search directions found via a symmetric indefinite KKT system, chosen to allow stable factorization with a fixed pivoting order. The indefinite system is solved using Davis' SparseLDL package, which we modify by adding dynamic regularization and iterative refinement for stability and reliability, as is done in the CVXGEN code generation system, allowing us to avoid all numerical pivoting; the elimination ordering is found entirely symbolically. This keeps the solver simple, only 750 lines of code, with virtually no variation in run time. For small problems, ECOS is faster than most existing SOCP solvers; it is still competitive for medium-sized problems up to tens of thousands of variables.

Abstract: Direction of arrival (DOA) estimation is a classical problem in signal processing with many practical applications. Its research has recently been advanced owing to the development of methods based on sparse signal reconstruction. While these methods have shown advantages over conventional ones, there are still difficulties in practical situations where true DOAs are not on the discretized sampling grid. To deal with such an off-grid DOA estimation problem, this paper studies an off-grid model that takes into account effects of the off-grid DOAs and has a smaller modeling error. An iterative algorithm is developed based on the off-grid model from a Bayesian perspective while joint sparsity among different snapshots is exploited by assuming a Laplace prior for signals at all snapshots. The new approach applies to both single snapshot and multi-snapshot cases. Numerical simulations show that the proposed algorithm has improved accuracy in terms of mean squared estimation error. The algorithm can maintain high estimation accuracy even under a very coarse sampling grid.

Abstract: Registration is a fundamental task in computer vision. The Iterative Closest Point (ICP) algorithm is one of the widely-used methods for solving the registration problem. Based on local iteration, ICP is however well-known to suffer from local minima. Its performance critically relies on the quality of initialization, and only local optimality is guaranteed. This paper provides the very first globally optimal solution to Euclidean registration of two 3D point sets or two 3D surfaces under the L2 error. Our method is built upon ICP, but combines it with a branch-and-bound (BnB) scheme which searches the 3D motion space SE(3) efficiently. By exploiting the special structure of the underlying geometry, we derive novel upper and lower bounds for the ICP error function. The integration of local ICP and global BnB enables the new method to run efficiently in practice, and its optimality is exactly guaranteed. We also discuss extensions, addressing the issue of outlier robustness.

Abstract: We present a randomized iterative algorithm that exponentially converges in the mean square to the minimum $\ell_2$-norm least squares solution of a given linear system of equations. The expected number of arithmetic operations required to obtain an estimate of given accuracy is proportional to the squared condition number of the system multiplied by the number of nonzero entries of the input matrix. The proposed algorithm is an extension of the randomized Kaczmarz method that was analyzed by Strohmer and Vershynin.

Abstract: This book is devoted to a novel approach for dimensionality reduction based on the famous nearest neighbor method that is a powerful classification and regression approach. It starts with an introduction to machine learning concepts and a real-world application from the energy domain. Then, unsupervised nearest neighbors (UNN) is introduced as efficient iterative method for dimensionality reduction. Various UNN models are developed step by step, reaching from a simple iterative strategy for discrete latent spaces to a stochastic kernel-based algorithm for learning submanifolds with independent parameterizations. Extensions that allow the embedding of incomplete and noisy patterns are introduced. Various optimization approaches are compared, from evolutionary to swarm-based heuristics. Experimental comparisons to related methodologies taking into account artificial test data sets and also real-world data demonstrate the behavior of UNN in practical scenarios. The book contains numerous color figures to illustrate the introduced concepts and to highlight the experimental results.

Abstract: In this paper, we present a new method for the control of soft robots with elastic behavior, piloted by several actuators. The central contribution of this work is the use of the Finite Element Method (FEM), computed in real-time, in the control algorithm. The FEM based simulation computes the nonlinear deformations of the robots at interactive rates. The model is completed by Lagrange multipliers at the actuation zones and at the end-effector position. A reduced compliance matrix is built in order to deal with the necessary inversion of the model. Then, an iterative algorithm uses this compliance matrix to find the contribution of the actuators (force and/or position) that will deform the structure so that the terminal end of the robot follows a given position. Additional constraints, like rigid or deformable obstacles, or the internal characteristics of the actuators are integrated in the control algorithm. We illustrate our method using simulated examples of both serial and parallel structures and we validate it on a real 3D soft robot made of silicone.

Abstract: In this paper we address the problem of robust and efficient averaging of relative 3D rotations. Apart from having an interesting geometric structure, robust rotation averaging addresses the need for a good initialization for large scale optimization used in structure-from-motion pipelines. Such pipelines often use unstructured image datasets harvested from the internet thereby requiring an initialization method that is robust to outliers. Our approach works on the Lie group structure of 3D rotations and solves the problem of large-scale robust rotation averaging in two ways. Firstly, we use modern l1 optimizers to carry out robust averaging of relative rotations that is efficient, scalable and robust to outliers. In addition, we also develop a two step method that uses the l1 solution as an initialisation for an iteratively reweighted least squares (IRLS) approach. These methods achieve excellent results on large-scale, real world datasets and significantly outperform existing methods, i.e. the state-of-the-art discrete-continuous optimization method of [3] as well as the Weiszfeld method of [8]. We demonstrate the efficacy of our method on two large scale real world datasets and also provide the results of the two aforementioned methods for comparison.

Abstract: In this paper, a new iterative adaptive dynamic programming (ADP) algorithm is developed to solve optimal control problems for infinite-horizon discrete-time nonlinear systems with finite approximation errors. The idea is to use an iterative ADP algorithm to obtain the iterative control law that makes the iterative performance index function reach the optimum. When the iterative control law and the iterative performance index function in each iteration cannot be accurately obtained, the convergence conditions of the iterative ADP algorithm are obtained. When convergence conditions are satisfied, it is shown that the iterative performance index functions can converge to a finite neighborhood of the greatest lower bound of all performance index functions under some mild assumptions. Neural networks are used to approximate the performance index function and compute the optimal control policy, respectively, for facilitating the implementation of the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

Abstract: We consider data transmissions in a full duplex (FD) multiuser multiple-input multiple-output (MU-MIMO) system, where a base station (BS) bidirectionally communicates with multiple users in the downlink (DL) and uplink (UL) channels on the same system resources. The system model of consideration has been thought to be impractical due to the self-interference (SI) between transmit and receive antennas at the BS. Interestingly, recent advanced techniques in hardware design have demonstrated that the SI can be suppressed to a degree that possibly allows for FD transmission. This paper goes one step further in exploring the potential gains in terms of the spectral efficiency (SE) and energy efficiency (EE) that can be brought by the FD MU-MIMO model. Toward this end, we propose low-complexity designs for maximizing the SE and EE, and evaluate their performance numerically. For the SE maximization problem, we present an iterative design that obtains a locally optimal solution based on a sequential convex approximation method. In this way, the nonconvex precoder design problem is approximated by a convex program at each iteration. Then, we propose a numerical algorithm to solve the resulting convex program based on the alternating and dual decomposition approaches, where analytical expressions for precoders are derived. For the EE maximization problem, using the same method, we first transform it into a concave-convex fractional program, which then can be reformulated as a convex program using the parametric approach. We will show that the resulting problem can be solved similarly to the SE maximization problem. Numerical results demonstrate that, compared to a half duplex system, the FD system of interest with the proposed designs achieves a better SE and a slightly smaller EE when the SI is small.

Abstract: Recently the sparse representation based classification (SRC) has been proposed for robust face recognition (FR). In SRC, the testing image is coded as a sparse linear combination of the training samples, and the representation fidelity is measured by the l2-norm or l1 -norm of the coding residual. Such a sparse coding model assumes that the coding residual follows Gaussian or Laplacian distribution, which may not be effective enough to describe the coding residual in practical FR systems. Meanwhile, the sparsity constraint on the coding coefficients makes the computational cost of SRC very high. In this paper, we propose a new face coding model, namely regularized robust coding (RRC), which could robustly regress a given signal with regularized regression coefficients. By assuming that the coding residual and the coding coefficient are respectively independent and identically distributed, the RRC seeks for a maximum a posterior solution of the coding problem. An iteratively reweighted regularized robust coding (IR3C) algorithm is proposed to solve the RRC model efficiently. Extensive experiments on representative face databases demonstrate that the RRC is much more effective and efficient than state-of-the-art sparse representation based methods in dealing with face occlusion, corruption, lighting, and expression changes, etc.

Abstract: This study derives a least-squares-based iterative algorithm and a gradient-based iterative algorithm for Hammerstein systems using the decomposition-based hierarchical identification principle. The simulation results confirm that the proposed two algorithms can give satisfactory identification accuracies and the least-squares-based iterative algorithm has faster convergence rates than the gradient-based iterative algorithm.

Abstract: In reversible data hiding techniques, the values of host data are modified according to some particular rules and the original host content can be perfectly restored after extraction of the hidden data on receiver side. In this paper, the optimal rule of value modification under a payload-distortion criterion is found by using an iterative procedure, and a practical reversible data hiding scheme is proposed. The secret data, as well as the auxiliary information used for content recovery, are carried by the differences between the original pixel-values and the corresponding values estimated from the neighbors. Here, the estimation errors are modified according to the optimal value transfer rule. Also, the host image is divided into a number of pixel subsets and the auxiliary information of a subset is always embedded into the estimation errors in the next subset. A receiver can successfully extract the embedded secret data and recover the original content in the subsets with an inverse order. This way, a good reversible data hiding performance is achieved.

Abstract: This paper proposes a two-stage robust optimization approach to solve the N- k contingency-constrained unit commitment (CCUC) problem. In our approach, both generator and transmission line contingencies are considered. Compared to the traditional approach using a given set of components as candidates for possible failures, our approach considers all possible component failure scenarios. We consider the objectives of minimizing the total generation cost under the worst-case contingency scenario and/or the total pre-contingency cost. We formulate CCUC as a two-stage robust optimization problem and develop a decomposition framework to enable tractable computation. In our framework, the master problem makes unit commitment decisions and the subproblem discovers the worst-case contingency scenarios. By using linearization techniques and duality theory, we transform the subproblem into a mixed-integer linear program (MILP). The most violated inequalities generated from the subproblem are fed back into the master problem during each iteration. Our approach guarantees a globally optimal solution in a finite number of iterations. In reported computational experiments, we test both primal and dual decomposition approaches. Our computational results verify the effectiveness of our proposed approach.

Abstract: In speaker diarization, standard approaches typically perform speaker clustering on some initial segmentation before refining the segment boundaries in a re-segmentation step to obtain a final diarization hypothesis. In this paper, we integrate an improved clustering method with an existing re-segmentation algorithm and, in iterative fashion, optimize both speaker cluster assignments and segmentation boundaries jointly. For clustering, we extend our previous research using factor analysis for speaker modeling. In continuing to take advantage of the effectiveness of factor analysis as a front-end for extracting speaker-specific features (i.e., i-vectors), we develop a probabilistic approach to speaker clustering by applying a Bayesian Gaussian Mixture Model (GMM) to principal component analysis (PCA)-processed i-vectors. We then utilize information at different temporal resolutions to arrive at an iterative optimization scheme that, in alternating between clustering and re-segmentation steps, demonstrates the ability to improve both speaker cluster assignments and segmentation boundaries in an unsupervised manner. Our proposed methods attain results that are comparable to those of a state-of-the-art benchmark set on the multi-speaker CallHome telephone corpus. We further compare our system with a Bayesian nonparametric approach to diarization and attempt to reconcile their differences in both methodology and performance.

Abstract: Nonconvex constraints are valuable regularizers in many optimization problems. In particular, sparsity constraints have had a significant impact on sampling theory, where they are used in compressed sensing and allow structured signals to be sampled far below the rate traditionally prescribed. Nearly, all of the theory developed for compressed sensing signal recovery assumes that samples are taken using linear measurements. In this paper, we instead address the compressed sensing recovery problem in a setting where the observations are nonlinear. We show that, under conditions similar to those required in the linear setting, the iterative hard thresholding algorithm can be used to accurately recover sparse or structured signals from few nonlinear observations. Similar ideas can also be developed in a more general nonlinear optimization framework. In the second part of this paper, we therefore present related result that shows how this can be done under sparsity and union of subspaces constraints, whenever a generalization of the restricted isometry property traditionally imposed on the compressed sensing system holds.

Abstract: In this paper we study energy efficient joint power allocation and beamforming for coordinated multicell multiuser downlink systems. The considered optimization problem is in a non-convex fractional form and hard to tackle. We propose to first transform the original problem into an equivalent optimization problem in a parametric subtractive form, by which we reach its solution through a two-layer optimization scheme. The outer layer only involves one-dimension search for the energy efficiency parameter which can be addressed using the bi-section search, the key issue lies in the inner layer where a non-fractional sub-problem needs to tackle. By exploiting the relationship between the user rate and the mean square error, we then develop an iterative algorithm to solve it. The convergence of this algorithm is proved and the solution is further derived in closed-form. Our analysis also shows that the proposed algorithm can be implemented in parallel with reasonable complexity. Numerical results illustrate that our algorithm has a fast convergence and achieves near-optimal energy efficiency. It is also observed that at the low transmit power region, our solution almost achieves the optimal sum rate and the optimal energy efficiency simultaneously; while at the middle-high transmit power region, a certain sum rate loss is suffered in order to guarantee the energy efficiency.

Abstract: In this paper, the author presents a new tool, called The Convergence Plane, that allows to study the real dynamics of iterative methods whose iterations depends on one parameter in an easy and compact way. This tool can be used, inter alia, to find the elements of a family that have good convergence properties and discard the bad ones or to see how the basins of attraction changes along the elements of the family. To show the applicability of the tool an example of the dynamics of the Damped Newton's method applied to a cubic polynomial is presented.

Abstract: We introduce a new iterative process which can be seen as a hybrid of Picard and Mann iterative processes. We show that the new process converges faster than all of Picard, Mann and Ishikawa iterative processes in the sense of Berinde (Iterative Approximation of Fixed Points, 2002) for contractions. We support our analytical proof by a numerical example. We prove a strong convergence theorem with the help of our process for the class of nonexpansive mappings in general Banach spaces and apply it to get a result in uniformly convex Banach spaces. Our weak convergence results are proved when the underlying space satisfies Opial’s condition or has Frechet differentiable norm or its dual satisfies the Kadec-Klee property.

Abstract: This brief presents a data-driven constrained norm-optimal iterative learning control framework for linear time-invariant systems that applies to both tracking and point-to-point motion problems. The key contribution of this brief is the estimation of the system's impulse response using input/output measurements from previous iterations, hereby eliminating time-consuming identification experiments. The estimated impulse response is used in a norm-optimal iterative learning controller, where actuator limitations can be formulated as linear inequality constraints. Experimental validation on a linear motor positioning system shows the ability of the proposed data-driven framework to: 1) achieve tracking accuracy up to the repeatability of the test setup; 2) minimize the rms value of the tracking error while respecting the actuator input constraints; 3) learn energy-optimal system inputs for point-to-point motions.

Abstract: Iterative learning control (ILC) is concerned with tracking a reference trajectory defined over a finite time duration, and is applied to systems which perform this action repeatedly. However, in many application domains the output is not critical at all points over the task duration. In this paper the facility to track an arbitrary subset of points is therefore introduced, and the additional flexibility this brings is used to address other control objectives in the framework of iterative learning. These comprise hard and soft constraints involving the system input, output and states. Experimental results using a robotic arm confirm that embedding constraints in the ILC framework leads to superior performance than can be obtained using standard ILC and an a priori specified reference.

Abstract: Spectral Matching (SM) is a computationally efficient approach to approximate the solution of pairwise matching problems that are np-hard. In this paper, we present a probabilistic interpretation of spectral matching schemes and derive a novel Probabilistic Matching (PM) scheme that is shown to outperform previous approaches. We show that spectral matching can be interpreted as a Maximum Likelihood (ML) estimate of the assignment probabilities and that the Graduated Assignment (GA) algorithm can be cast as a Maximum a Posteriori (MAP) estimator. Based on this analysis, we derive a ranking scheme for spectral matchings based on their reliability, and propose a novel iterative probabilistic matching algorithm that relaxes some of the implicit assumptions used in prior works. We experimentally show our approaches to outperform previous schemes when applied to exhaustive synthetic tests as well as the analysis of real image sequences.

Abstract: This article deals with learning dictionaries for sparse approximation whose atoms are both adapted to a training set of signals and mutually incoherent. To meet this objective, we employ a dictionary learning scheme consisting of sparse approximation followed by dictionary update and we add to the latter a decorrelation step in order to reach a target mutual coherence level. This step is accomplished by an iterative projection method complemented by a rotation of the dictionary. Experiments on musical audio data and a comparison with the method of optimal coherence-constrained directions (MOCOD) and the incoherent K-SVD (INK-SVD) illustrate that the proposed algorithm can learn dictionaries that exhibit a low mutual coherence while providing a sparse approximation with better signal-to-noise ratio (SNR) than the benchmark techniques.

Abstract: This paper presents the design of a controller for an autonomous ground vehicle. The goal is to track the lane centerline while avoiding collisions with obstacles. A nonlinear model predictive control (MPC) framework is used where the control inputs are the front steering angle and the braking torques at the four wheels. The focus of this work is on the development of a tailored algorithm for solving the nonlinear MPC problem. Hardware-in-the-loop simulations with the proposed algorithm show a reduction in the computational time as compared to general purpose nonlinear solvers. Experimental tests on a passenger vehicle at high speeds on low friction road surfaces show the effectiveness of the proposed algorithm.