Abstract: The special importance of L1/2 regularization has been recognized in recent studies on sparse modeling (particularly on compressed sensing). The L1/2 regularization, however, leads to a nonconvex, nonsmooth, and non-Lipschitz optimization problem that is difficult to solve fast and efficiently. In this paper, through developing a threshoding representation theory for L1/2 regularization, we propose an iterative half thresholding algorithm for fast solution of L1/2 regularization, corresponding to the well-known iterative soft thresholding algorithm for L1 regularization, and the iterative hard thresholding algorithm for L0 regularization. We prove the existence of the resolvent of gradient of ||x||1/21/2, calculate its analytic expression, and establish an alternative feature theorem on solutions of L1/2 regularization, based on which a thresholding representation of solutions of L1/2 regularization is derived and an optimal regularization parameter setting rule is formulated. The developed theory provides a successful practice of extension of the well- known Moreau's proximity forward-backward splitting theory to the L1/2 regularization case. We verify the convergence of the iterative half thresholding algorithm and provide a series of experiments to assess performance of the algorithm. The experiments show that the half algorithm is effective, efficient, and can be accepted as a fast solver for L1/2 regularization. With the new algorithm, we conduct a phase diagram study to further demonstrate the superiority of L1/2 regularization over L1 regularization.

Abstract: Nonnegative matrix factorization (NMF) is a powerful matrix decomposition technique that approximates a nonnegative matrix by the product of two low-rank nonnegative matrix factors. It has been widely applied to signal processing, computer vision, and data mining. Traditional NMF solvers include the multiplicative update rule (MUR), the projected gradient method (PG), the projected nonnegative least squares (PNLS), and the active set method (AS). However, they suffer from one or some of the following three problems: slow convergence rate, numerical instability and nonconvergence. In this paper, we present a new efficient NeNMF solver to simultaneously overcome the aforementioned problems. It applies Nesterov's optimal gradient method to alternatively optimize one factor with another fixed. In particular, at each iteration round, the matrix factor is updated by using the PG method performed on a smartly chosen search point, where the step size is determined by the Lipschitz constant. Since NeNMF does not use the time consuming line search and converges optimally at rate in optimizing each matrix factor, it is superior to MUR and PG in terms of efficiency as well as approximation accuracy. Compared to PNLS and AS that suffer from numerical instability problem in the worst case, NeNMF overcomes this deficiency. In addition, NeNMF can be used to solve -norm, -norm and manifold regularized NMF with the optimal convergence rate. Numerical experiments on both synthetic and real-world datasets show the efficiency of NeNMF for NMF and its variants comparing to representative NMF solvers. Extensive experiments on document clustering suggest the effectiveness of NeNMF.

Abstract: We propose a noniterative solution for the Perspective-n-Point ({\rm P}n{\rm P}) problem, which can robustly retrieve the optimum by solving a seventh order polynomial. The central idea consists of three steps: 1) to divide the reference points into 3-point subsets in order to achieve a series of fourth order polynomials, 2) to compute the sum of the square of the polynomials so as to form a cost function, and 3) to find the roots of the derivative of the cost function in order to determine the optimum. The advantages of the proposed method are as follows: First, it can stably deal with the planar case, ordinary 3D case, and quasi-singular case, and it is as accurate as the state-of-the-art iterative algorithms with much less computational time. Second, it is the first noniterative {\rm P}n{\rm P} solution that can achieve more accurate results than the iterative algorithms when no redundant reference points can be used (n\le 5). Third, large-size point sets can be handled efficiently because its computational complexity is O(n).

Abstract: 1 Introduction- 2 Algorithmic Operators- 3 Convergence of Iterative Methods- 4 Algorithmic Projection Operators- 5 Projection methods

Abstract: Power allocations in an interference-limited wireless network for global maximization of the weighted sum throughput or global optimization of the minimum weighted rate among network links are not only important but also very hard optimization problems due to their nonconvexity nature. Recently developed methods are either unable to locate the global optimal solutions or prohibitively complex for practical applications. This paper exploits the d.c. (difference of two convex functions/sets) structure of either the objective function or constraints of these global optimization problems to develop efficient iterative algorithms with very low complexity. Numerical results demonstrate that the developed algorithms are able to locate the global optimal solutions by only a few iterations and they are superior to the previously-proposed methods in both performance and computation complexity.

Abstract: In contrast to the positive definite Helmholtz equation, the deceivingly similar looking indefinite Helmholtz equation is difficult to solve using classical iterative methods. Simply using a Krylov method is much less effective, especially when the wave number in the Helmholtz operator becomes large, and also algebraic preconditioners such as incomplete LU factorizations do not remedy the situation. Even more powerful preconditioners such as classical domain decomposition and multigrid methods fail to lead to a convergent method, and often behave differently from their usual behavior for positive definite problems. For example increasing the overlap in a classical Schwarz method degrades its performance, as does increasing the number of smoothing steps in multigrid. The purpose of this review paper is to explain why classical iterative methods fail to be effective for Helmholtz problems, and to show different avenues that have been taken to address this difficulty.

Abstract: Multivariate transfer entropy (TE) is a model-free approach to detect causalities in multivariate time series. It is able to distinguish direct from indirect causality and common drivers without assuming any underlying model. But despite these advantages it has mostly been applied in a bivariate setting as it is hard to estimate reliably in high dimensions since its definition involves infinite vectors. To overcome this limitation, we propose to embed TE into the framework of graphical models and present a formula that decomposes TE into a sum of finite-dimensional contributions that we call decomposed transfer entropy. Graphical models further provide a richer picture because they also yield the causal coupling delays. To estimate the graphical model we suggest an iterative algorithm, a modified version of the PC-algorithm with a very low estimation dimension. We present an appropriate significance test and demonstrate the method's performance using examples of nonlinear stochastic delay-differential equations and observational climate data (sea level pressure).

Abstract: Multibeam satellite systems have been employed to provide interactive broadband services to geographical areas under-served by terrestrial infrastructure. In this context, this paper studies joint multiuser linear precoding design in the forward link of fixed multibeam satellite systems. We provide a generic optimization framework for linear precoding design to handle any objective functions of data rate with general linear and nonlinear power constraints. To achieve this, an iterative algorithm which optimizes the precoding vectors and power allocation alternatingly is proposed and most importantly, the proposed algorithm is proved to always converge. The proposed optimization algorithm is also applicable to nonlinear dirty paper coding. As a special case, a more efficient algorithm is devised to find the optimal solution to the problem of maximizing the proportional fairness among served users. In addition, the aforementioned problems and algorithms are extended to the case that each terminal has multiple co-polarization or dual-polarization antennas. Simulation results demonstrate substantial performance improvement of the proposed schemes over conventional multibeam satellite systems, zero-forcing and regularized zero-forcing precoding schemes in terms of meeting the traffic demand, e.g., using real beam patterns, over twice higher throughput can be achieved compared with the conventional scheme. The performance of the proposed linear precoding scheme is also shown to be very close to the dirty paper coding.

Abstract: We propose a prototypical Split Inverse Problem (SIP) and a new variational problem, called the Split Variational Inequality Problem (SVIP), which is a SIP. It entails finding a solution of one inverse problem (e.g., a Variational Inequality Problem (VIP)), the image of which under a given bounded linear transformation is a solution of another inverse problem such as a VIP. We construct iterative algorithms that solve such problems, under reasonable conditions, in Hilbert space and then discuss special cases, some of which are new even in Euclidean space.

Abstract: The split feasibility problem (SFP) consists in finding a point in a given closed convex subset of a Hilbert space such that its image under a bounded linear operator belongs to a given closed convex subset of another Hilbert space. Iterative methods can be employed to solve the SFP. The most popular iterative method is Byrne’s CQ algorithm. However, to employ Byrne’s CQ algorithm, one needs to know a priori the norm (or at least an estimate of the norm) of the bounded linear operator (matrix in the finite-dimensional framework). It is the purpose of this paper to introduce a way of selecting the stepsizes such that the implementation of the CQ algorithm does not need any prior information about the operator norm. We also practise this way of selecting stepsizes for variants of the CQ algorithm, including a relaxed CQ algorithm where the two closed convex sets are both level sets of convex functions, and a Halpern-type algorithm. Both weak and strong convergence are investigated. Numerical experiments are included to illustrate the applications in signal processing of the CQ algorithm with stepsizes selected in an adaptive way.

Abstract: We present l1 -SPIRiT, a simple algorithm for auto calibrating parallel imaging (acPI) and compressed sensing (CS) that permits an efficient implementation with clinically-feasible runtimes. We propose a CS objective function that minimizes cross-channel joint sparsity in the wavelet domain. Our reconstruction minimizes this objective via iterative soft-thresholding, and integrates naturally with iterative self-consistent parallel imaging (SPIRiT). Like many iterative magnetic resonance imaging reconstructions, l1-SPIRiT's image quality comes at a high computational cost. Excessively long runtimes are a barrier to the clinical use of any reconstruction approach, and thus we discuss our approach to efficiently parallelizing l1 -SPIRiT and to achieving clinically-feasible runtimes. We present parallelizations of l1 -SPIRiT for both multi-GPU systems and multi-core CPUs, and discuss the software optimization and parallelization decisions made in our implementation. The performance of these alternatives depends on the processor architecture, the size of the image matrix, and the number of parallel imaging channels. Fundamentally, achieving fast runtime requires the correct trade-off between cache usage and parallelization overheads. We demonstrate image quality via a case from our clinical experimentation, using a custom 3DFT spoiled gradient echo (SPGR) sequence with up to 8× acceleration via Poisson-disc undersampling in the two phase-encoded directions.

Abstract: 1. Introduction 2. Krylov subspace methods 3. Matching moments and model reduction view 4. Short recurrences for generating orthogonal Krylov subspace bases 5. Cost of computations using Krylov subspace methods

Abstract: The Holomorphic Embedding Load Flow is a novel general-purpose method for solving the steady state equations of power systems. Based on the techniques of Complex Analysis, it has been granted two US Patents. Experience has proven it is performant and competitive with respect established iterative methods, but its main practical features are that it is non-iterative and deterministic, yielding the correct solution when it exists and, conversely, unequivocally signaling voltage collapse when it does not. This paper reviews the embedded load flow method and highlights the technological breakthroughs that it enables: reliable real-time applications based on unsupervised exploratory load flows, such as Contingency Analysis, OPF, Limit-Violations solvers, and Restoration plan builders. We also report on the experience with the method in the implementation of several real-time EMS products now operating at large utilities.

Abstract: An iterative procedure for the synthesis of sparse arrays radiating focused or shaped beampattern is presented The algorithm consists in solving a sequence of weighted l1 convex optimization problems The method can thus be readily implemented and efficiently solved In the optimization procedure, the objective is the minimization of the number of radiating elements and the constraints correspond to the pattern requirements The method can be applied to synthesize either focused or shaped beampattern and there is no restriction regarding the array geometry and individual element patterns Numerical comparisons with standard benchmark problems assess the efficiency of the proposed approach, whose computation time is several orders of magnitude below those of so-called global optimization algorithms

Abstract: In this paper, a neuro-optimal control scheme for a class of unknown discrete-time nonlinear systems with discount factor in the cost function is developed. The iterative adaptive dynamic programming algorithm using globalized dual heuristic programming technique is introduced to obtain the optimal controller with convergence analysis in terms of cost function and control law. In order to carry out the iterative algorithm, a neural network is constructed first to identify the unknown controlled system. Then, based on the learned system model, two other neural networks are employed as parametric structures to facilitate the implementation of the iterative algorithm, which aims at approximating at each iteration the cost function and its derivatives and the control law, respectively. Finally, a simulation example is provided to verify the effectiveness of the proposed optimal control approach. Note to Practitioners-The increasing complexity of the real-world industry processes inevitably leads to the occurrence of nonlinearity and high dimensions, and their mathematical models are often difficult to build. How to design the optimal controller for nonlinear systems without the requirement of knowing the explicit model has become one of the main foci of control practitioners. However, this problem cannot be handled by only relying on the traditional dynamic programming technique because of the "curse of dimensionality". To make things worse, the backward direction of solving process of dynamic programming precludes its wide application in practice. Therefore, in this paper, the iterative adaptive dynamic programming algorithm is proposed to deal with the optimal control problem for a class of unknown nonlinear systems forward-in-time. Moreover, the detailed implementation of the iterative ADP algorithm through the globalized dual heuristic programming technique is also presented by using neural networks. Finally, the effectiveness of the control strategy is illustrated via simulation study.

Abstract: We develop an algorithm for computing the solution of a large system of linear ordinary differential equations (ODEs) with polynomial inhomogeneity. This is equivalent to computing the action of a certain matrix function on the vector representing the initial condition. The matrix function is a linear combination of the matrix exponential and other functions related to the exponential (the so-called ϕ-functions). Such computations are the major computational burden in the implementation of exponential integrators, which can solve general ODEs. Our approach is to compute the action of the matrix function by constructing a Krylov subspace using Arnoldi or Lanczos iteration and projecting the function on this subspace. This is combined with time-stepping to prevent the Krylov subspace from growing too large. The algorithm is fully adaptive: it varies both the size of the time steps and the dimension of the Krylov subspace to reach the required accuracy. We implement this algorithm in the matlab function phipm and we give instructions on how to obtain and use this function. Various numerical experiments show that the phipm function is often significantly more efficient than the state-of-the-art.

Abstract: Regularized iterative reconstruction algorithms for imaging inverse problems require selection of appropriate regularization parameter values. We focus on the challenging problem of tuning regularization parameters for nonlinear algorithms for the case of additive (possibly complex) Gaussian noise. Generalized cross-validation (GCV) and (weighted) mean-squared error (MSE) approaches [based on Stein's unbiased risk estimate (SURE)] need the Jacobian matrix of the nonlinear reconstruction operator (representative of the iterative algorithm) with respect to the data. We derive the desired Jacobian matrix for two types of nonlinear iterative algorithms: a fast variant of the standard iterative reweighted least-squares method and the contemporary split-Bregman algorithm, both of which can accommodate a wide variety of analysis- and synthesis-type regularizers. The proposed approach iteratively computes two weighted SURE-type measures: predicted-SURE and projected-SURE (which require knowledge of noise variance σ2), and GCV (which does not need σ2) for these algorithms. We apply the methods to image restoration and to magnetic resonance image (MRI) reconstruction using total variation and an analysis-type l1-regularization. We demonstrate through simulations and experiments with real data that minimizing predicted-SURE and projected-SURE consistently lead to near-MSE-optimal reconstructions. We also observe that minimizing GCV yields reconstruction results that are near-MSE-optimal for image restoration and slightly suboptimal for MRI. Theoretical derivations in this paper related to Jacobian matrix evaluations can be extended, in principle, to other types of regularizers and reconstruction algorithms.

Abstract: Image formation algorithms in a variety of applications have explicit or implicit dependence on a mathematical model of the observation process. Inaccuracies in the observation model may cause various degradations and artifacts in the reconstructed images. The application of interest in this paper is synthetic aperture radar (SAR) imaging, which particularly suffers from motion-induced model errors. These types of errors result in phase errors in SAR data, which cause defocusing of the reconstructed images. Particularly focusing on imaging of fields that admit a sparse representation, we propose a sparsity-driven method for joint SAR imaging and phase error correction. Phase error correction is performed during the image formation process. The problem is set up as an optimization problem in a nonquadratic regularization-based framework. The method involves an iterative algorithm, where each iteration of which consists of consecutive steps of image formation and model error correction. Experimental results show the effectiveness of the approach for various types of phase errors, as well as the improvements that it provides over existing techniques for model error compensation in SAR.

Abstract: Iterative image reconstruction algorithms for optoacoustic tomography (OAT), also known as photoacoustic tomography, have the ability to improve image quality over analytic algorithms due to their ability to incorporate accurate models of the imaging physics, instrument response and measurement noise. However, to date, there have been few reported attempts to employ advanced iterative image reconstruction algorithms for improving image quality in three-dimensional (3D) OAT. In this work, we implement and investigate two iterative image reconstruction methods for use with a 3D OAT small animal imager: namely a penalized least-squares (PLS) method employing a quadratic smoothness penalty and a PLS method employing a total variation norm penalty. The reconstruction algorithms employ accurate models of the ultrasonic transducer impulse responses. Experimental data sets are employed to compare the performances of the iterative reconstruction algorithms to that of a 3D filtered backprojection (FBP) algorithm. By the use of quantitative measures of image quality, we demonstrate that the iterative reconstruction algorithms can mitigate image artifacts and preserve spatial resolution more effectively than FBP algorithms. These features suggest that the use of advanced image reconstruction algorithms can improve the effectiveness of 3D OAT while reducing the amount of data required for biomedical applications.

Abstract: We consider an incompressible flow problem in a N -dimensional fractured porous domain (Darcy’s problem). The fracture is represented by a (N − 1)-dimensional interface, exchanging fluid with the surrounding media. In this paper we consider the lowest-order (ℝ T0 , ℙ0 ) Raviart-Thomas mixed finite element method for the approximation of the coupled Darcy’s flows in the porous media and within the fracture, with independent meshes for the respective domains. This is achieved thanks to an enrichment with discontinuous basis functions on triangles crossed by the fracture and a weak imposition of interface conditions. First, we study the stability and convergence properties of the resulting numerical scheme in the uncoupled case, when the known solution of the fracture problem provides an immersed boundary condition. We detail the implementation issues and discuss the algebraic properties of the associated linear system. Next, we focus on the coupled problem and propose an iterative porous domain/fracture domain iterative method to solve for fluid flow in both the porous media and the fracture and compare the results with those of a traditional monolithic approach. Numerical results are provided confirming convergence rates and algebraic properties predicted by the theory. In particular, we discuss preconditioning and equilibration techniques to make the condition number of the discrete problem independent of the position of the immersed interface. Finally, two and three dimensional simulations of Darcy’s flow in different configurations (highly and poorly permeable fracture) are analyzed and discussed.

Abstract: In this paper, resource allocation for energy efficient communication in space division multiple access (SDMA) downlink networks with large numbers of transmit antennas is studied. The considered problem is modeled as a non-convex optimization problem which takes into account the circuit power consumption and a minimum required data rate. By exploiting the properties of fractional programming, the considered non-convex optimization problem in fractional form is transformed into an equivalent optimization problem in subtractive form, which enables the derivation of an efficient iterative resource allocation algorithm. The optimal power allocation solution for each iteration is derived based on a low complexity user selection policy for maximization of the energy efficiency of data transmission (bit/Joule delivered to the users). Simulation results illustrate that the proposed iterative resource allocation algorithm converges in a small number of iterations and unveil the trade-off between energy efficiency and the number of antennas.

Abstract: Highlights? An application of Gradient Boosting (GB) to predict insurance losses is outlined. ? GB shows higher predictive accuracy relative to the conventional GLM method. ? GB provides a suitable framework to build insurance pricing models. ? GB provides interpretable results, a desirable feature in a business environment. Gradient Boosting (GB) is an iterative algorithm that combines simple parameterized functions with "poor" performance (high prediction error) to produce a highly accurate prediction rule. In contrast to other statistical learning methods usually providing comparable accuracy (e.g., neural networks and support vector machines), GB gives interpretable results, while requiring little data preprocessing and tuning of the parameters. The method is highly robust to less than clean data and can be applied to classification or regression problems from a variety of response distributions (Gaussian, Bernoulli, Poisson, and Laplace). Complex interactions are modeled simply, missing values in the predictors are managed almost without loss of information, and feature selection is performed as an integral part of the procedure. These properties make GB a good candidate for insurance loss cost modeling. However, to the best of our knowledge, the application of this method to insurance pricing has not been fully documented to date. This paper presents the theory of GB and its application to the problem of predicting auto "at-fault" accident loss cost using data from a major Canadian insurer. The predictive accuracy of the model is compared against the conventional Generalized Linear Model (GLM) approach.

Abstract: In this paper, we propose an iterative interference alignment (IA) algorithm for MIMO cellular networks with partial connectivity, which is induced by heterogeneous path losses and spatial correlation. Such systems impose several key technical challenges in the IA algorithm design, namely the overlapping between the direct and interfering links due to the MIMO cellular topology as well as how to exploit the partial connectivity. We shall address these challenges and propose a three stage IA algorithm. As illustration, we analyze the achievable degree of freedom (DoF) of the proposed algorithm for a symmetric partially connected MIMO cellular network. We show that there is significant DoF gain compared with conventional IA algorithms due to partial connectivity. The derived DoF bound is also backward compatible with that achieved on fully connected K-pair MIMO interference channels.

Abstract: We present a randomized iterative algorithm that exponentially converges in expectation to the minimum Euclidean norm least squares solution of a given linear system of equations. The expected number of arithmetic operations required to obtain an estimate of given accuracy is proportional to the square condition number of the system multiplied by the number of non-zeros entries of the input matrix. The proposed algorithm is an extension of the randomized Kaczmarz method that was analyzed by Strohmer and Vershynin.

Abstract: Probing signal waveforms play a central role in the signal processing performance of a multiple-input multiple-output (MIMO) radar. In practice, for a given desired beam pattern, we need to design a probing signal waveform whose beam pattern closely matches the desired one and whose autocorrelation and cross-correlation sidelobes are kept low. The latter properties are important to mitigate undesirable interference caused by multiple targets or scatterers. In this correspondence, we present an efficient optimization method to design a constant modulus probing signal which can synthesize a desired beam pattern while maximally suppressing both the autocorrelation and cross-correlation sidelobes at/between given spacial angles. We formulate this problem as an unconstrained minimization of a fourth order trigonometric polynomial and propose an efficient quasi-Newton iterative algorithm to solve it. Besides, we provide an analysis of the local minima of the fourth-order trigonometric polynomial and prove that any local minima is a 1/2-approximation of its global optimal solution. Numerical examples show that the proposed approach compares favorably with the existing approach.

Abstract: This paper integrates recent work on Path Integral (PI) and Kullback Leibler (KL) divergence stochastic optimal control theory with earlier work on risk sensitivity and the fundamental dualities between free energy and relative entropy. We derive the path integral optimal control framework and its iterative version based on the aforemetioned dualities. The resulting formulation of iterative path integral control is valid for general feedback policies and in contrast to previous work, it does not rely on pre-specified policy parameterizations. The derivation is based on successive applications of Girsanov's theorem and the use of Radon-Nikodým derivative as applied to diffusion processes due to the change of measure in the stochastic dynamics. We compare the PI control derived based on Dynamic Programming with PI based on the duality between free energy and relative entropy. Moreover we extend our analysis on the applicability of the relationship between free energy and relative entropy to optimal control of markov jump diffusions processes. Furthermore, we present the links between KL stochastic optimal control and the aforementioned dualities and discuss its generalizability.