Abstract: Iterative shrinkage/thresholding (1ST) algorithms have been recently proposed to handle a class of convex unconstrained optimization problems arising in image restoration and other linear inverse problems. This class of problems results from combining a linear observation model with a nonquadratic regularizer (e.g., total variation or wavelet-based regularization). It happens that the convergence rate of these 1ST algorithms depends heavily on the linear observation operator, becoming very slow when this operator is ill-conditioned or ill-posed. In this paper, we introduce two-step 1ST (TwIST) algorithms, exhibiting much faster convergence rate than 1ST for ill-conditioned problems. For a vast class of nonquadratic convex regularizers (lscrP norms, some Besov norms, and total variation), we show that TwIST converges to a minimizer of the objective function, for a given range of values of its parameters. For noninvertible observation operators, we introduce a monotonic version of TwIST (MTwIST); although the convergence proof does not apply to this scenario, we give experimental evidence that MTwIST exhibits similar speed gains over IST. The effectiveness of the new methods are experimentally confirmed on problems of image deconvolution and of restoration with missing samples.

Abstract: The algorithm for traditional absolute orientation is an iterative algorithm which needs relatively accurate iterative initial valueBased on quaternion,a close-form solution is put forward in the paper which is used to calculate absolute orientation parameters without iteration method by way of strict theoretical deductionThe principal theories are as followsFirstly,the unit quaternion is used to describe the rotational conversion relation of the coordinatesThen the problem about absolute orientation is transformed into a problem about optimization to solveFinally,a simulation test using analogue data is carried out to validate the correctness and reliability of the algorithm

Abstract: In this paper, the iterative learning control (ILC) literature published between 1998 and 2004 is categorized and discussed, extending the earlier reviews presented by two of the authors. The papers includes a general introduction to ILC and a technical description of the methodology. The selected results are reviewed, and the ILC literature is categorized into subcategories within the broader division of application-focused and theory-focused results.

Abstract: An unsupervised learning algorithm for the separation of sound sources in one-channel music signals is presented. The algorithm is based on factorizing the magnitude spectrogram of an input signal into a sum of components, each of which has a fixed magnitude spectrum and a time-varying gain. Each sound source, in turn, is modeled as a sum of one or more components. The parameters of the components are estimated by minimizing the reconstruction error between the input spectrogram and the model, while restricting the component spectrograms to be nonnegative and favoring components whose gains are slowly varying and sparse. Temporal continuity is favored by using a cost term which is the sum of squared differences between the gains in adjacent frames, and sparseness is favored by penalizing nonzero gains. The proposed iterative estimation algorithm is initialized with random values, and the gains and the spectra are then alternatively updated using multiplicative update rules until the values converge. Simulation experiments were carried out using generated mixtures of pitched musical instrument samples and drum sounds. The performance of the proposed method was compared with independent subspace analysis and basic nonnegative matrix factorization, which are based on the same linear model. According to these simulations, the proposed method enables a better separation quality than the previous algorithms. Especially, the temporal continuity criterion improved the detection of pitched musical sounds. The sparseness criterion did not produce significant improvements

Abstract: We describe and analyze a simple and effective iterative algorithm for solving the optimization problem cast by Support Vector Machines (SVM). Our method alternates between stochastic gradient descent steps and projection steps. We prove that the number of iterations required to obtain a solution of accuracy e is O(1/e). In contrast, previous analyses of stochastic gradient descent methods require Ω (1/e2) iterations. As in previously devised SVM solvers, the number of iterations also scales linearly with 1/λ, where λ is the regularization parameter of SVM. For a linear kernel, the total run-time of our method is O (d/(λe)), where d is a bound on the number of non-zero features in each example. Since the run-time does not depend directly on the size of the training set, the resulting algorithm is especially suited for learning from large datasets. Our approach can seamlessly be adapted to employ non-linear kernels while working solely on the primal objective function. We demonstrate the efficiency and applicability of our approach by conducting experiments on large text classification problems, comparing our solver to existing state-of-the-art SVM solvers. For example, it takes less than 5 seconds for our solver to converge when solving a text classification problem from Reuters Corpus Volume 1 (RCV1) with 800,000 training examples.

Abstract: This communication describes version 4.0 of Regularization Tools, a Matlab package for analysis and solution of discrete ill-posed problems. The new version allows for under-determined problems, and it is expanded with several new iterative methods, as well as new test problems and new parameter-choice methods.

Abstract: This paper describes new extensions to the previously published multivariate alteration detection (MAD) method for change detection in bi-temporal, multi- and hypervariate data such as remote sensing imagery. Much like boosting methods often applied in data mining work, the iteratively reweighted (IR) MAD method in a series of iterations places increasing focus on "difficult" observations, here observations whose change status over time is uncertain. The MAD method is based on the established technique of canonical correlation analysis: for the multivariate data acquired at two points in time and covering the same geographical region, we calculate the canonical variates and subtract them from each other. These orthogonal differences contain maximum information on joint change in all variables (spectral bands). The change detected in this fashion is invariant to separate linear (affine) transformations in the originally measured variables at the two points in time, such as 1) changes in gain and offset in the measuring device used to acquire the data, 2) data normalization or calibration schemes that are linear (affine) in the gray values of the original variables, or 3) orthogonal or other affine transformations, such as principal component (PC) or maximum autocorrelation factor (MAF) transformations. The IR-MAD method first calculates ordinary canonical and original MAD variates. In the following iterations we apply different weights to the observations, large weights being assigned to observations that show little change, i.e., for which the sum of squared, standardized MAD variates is small, and small weights being assigned to observations for which the sum is large. Like the original MAD method, the iterative extension is invariant to linear (affine) transformations of the original variables. To stabilize solutions to the (IR-)MAD problem, some form of regularization may be needed. This is especially useful for work on hyperspectral data. This paper describes ordinary two-set canonical correlation analysis, the MAD transformation, the iterative extension, and three regularization schemes. A simple case with real Landsat Thematic Mapper (TM) data at one point in time and (partly) constructed data at the other point in time that demonstrates the superiority of the iterative scheme over the original MAD method is shown. Also, examples with SPOT High Resolution Visible data from an agricultural region in Kenya, and hyperspectral airborne HyMap data from a small rural area in southeastern Germany are given. The latter case demonstrates the need for regularization

Abstract: Iterative projection algorithms are successfully being used as a substitute of lenses to recombine, numerically rather than optically, light scattered by illuminated objects. Images obtained computationally allow aberration-free diffraction-limited imaging and the possibility of using radiation for which no lenses exist. The challenge of this imaging technique is transferred from the lenses to the algorithms. We evaluate these new computational "instruments" developed for the phase-retrieval problem, and discuss acceleration strategies.

Abstract: We propose a new iterative algorithm for obtaining optimal holograms targeted to the generation of arbitrary three dimensional structures of optical traps. The algorithm basic idea and performance are discussed in conjunction to other available algorithms. We show that all algorithms lead to a phase distribution maximizing a specific performance quantifier, expressed as a function of the trap intensities. In this scheme we go a step further by introducing a new quantifier and the associated algorithm leading to unprecedented efficiency and uniformity in trap light distributions. The algorithms performances are investigated both numerically and experimentally.

Abstract: The Kaczmarz method for solving linear systems of equations is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for consistent, overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system, but only a small random part of it. It thus outperforms all previously known methods on general extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations as well as theoretical analysis reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm. Furthermore, our theory and numerical simulations confirm a prediction of Feichtinger et al. in the context of reconstructing bandlimited functions from nonuniform sampling.

Abstract: Many advances in the development of Krylov subspace methods for the iterative solution of linear systems during the last decade and a half are reviewed. These new developments include different versions of restarted, augmented, deflated, flexible, nested, and inexact methods. Also reviewed are methods specifically tailored to systems with special properties such as special forms of symmetry and those depending on one or more parameters. Copyright © 2006 John Wiley & Sons, Ltd.

Abstract: Summary Interpretation of earth electrical measurements can often be assisted by inversion, which is a non-linear model-fitting problem in these cases. Iterative methods are normally used, and the solution is defined by ' best fit ' in the sense of generalized least-squares. The inverse problems we describe are ill-posed. That is, small changes in the data can lead to large changes in both the solution and in the iterative process that finds the solution. Through an analysis of the problem, based on local linearization, we define a class of methods that stabilize the iteration, and provide a robust solution. These methods are seen as generalizations of the well-known Singular Value Truncation and Marquardt Methods of iterative inversion. Here, and in a companion paper, we give examples illustrating the successful application of the method to ill-posed problems relating to the resistivity of the Earth. In this paper we present an analysis of the solution to a number of geophysical inverse problems. We also provide a reference for the companion paper (Joint Inversion of Geophysical Data, Vozoff & Jupp 1975), where the results are applied to some specific examples. Solutions to geophysical inverse problems are generally non-unique (Backus & Gilbert 1967, 1968, 1970), and it is usual to reduce the non-uniqueness by restricting the complexity of the Earth models. The mathematical problem that arises is commonly ill-posed (unstable) in the sense that small changes in the data lead to large changes in the solution. The solution methods must take careful account of this inherent problem. In the companion paper, and the example given in Section 3 we have data in the form of apparent resistivity measurements for both magnetotelluric (MT), and Direct Current (DC) survey methods. The restricted class of earth models consists of horizontally layered, isotropic media, with constant resistivity in each layer. The simplified inverse problem is, in this case, to find the layer resistivities and thicknesses that best fit the observed data. The analysis of the problem is not, however, restricted to layered models, but applies to any geophysical inverse problem in which the partial derivatives of the (predicted) data with respect to the (unknown) model parameters can be calculated.

Abstract: Regularization is a required component of geophysical-estimation problems that operate with insufficient data. The goal of regularization is to impose additional constraints on the estimated model. I introduce shaping regularization, a general method for imposing constraints by explicit mapping of the estimated model to the space of admissible models. Shaping regularization is integrated in a conjugate-gradient algorithm for iterative least-squares estimation. It provides the advantage of better control on the estimated model in comparison with traditional regularization methods and, in some cases, leads to a faster iterative convergence. Simple data interpolation and seismic-velocity estimation examples illustrate the concept.

Abstract: We establish a class of accelerated Hermitian and skew-Hermitian splitting (AHSS) iteration methods for large sparse saddle-point problems by making use of the Hermitian and skew-Hermitian splitting (HSS) iteration technique. These methods involve two iteration parameters whose special choices can recover the known preconditioned HSS iteration methods, as well as yield new ones. Theoretical analyses show that the new methods converge unconditionally to the unique solution of the saddle-point problem. Moreover, the optimal choices of the iteration parameters involved and the corresponding asymptotic convergence rates of the new methods are computed exactly. In addition, theoretical properties of the preconditioned Krylov subspace methods such as GMRES are investigated in detail when the AHSS iterations are employed as their preconditioners. Numerical experiments confirm the correctness of the theory and the effectiveness of the methods.

Abstract: RELIEF is considered one of the most successful algorithms for assessing the quality of features. In this paper, we propose a set of new feature weighting algorithms that perform significantly better than RELIEF, without introducing a large increase in computational complexity. Our work starts from a mathematical interpretation of the seemingly heuristic RELIEF algorithm as an online method solving a convex optimization problem with a margin-based objective function. This interpretation explains the success of RELIEF in real application and enables us to identify and address its following weaknesses. RELIEF makes an implicit assumption that the nearest neighbors found in the original feature space are the ones in the weighted space and RELIEF lacks a mechanism to deal with outlier data. We propose an iterative RELIEF (I-RELIEF) algorithm to alleviate the deficiencies of RELIEF by exploring the framework of the expectation-maximization algorithm. We extend I-RELIEF to multiclass settings by using a new multiclass margin definition. To reduce computational costs, an online learning algorithm is also developed. Convergence analysis of the proposed algorithms is presented. The results of large-scale experiments on the UCI and microarray data sets are reported, which demonstrate the effectiveness of the proposed algorithms, and verify the presented theoretical results

Abstract: two approaches for solving the corresponding nonlinear eigenvalue problem are proposed. The first one is based on an asymptotic expansion of the solution, the baseline being the acoustic modes and frequencies for a steady (or passive) flame and appropriate boundary conditions. This method allows a quick assessment of any acoustic mode stabilitybutisvalidonlyforcaseswherethecouplingbetweenthe flameandtheacousticwavesissmallinamplitude. The second approach is based on an iterative algorithm where a quadratic eigenvalue problem is solved at each subiteration. It is more central processing unit demanding but remains valid even in cases where the response of the flametoacousticperturbationsislarge.Frequency-dependentboundaryimpedancesareaccountedforinbothcases. A parallel implementation of the Arnoldi iterative method is used to solve the large eigenvalue problem that arises fromthespacediscretization ofthe Helmholtzequation.Several academicandindustrial testcasesareconsideredto illustrate the potential of the method.

Abstract: In this paper, we introduce two iterative sequences for finding a common element of the set of fixed points of a nonexpansive mapping and the set of solutions of an equilibrium problem in a Hilbert space. Then, we show that one of the sequences converges strongly and the other converges weakly.

Abstract: Traditionally, three-dimensional reconstruction methods have been classified into two major groups, Fourier reconstruction methods and direct methods (e.g., Crowther et al., 1970; Gilbert, 1972). Fourier methods are defined as algorithms that restore the Fourier transform of the object from the Fourier transforms of the projections and then obtain the real-space distribution of the object by inverse Fourier transformation. Included in this group are also equivalent reconstruction schemes that use expansions of object and projections into orthogonal function systems (e.g., Cormack, 1963, 1964; Smith et al., 1973; Zeitler, Chapter 4). In contrast, direct methods are defined as those that carry out all calculations in real space. These include the convolution back-projection algorithms (Bracewell and Riddle, 1967; Ramachandran and Lakshminarayanan, 1971; Gilbert, 1972) and iterative algorithms (Gordon et al., 1970; Colsher, 1977). Weighted back-projection methods are difficult to classify in this scheme, since they are equivalent to convolution back-projection algorithms, but work on the real-space data as well as the Fourier transform data of either the object or the projections. Both convolution back-projection and weighted back-projection algorithms are based on the same theory as Fourier reconstruction methods, whereas iterative methods normally do not take into account the Fourier relations between object transform and projection transforms. Thus, it seems justified to classify the reconstruction algorithms into three groups: Fourier reconstruction methods, modified back-projection methods, and iterative direct space methods, where the second group includes convolution backprojection as well as weighted back-projection methods.

Abstract: We present a method for learning sparse representations shared across multiple tasks. This method is a generalization of the well known single task 1-norm regularization. It is based on a novel non-convex regularizer which controls the number of learned features common across the tasks. We prove that the method is equivalent to solving a convex optimization problem for which there is an iterative algorithm which, as we prove, converges to an optimal solution. The algorithm has a simple interpretation: it alternately performs a supervised and an unsupervised step, where in the former step it learns task functions and in the latter step it learns common across tasks sparse representations for these functions. We also provide an extension of the algorithm which learns sparse nonlinear representations using kernels. We report experiments on simulated and real data sets which demonstrate that the proposed method can both improve the performance relative to learning each task independently and lead to a few learned features common across related tasks. Our algorithm can also be used, as a special case, to simply select - not learn - a few common variables across the tasks.

Abstract: Breast-cancer screening using microwave imaging is emerging as a new promising technique as a supplement to X-ray mammography. To create tomographic images from microwave measurements, it is necessary to solve a nonlinear inversion problem, for which an algorithm based on the iterative Gauss-Newton method has been developed at Dartmouth College. This algorithm determines the update values at each iteration by solving the set of normal equations of the problem using the Tikhonov algorithm. In this paper, a new algorithm for determining the iteration update values in the Gauss-Newton algorithm is presented which is based on the conjugate gradient least squares (CGLS) algorithm. The iterative CGLS algorithm is capable of solving the update problem by operating on just the Jacobian and the regularizing effects of the algorithm can easily be controlled by adjusting the number of iterations. The new algorithm is compared to the Gauss-Newton algorithm with Tikhonov regularization and is shown to reconstruct images of similar quality using fewer iterations.

Abstract: In this article a unified approach to iterative soft-thresholding algorithms for the solution of linear operator equations in infinite dimensional Hilbert spaces is presented. We formulate the algorithm in the framework of generalized gradient methods and present a new convergence analysis. As main result we show that the algorithm converges with linear rate as soon as the underlying operator satisfies the so-called finite basis injectivity property or the minimizer possesses a so-called strict sparsity pattern. Moreover it is shown that the constants can be calculated explicitly in special cases (i.e. for compact operators). Furthermore, the techniques also can be used to establish linear convergence for related methods such as the iterative thresholding algorithm for joint sparsity and the accelerated gradient projection method.

Abstract: SUMMARY The computational bottleneck of topology optimization is the solution of a large number of linear systems arising in the finite element analysis. We propose fast iterative solvers for large threedimensional topology optimization problems to address this problem. Since the linear systems in the sequence of optimization steps change slowly from one step to the next, we can significantly reduce the number of iterations and the runtime of the linear solver by recycling selected search spaces from previous linear systems. In addition, we introduce a MINRES (Minimum Residual method) version with recycling (and a short term recurrence) to make recycling more efficient for symmetric problems. Furthermore, we discuss preconditioning to ensure fast convergence. We show that a proper rescaling of the linear systems reduces the huge condition numbers that typically occur in topology optimization to roughly those arising for a problem with constant density. We demonstrate the effectiveness of our solvers by solving a topology optimization problem with more than a million unknowns on a fast PC.

Abstract: A technique for the determination of the equivalent currents distribution from a known radiated field is described. This Inverse Radiation Problem is solved through an Integral Equation algorithm that allows the characterization of antennas of complex geometry both for near field to far field (NF-FF) transformation purposes as well as for diagnostic tasks. The algorithm is based on the representation of the radiating structure by means of a set of equivalent currents over a three-dimensional (3D) surface that can be fitted to the arbitrary geometry of the antenna. The innovative formulation uses an integral equation involving the electric field due to the currents tangential components to the represented antenna 3D surface. For that purpose, both the magnetic and electric equivalent currents are considered in the integral equations. Regularization techniques are also introduced to improve the convergence of the proposed iterative solution. The paper concludes with several results related to the practical verification of the Equivalence Principle and the characterization of a horn antenna.

Abstract: The Gauss-Newton algorithm is an iterative method regularly used for solving nonlinear least squares problems. It is particularly well suited to the treatment of very large scale variational data assimilation problems that arise in atmosphere and ocean forecasting. The procedure consists of a sequence of linear least squares approximations to the nonlinear problem, each of which is solved by an “inner” direct or iterative process. In comparison with Newton’s method and its variants, the algorithm is attractive because it does not require the evaluation of second-order derivatives in the Hessian of the objective function. In practice the exact Gauss-Newton method is too expensive to apply operationally in meteorological forecasting, and various approximations are made in order to reduce computational costs and to solve the problems in real time. Here we investigate the effects on the convergence of the Gauss-Newton method of two types of approximation used commonly in data assimilation. First, we examine “truncated” Gauss-Newton methods where the inner linear least squares problem is not solved exactly, and second, we examine “perturbed” Gauss-Newton methods where the true linearized inner problem is approximated by a simplified, or perturbed, linear least squares problem. We give conditions ensuring that the truncated and perturbed Gauss-Newton methods converge and also derive rates of convergence for the iterations. The results are illustrated by a simple numerical example. A practical application to the problem of data assimilation in a typical meteorological system is presented.

Abstract: This paper proposes an iterative synthesis approach to Petri net (PN)-based deadlock prevention policy for flexible manufacturing systems (FMS). Given the PN model (PNM) of an FMS prone to deadlock, the goal is to synthesize a live controlled PNM. Its use for FMS control guarantees its deadlock-free operation and high performance in terms of resource utilization and system throughput. The proposed method is an iterative approach. At each iteration, a first-met bad marking is singled out from the reachability graph of a given PNM. The objective is to prevent this marking from being reached via a place invariant of the PN. A well-established invariant-based control method is used to derive a control place. This process is carried out until the net model becomes live. The proposed method is generally applicable, easy to use, effective, and straightforward although its off-line computation is of exponential complexity. Two FMS are used to show its effectiveness and applicability

Abstract: Existing methods for estimating the covariance of the ICP (iterative closest/corresponding point) algorithm are either inaccurate or are computationally too expensive to be used online. This paper proposes a new method, based on the analysis of the error function being minimized. It considers that the correspondences are not independent (the same measurement being used in more than one correspondence), and explicitly utilizes the covariance matrix of the measurements, which are not assumed to be independent either. The validity of the approach is verified through extensive simulations: it is more accurate than previous methods and its computational load is negligible. The ill-posedness of the surface matching problem is explicitly tackled for under-constrained situations by performing an observability analysis; in the analyzed cases the method still provides a good estimate of the error projected on the observable manifold.