Abstract: An effective technique in locating a source based on intersections of hyperbolic curves defined by the time differences of arrival of a signal received at a number of sensors is proposed. The approach is noniterative and gives an explicit solution. It is an approximate realization of the maximum-likelihood estimator and is shown to attain the Cramer-Rao lower bound near the small error region. Comparisons of performance with existing techniques of beamformer, spherical-interpolation, divide and conquer, and iterative Taylor-series methods are made. The proposed technique performs significantly better than spherical-interpolation, and has a higher noise threshold than divide and conquer before performance breaks away from the Cramer-Rao lower bound. It provides an explicit solution form that is not available in the beamforming and Taylor-series methods. Computational complexity is comparable to spherical-interpolation but substantially less than the Taylor-series method. >

Abstract: An iterative algorithm is proposed for nonlinearly constrained optimization calculations when there are no derivatives. Each iteration forms linear approximations to the objective and constraint functions by interpolation at the vertices of a simplex and a trust region bound restricts each change to the variables. Thus a new vector of variables is calculated, which may replace one of the current vertices, either to improve the shape of the simplex or because it is the best vector that has been found so far, according to a merit function that gives attention to the greatest constraint violation. The trust region radius ρ is never increased, and it is reduced when the approximations of a well-conditioned simplex fail to yield an improvement to the variables, until ρ reaches a prescribed value that controls the final accuracy. Some convergence properties and several numerical results are given, but there are no more than 9 variables in these calculations because linear approximations can be highly inefficient. Nevertheless, the algorithm is easy to use for small numbers of variables.

Abstract: The expectation-maximization (EM) method can facilitate maximizing likelihood functions that arise in statistical estimation problems. In the classical EM paradigm, one iteratively maximizes the conditional log-likelihood of a single unobservable complete data space, rather than maximizing the intractable likelihood function for the measured or incomplete data. EM algorithms update all parameters simultaneously, which has two drawbacks: 1) slow convergence, and 2) difficult maximization steps due to coupling when smoothness penalties are used. The paper describes the space-alternating generalized EM (SAGE) method, which updates the parameters sequentially by alternating between several small hidden-data spaces defined by the algorithm designer. The authors prove that the sequence of estimates monotonically increases the penalized-likelihood objective, derive asymptotic convergence rates, and provide sufficient conditions for monotone convergence in norm. Two signal processing applications illustrate the method: estimation of superimposed signals in Gaussian noise, and image reconstruction from Poisson measurements. In both applications, the SAGE algorithms easily accommodate smoothness penalties and converge faster than the EM algorithms. >

Abstract: Two-dimensional (2D) phase unwrapping continues to find applications in a wide variety of scientific and engineering areas including optical and microwave interferometry, adaptive optics, compensated imaging, and synthetic-aperture-radar phase correction, and image processing. We have developed a robust method (not based on any path-following scheme) for unwrapping 2D phase principal values (in a least-squares sense) by using fast cosine transforms. If the 2D phase values are associated with a 2D weighting, the fast transforms can still be used in iterative methods for solving the weighted unwrapping problem. Weighted unwrapping can be used to isolate inconsistent regions (i.e., phase shear) in an elegant fashion.

Abstract: This Note is devoted to a new iterative algorithm to compute the local and overall response of a composite from images of its (complex) microstructure. The elastic problem for a heterogeneous material is formulated with the help of a homogeneous reference medium and written under the form of a periodic Lippman-Schwinger equation. Using the fact that the Green's function of the pertinent operator is known explicitely in Fourier space, this equation is solved iteratively.The method is extended to the case where the individual constituents are elastic-plastic Von Mises materials with isotropic hardening

Abstract: It is well known that calibration errors can seriously degrade performance in adaptive arrays, particularly when the input signal-to-noise ratio is large. The effect is caused by the perturbation of the presumed steering vector from its optimal value. Although it is not as widely known, similar degradation occurs in sampled matrix inversion processing when the covariance matrix is estimated while the desired signal is present in the snapshot data. Under these conditions, performance loss is due to errors in the estimated covariance matrix and occurs even when the steering vector is known exactly. In the paper, a new method based of modification of the steering vector is proposed to overcome both the problems of perturbation and of sample covariance errors. The method involves projection of the presumed steering vector onto the observed signal-plus-interference subspace. An analysis is also presented illustrating that the sample covariance errors can be viewed as a particular type of perturbation error and a simple approximation is derived for the expected beamformer performance as a function of the number of data snapshots. Both analytical and experimental results are presented that illustrate the advantages of the proposed method. >

Abstract: In this paper we present the stability and convergence results for dynamic programming-based reinforcement learning applied to linear quadratic regulation (LQR). The specific algorithm we analyze is based on Q-learning and it is proven to converge to an optimal controller provided that the underlying system is controllable and a particular signal vector is persistently excited. This is the first convergence result for DP-based reinforcement learning algorithms for a continuous problem.

Abstract: In this paper, a new algorithm for the solution of large-scale systems of differential-algebraic equations is described. It is based on the integration methods in the solver DASSL, but instead of a direct method for the associated linear systems which arise at each time step, we apply the preconditioned GMRES iteration in combination with an Inexact Newton Method. The algorithm, along with those in DASSL, is implemented in a new solver called DASPK. We outline the algorithms and strategies used, and discuss the use of the solver. We develop and analyze some preconditioners for a certain class of DAE stems, and finally demonstrate the application of DASPK on two example problems.

Abstract: Mixed finite element approximation of the classical Stokes problem describing slow viscous incompressible flow gives rise to symmetric indefinite systems for the discrete velocity and pressure variables. Iterative solution of such indefinite systems is feasible and is an attractive approach for large problems. Part I of this work described a conjugate gradient-like method (the method of preconditioned conjugate residuals) which is applicable to symmetric indefinite problems [A. J. Wathen and D. J. Silvester, SIAM J. Numer. Anal., 30 (1993), pp. 630–649]. Using simple arguments, estimates for the eigenvalue distribution of the discrete Stokes operator on which the convergence rate of the iteration depends are easily derived. Part I discussed the important case of diagonal preconditioning (scaling). This paper considers the more general class of block preconditioners, where the partitioning into blocks corresponds to the natural partitioning into the velocity and pressure variables. It is shown that, provid...

Abstract: The expectation-maximization (EM) algorithm is an important tool for maximum-likelihood (ML) estimation and image reconstruction, especially in medical imaging. It is a non-linear iterative algorithm that attempts to find the ML estimate of the object that produced a data set. The convergence of the algorithm and other deterministic properties are well established, but relatively little is known about how noise in the data influences noise in the final reconstructed image. In this paper we present a detailed treatment of these statistical properties. The specific application we have in mind is image reconstruction in emission tomography, but the results are valid for any application of the EM algorithm in which the data set can be described by Poisson statistics. We show that the probability density function for the grey level at a pixel in the image is well approximated by a log-normal law. An expression is derived for the variance of the grey level and for pixel-to-pixel covariance. The variance increases rapidly with iteration number at first, but eventually saturates as the ML estimate is approached. Moreover, the variance at any iteration number has a factor proportional to the square of the mean image (though other factors may also depend on the mean image), so a map of the standard deviation resembles the object itself. Thus low-intensity regions of the image tend to have low noise. By contrast, linear reconstruction methods, such as filtered back-projection in tomography, show a much more global noise pattern, with high-intensity regions of the object contributing to noise at rather distant low-intensity regions. The theoretical results of this paper depend on two approximations, but in the second paper in this series we demonstrate through Monte Carlo simulation that the approximations are justified over a wide range of conditions in emission tomography. The theory can, therefore, be used as a basis for calculation of objective figures of merit for image quality.

Abstract: The generalized Lloyd algorithm plays an important role in the design of vector quantizers (VQ) and in feature clustering for pattern recognition. In the VQ context, this algorithm provides a procedure to iteratively improve a codebook and results in a local minimum that minimizes the average distortion function. We propose an efficient method to obtain a good initial codebook that can accelerate the convergence of the generalized Lloyd algorithm and achieve a better local minimum as well. >

Abstract: In this contribution we propose an optimization approach to the design of a restricted complexity controller. The design criterion is of LQG type containing two terms. The first term is the quadratic norm of the error between the output of the true closed loop and a desired response. The second term is the quadratic norm of the input signal. It is shown that the minimization of this criterion does not require a model of the system. Closed loop experimental data can be used instead. The result is an iterative scheme of closed loop experiments and controller updates which converges to a local minimum of the design criterion under the condition of bounded signals. >

Abstract: Picard iteration is a widely used procedure for solving the nonlinear equation governing flow in variably saturated porous media. The method is simple to code and computationally cheap, but has been known to fail or converge slowly. The Newton method is more complex and expensive (on a per-iteration basis) than Picard, and as such has not received very much attention. Its robustness and higher rate of convergence, however, make it an attractive alternative to the Picard method, particularly for strongly nonlinear problems. In this paper the Picard and Newton schemes are implemented and compared in one-, two-, and three-dimensional finite element simulations involving both steady state and transient flow. The eight test cases presented highlight different aspects of the performance of the two iterative methods and the different factors that can affect their convergence and efficiency, including problem size, spatial and temporal discretization, initial solution estimates, convergence error norm, mass lumping, time weighting, conductivity and moisture content characteristics, boundary conditions, seepage faces, and the extent of fully saturated zones in the soil. Previous strategies for enhancing the performance of the Picard and Newton schemes are revisited, and new ones are suggested. The strategies include chord slope approximations for the derivatives of the characteristic equations, relaxing convergence requirements along seepage faces, dynamic time step control, nonlinear relaxation, and a mixed Picard-Newton approach. The tests show that the Picard or relaxed Picard schemes are often adequate for solving Richards' equation, but that in cases where these fail to converge or converge slowly, the Newton method should be used. The mixed Picard-Newton approach can effectively overcome the Newton scheme's sensitivity to initial solution estimates, while comparatively poor performance is reported for the various chord slope approximations. Finally, given the reliability and efficiency of current conjugate gradient-like methods for solving linear nonsymmetric systems, the only real drawback of using Newton rather than Picard iteration is the algebraic complexity and computational cost of assembling the derivative terms of the Jacobian matrix, and it is suggested that both methods can be effectively implemented and used in numerical models of Richards' equation.

Abstract: A new approach to the design of a digital algorithm for voltage phasor and local system frequency estimation is presented. The estimation problem is considered as an unconstrained optimization problem. The algorithm is derived using Newton's iterative method, very commonly used in load-flow studies. The algorithm showed a very high level of robustness as well as high measurement accuracy over a wide range of frequency changes. The algorithm convergence of order two provided fast response and adaptability. To demonstrate the performance of the algorithm developed, computer simulated, experimentally obtained and real-life data records are processed. The presented work is a part of a project concerning the application of microprocessors in frequency relaying. >

Abstract: With the increased availability of faster computers, it is now practical to employ numerical modeling techniques to invert resistivity data for 3-D structure. Full and approximate 3-D inversion methods using the finite-element solution for the forward problem have been developed. Both methods use reciprocity for efficient evaluations of the partial derivatives of apparent resistivity with respect to model resistivities. In the approximate method, the partial derivatives are approximated by those for a homogeneous half-space, and thus the computation time and memory requirement are further reduced. The methods are applied to synthetic data sets from 3-D models to illustrate their effectiveness. They give a good approximation of the actual 3-D structure after several iterations in practical situations where the effects of model inadequacy and topography exist. Comparisons of numerical examples show that the full inversion method gives a better resolution, particularly for the near-surface features, than does the approximate method. Since the full derivatives are more sensitive to local features of resistivity variations than are the approximate derivatives, the resolution of the full method may be further improved when the finite-element solutions are performed more accurately and more efficiently.

Abstract: Neural networks can be viewed as applications that map one space, the input space, into some output space. In order to simulate the desired mapping the network has to go through a learning process consisting of an iterative change of the internal parameters, through the presentation of many input patterns and their corresponding output patterns. The training process is accomplished if the error between the computed output and the desired output pattern is minimal for all examples in the training set. The network will then simulate the desired mapping on the restricted domain of the training examples. We describe an experiment where a neural network is designed to accept a synthetic common shot gather (i.e., a set of seismograms obtained from a single source), as its input pattern and to compute the corresponding one-dimensional large-scale velocity model as its output. The subsurface models are built up of eight layers with constant layer thickness over a homogeneous half-space, 450 examples are used to train the network. After the training process the network never computes a subsurface model which perfectly fits the desired one, but the approximation of the network is sufficient to take this model as starting model for further seismic imaging algorithms. The trained network computes satisfactory velocity profiles for 80% of the new seismic gathers not included in the training set. Although the network gives results that are stable when the input is contaminated with white noise, the network is not robust against strong, i.e., correlated, noise. This application proves that neural networks are able to solve nontrivial inverse problems.

Abstract: The Lanczos process is a well known technique for computing a few, sayk, eigenvalues and associated eigenvectors of a large symmetric nn matrix. However, loss of orthogonality of the computed Krylov subspace basis can reduce the accuracy of the computed approximate eigenvalues. In the implicitly restarted Lanczos method studied in the present paper, this problem is addressed by xing the number of steps in the Lanczos process at a prescribed value, k +p ,w herep typically is not much larger, and may be smaller, than k. Orthogonality of the k +p basis vectors of the Krylov subspace is secured by reorthogonalizing these vectors when necessary. The implicitly restarted Lanczos method exploits that the residual vector obtained by the Lanczos process is a function of the initial Lanczos vector. The method updates the initial Lanczos vector through an iterative scheme. The purpose of the iterative scheme is to determine an initial vector such that the associated residual vector is tiny. If the residual vector vanishes, then an invariant subspace has been found. This paper studies several iterative schemes, among them schemes based on Leja points. The resulting algorithms are capable of computing a few of the largest or smallest eigenvalues and associated eigenvectors. This is accomplished using only (k +p)n +O((k +p) 2 ) storage locations in addition to the storage required for the matrix, where the second term is independent of n.

Abstract: We consider the problem in which we want to separate two (or more) signals that are coupled to each other through an unknown multiple-input-multiple-output linear system (channel). We prove that the signals can be decoupled, or separated, using only the condition that they are statistically independent, and find even weaker sufficient conditions involving their cross-polyspectra. By imposing these conditions on the reconstructed signals, we obtain a class of criteria for signal separation. These criteria are universal in the sense that they do not require any prior knowledge or information concerning The nature of the source signals. They may be communication signals, or speech signals, or any other 1-D or multidimensional signals (e.g., images). Computationally efficient algorithms for implementing the proposed criteria, that only involve the iterative solution to a linear least squares problem, are presented. >

Abstract: Multilevel Schwarz methods are developed for a conforming finite element approximation of second order elliptic problems. We focus on problems in three dimensions with possibly large jumps in the coefficients across the interface separating the subregions. We establish a condition number estimate for the iterative operator, which is independent of the coefficients, and grows at most as the square of the number of levels. We also characterize a class of distributions of the coefficients, called quasi-monotone, for which the weighted\(L^{2}\) -projection is stable and for which we can use the standard piecewise linear functions as a coarse space. In this case, we obtain optimal methods, i.e. bounds which are independent of the number of levels and subregions. We also design and analyze multilevel methods with new coarse spaces given by simple explicit formulas. We consider nonuniform meshes and conclude by an analysis of multilevel iterative substructuring methods.

Abstract: Domain decomposition methods provide powerful preconditioners for the iterative solution of the large systems of algebraic equations that arise in finite element or finite difference approximations...

Abstract: The problem of entropy-constrained multiple-description scalar quantizer design is posed as an optimization problem, necessary conditions for optimality are derived, and an iterative design algorithm is presented. Performance results are presented for a Gaussian source, along with comparisons to the multiple-description rate distortion bound and a reference system. >

Abstract: The Lanczos process is a well known technique for computing a few, sayk, eigenvalues and associated eigenvectors of a large symmetric nn matrix. However, loss of orthogonality of the computed Krylov subspace basis can reduce the accuracy of the computed approximate eigenvalues. In the implicitly restarted Lanczos method studied in the present paper, this problem is addressed by xing the number of steps in the Lanczos process at a prescribed value, k +p ,w herep typically is not much larger, and may be smaller, than k. Orthogonality of the k +p basis vectors of the Krylov subspace is secured by reorthogonalizing these vectors when necessary. The implicitly restarted Lanczos method exploits that the residual vector obtained by the Lanczos process is a function of the initial Lanczos vector. The method updates the initial Lanczos vector through an iterative scheme. The purpose of the iterative scheme is to determine an initial vector such that the associated residual vector is tiny. If the residual vector vanishes, then an invariant subspace has been found. This paper studies several iterative schemes, among them schemes based on Leja points. The resulting algorithms are capable of computing a few of the largest or smallest eigenvalues and associated eigenvectors. This is accomplished using only (k +p)n +O((k +p) 2 ) storage locations in addition to the storage required for the matrix, where the second term is independent of n.

Abstract: In many applications of image analysis, simply connected objects are to be located in noisy images. During the last 5-6 years active contour models have become popular for finding the contours of such objects. Connected to these models are iterative algorithms for finding the minimizing energy curves making the curves behave dynamically through the iterations. These approaches do however have several disadvantages. The numerical algorithms that are in use constrain the models that can be used. Furthermore, in many cases only local minima can be achieved. In this paper, the author discusses a method for curve detection based on a fully Bayesian approach. A model for image contours which allows the number of nodes on the contours to vary is introduced. Iterative algorithms based on stochastic sampling is constructed, which make it possible to simulate samples from the posterior distribution, making estimates and uncertainty measures of specific quantities available. Further, simulated annealing schemes making the curve move dynamically towards the global minimum energy configuration are presented. In theory, no restrictions on the models are made. In practice, however, computational aspects must be taken into consideration when choosing the models. Much more general models than the one used for active contours may however be applied. The approach is applied to ultrasound images of the left ventricle and to magnetic resonance images of the human brain, and show promising results. >

Abstract: We give an overview of various Lanczos/Krylov space methods and the way in which they are being used for solving certain problems in Control Systems Theory based on state-space models. The matrix methods used are based on Krylov sequences and are closely related to modern iterative methods for standard matrix problems such as sets of linear equations and eigenvalue calculations. We show how these methods can be applied to problems in Control Theory such as controllability, observability, and model reduction. All the methods are based on the use of state-space models, which may be very sparse and of high dimensionality. For example, we show how one may compute an approximate solution to a Lyapunov equation arising from a discrete-time linear dynamic system with a large sparse system matrix by the use of the Arnoldi algorithm, and so obtain an approximate Gramian matrix. This has applications in model reduction. The close relation between the matrix Lanczos algorithm and the algebraic structure of linear control systems is also explored.

Abstract: Structured rank-deficient matrices arise in many applications in signal processing, system identification, and control theory. The author discusses the structured total least squares (STLS) problem, which is the problem of approximating affinely structured matrices (i.e., matrices affine in the parameters) by similarly structured rank-deficient ones, while minimizing an L/sub 2/-error criterion. It is shown that the optimality conditions lead to a nonlinear generalized singular value decomposition, which can be solved via an algorithm that is inspired by inverse iteration. Next the author concentrates on the so-called L/sub 2/-optimal noisy realization problem, which is equivalent with approximating a given data sequence by the impulse response of a finite dimensional, time invariant linear system of a given order. This can be solved as a structured total least squares problem. It is shown with some simple counter examples that "classical" algorithms such as the Steiglitz-McBride (1965), iterative quadratic maximum likelihood and Cadzow's (1988) iteration do not converge to the optimal L/sub 2/ solution, despite misleading claims in the literature. >

Abstract: This paper is a continuation of Part I [M. H. Gutknecht, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 594--639], where the theory of the "unsymmetric" Lanczos biorthogonalization (BO) algorithm and the corresponding iterative method BIORES for non-Hermitian linear systems was extended to the nongeneric case. The analogous extension is obtained here for the biconjugate gradient (or BIOMIN) method and for the related BIODIR method. Here, too, the breakdowns of these methods can be cured. As a preparation, mixed recurrence formulas are derived for a pair of sequences of formal orthogonal polynomials belonging to two adjacent diagonals in a nonnormal Pade table, and a matrix interpretation of these recurrences is developed. This matrix interpretation leads directly to a completed formulation of the progressive qd algorithm, valid also in the case of a nonnormal Pade table. Finally, it is shown how the cure for exact breakdown can be extended to near-breakdown in such a way that (in exact arithmetic) the well-conditioned formal orthogonal polynomials and the corresponding Krylov space vectors do not depend on the threshold specifying the near-breakdown.

Abstract: In this paper we introduce a new iterative scheme for the numerical solution of a class of linear variational inequalities. Each iteration of the method consists essentially only of a projection to a closed convex set and two matrix-vector multiplications. Both the method and the convergence proof are very simple.

Abstract: This paper describes a system which can perform full 3-D pose estimation of a single arbitrarily shaped, rigid object at rates up to 10 Hz. A triangular mesh model of the object to be tracked is generated offline using conventional range sensors. Real-time range data of the object is sensed by the CMU high speed VLSI range sensor. Pose estimation is performed by registering the real-time range data to the triangular mesh model using an enhanced implementation of the Iterative Closest Point (ICP) Algorithm introduced by Besl and McKay (1992). The method does not require explicit feature extraction or specification of correspondence. Pose estimation accuracies of the order of 1% of the object size in translation, and 1 degree in rotation have been measured. >