Abstract: Finite representations Finite evaluation Finite convergence Computable sufficient conditions for existence and convergence Safe starting regions for iterative methods Applications to mathematical programming Applications to operator equations An application in finance Internal rates-of-return.

Abstract: Preface Symbols and Acronyms 1. Setting the Stage. Problems With Ill-Conditioned Matrices Ill-Posed and Inverse Problems Prelude to Regularization Four Test Problems 2. Decompositions and Other Tools. The SVD and its Generalizations Rank-Revealing Decompositions Transformation to Standard Form Computation of the SVE 3. Methods for Rank-Deficient Problems. Numerical Rank Truncated SVD and GSVD Truncated Rank-Revealing Decompositions Truncated Decompositions in Action 4. Problems with Ill-Determined Rank. Characteristics of Discrete Ill-Posed Problems Filter Factors Working with Seminorms The Resolution Matrix, Bias, and Variance The Discrete Picard Condition L-Curve Analysis Random Test Matrices for Regularization Methods The Analysis Tools in Action 5. Direct Regularization Methods. Tikhonov Regularization The Regularized General Gauss-Markov Linear Model Truncated SVD and GSVD Again Algorithms Based on Total Least Squares Mollifier Methods Other Direct Methods Characterization of Regularization Methods Direct Regularization Methods in Action 6. Iterative Regularization Methods. Some Practicalities Classical Stationary Iterative Methods Regularizing CG Iterations Convergence Properties of Regularizing CG Iterations The LSQR Algorithm in Finite Precision Hybrid Methods Iterative Regularization Methods in Action 7. Parameter-Choice Methods. Pragmatic Parameter Choice The Discrepancy Principle Methods Based on Error Estimation Generalized Cross-Validation The L-Curve Criterion Parameter-Choice Methods in Action Experimental Comparisons of the Methods 8. Regularization Tools Bibliography Index.

Abstract: Preface How to Get the Software Part I. Linear Equations. 1. Basic Concepts and Stationary Iterative Methods 2. Conjugate Gradient Iteration 3. GMRES Iteration Part II. Nonlinear Equations. 4. Basic Concepts and Fixed Point Iteration 5. Newton's Method 6. Inexact Newton Methods 7. Broyden's Method 8. Global Convergence Bibliography Index.

Abstract: This book presents a carefully selected group of methods for unconstrained and bound constrained optimization problems and analyzes them in depth both theoretically and algorithmically. It focuses on clarity in algorithmic description and analysis rather than generality, and while it provides pointers to the literature for the most general theoretical results and robust software, the author thinks it is more important that readers have a complete understanding of special cases that convey essential ideas. A companion to Kelley's book, Iterative Methods for Linear and Nonlinear Equations (SIAM, 1995), this book contains many exercises and examples and can be used as a text, a tutorial for self-study, or a reference. Iterative Methods for Optimization does more than cover traditional gradient-based optimization: it is the first book to treat sampling methods, including the Hooke& Jeeves, implicit filtering, MDS, and Nelder& Mead schemes in a unified way.

Abstract: List of symbols and acronyms List of iterative algorithm templates List of direct algorithms List of figures List of tables 1: Introduction 2: A brief tour of Eigenproblems 3: An introduction to iterative projection methods 4: Hermitian Eigenvalue problems 5: Generalized Hermitian Eigenvalue problems 6: Singular Value Decomposition 7: Non-Hermitian Eigenvalue problems 8: Generalized Non-Hermitian Eigenvalue problems 9: Nonlinear Eigenvalue problems 10: Common issues 11: Preconditioning techniques Appendix: of things not treated Bibliography Index .

Abstract: List of Algorithms Preface 1. Introduction. Brief Overview of the State of the Art Notation Review of Relevant Linear Algebra Part I. Krylov Subspace Approximations. 2. Some Iteration Methods. Simple Iteration Orthomin(1) and Steepest Descent Orthomin(2) and CG Orthodir, MINRES, and GMRES Derivation of MINRES and CG from the Lanczos Algorithm 3. Error Bounds for CG, MINRES, and GMRES. Hermitian Problems-CG and MINRES Non-Hermitian Problems-GMRES 4. Effects of Finite Precision Arithmetic. Some Numerical Examples The Lanczos Algorithm A Hypothetical MINRES/CG Implementation A Matrix Completion Problem Orthogonal Polynomials 5. BiCG and Related Methods. The Two-Sided Lanczos Algorithm The Biconjugate Gradient Algorithm The Quasi-Minimal Residual Algorithm Relation Between BiCG and QMR The Conjugate Gradient Squared Algorithm The BiCGSTAB Algorithm Which Method Should I Use? 6. Is There A Short Recurrence for a Near-Optimal Approximation? The Faber and Manteuffel Result Implications 7. Miscellaneous Issues. Symmetrizing the Problem Error Estimation and Stopping Criteria Attainable Accuracy Multiple Right-Hand Sides and Block Methods Computer Implementation Part II. Preconditioners. 8. Overview and Preconditioned Algorithms. 9. Two Example Problems. The Diffusion Equation The Transport Equation 10. Comparison of Preconditioners. Jacobi, Gauss--Seidel, SOR The Perron--Frobenius Theorem Comparison of Regular Splittings Regular Splittings Used with the CG Algorithm Optimal Diagonal and Block Diagonal Preconditioners 11. Incomplete Decompositions. Incomplete Cholesky Decomposition Modified Incomplete Cholesky Decomposition 12. Multigrid and Domain Decomposition Methods. Multigrid Methods Basic Ideas of Domain Decomposition Methods.

Abstract: The aim of this paper is to study the convergence properties of the gradient projection method and to apply these results to algorithms for linearly constrained problems. The main convergence result is obtained by defining a projected gradient, and proving that the gradient projection method forces the sequence of projected gradients to zero. A consequence of this result is that if the gradient projection method converges to a nondegenerate point of a linearly constrained problem, then the active and binding constraints are identified in a finite number of iterations. As an application of our theory, we develop quadratic programming algorithms that iteratively explore a subspace defined by the active constraints. These algorithms are able to drop and add many constraints from the active set, and can either compute an accurate minimizer by a direct method, or an approximate minimizer by an iterative method of the conjugate gradient type. Thus, these algorithms are attractive for large scale problems. We show that it is possible to develop a finite terminating quadratic programming algorithm without non-degeneracy assumptions.

Abstract: A new approach to the design of computer-generated holograms makes optimal use of the available device resolution. An iterative search algorithm minimizes an error criterion by directly manipulating the binary hologram and observing the effect on the desired reconstruction. Several measures of error and efficiency useful in assessing the optimality of digital holograms are defined. Methods for designing digital holograms that are based on projections and error diffusion are presented as established techniques for comparison to direct binary search.

Abstract: Conventional finite-difference operators for numerical differentiation become progressively inaccurate at higher frequencies and therefore require very fine computational grids. This problem is avoided when the derivatives are computed by multiplication in the Fourier domain. However, because matrix transpositions are involved, efficient application of this method is restricted to computational environments where the complete data volume required by each computational step can be kept in random access memory. To circumvent these problems a generalized numerical dispersion analysis for wave equation computations is developed. Operators for spatial differentiation can then be designed by minimizing the corresponding peak relative error in group velocity within a spatial frequency band. For specified levels of maximum relative error in group velocity ranging from 0.03% to 3%, differentiators have been designed that have the largest possible bandwidth for a given operator length. The relation between operator length and the required number of grid points per shortest wavelength, for a required accuracy, provides a useful starting point for the design of cost-effective numerical schemes. To illustrate this, different alternatives for numerical simulation of the time evolution of acoustic waves in three-dimensional inhomogeneous media are investigated. It is demonstrated that algorithms can be implemented that require fewer arithmetic and I/O operations by orders of magnitude compared to conventional second-order finite-difference schemes to yield results with a specified minimum accuracy.

Abstract: In this work we review the present status of numerical methods for partial differential equations on vector and parallel computers. A discussion of the relevant aspects of these computers and a brief review of their development is included, with particular attention paid to those characteristics that influence algorithm selection. Both direct and iterative methods are given for elliptic equations as well as explicit and implicit methods for initial-boundary value problems. The intent is to point out attractive methods as well as areas where this class of computer architecture cannot be fully utilized because of either hardware restrictions or the lack of adequate algorithms. A brief discussion of application areas utilizing these computers is included.

Abstract: A method for improving the convergence of the standard conjugate gradient method is given. This method involves the use of auxiliary subspaces It is shown how such subspaces may be constructed for boundary value problems and an analysis of convergence for second order problems is presented.

Abstract: A generalized full-wave Green's function completely defining the field inside a multilayer dielectric structure due to a current element arbitrarily placed between any two layers is derived in two-dimensional spectral-domain form. It is derived by solving a "standard" form containing the current element with two substrates on either side of it, and using an iterative algorithm to take care of additional layers. Another iterative algorithm is then used to find the field in any layer in terms of the field expressions in the two layers of the "standard" form. The locations of the poles of the Green's function are predicted, and an asymptotic form is derived along with the asymptotic limit, by use of which the multilayer Green's function can be used in numerical methods as efficiently as the single-layer grounded-dielectric-substrate Green's function. This Green's function is then applied to a few multilayer transmission lines for which data are not found in the literature to date.

Abstract: Several preconditioned conjugate gradient (PCG)-based domain decomposition techniques for self-adjoint elliptic partial differential equations in two dimensions are compared against each other and against conventional PCG iterative techniques in serial and parallel contexts. We consider preconditioners that make use of fast Poisson solvers on the subdomain interiors. Several preconditioners for the interfacial equations are tested on a set of model problems involving two or four subdomains, which are prototypes of the stripwise and boxwise decompositions of a two-dimensional region. Selected methods have been implemented on the Intel Hypercube by assigning one processor to each subdomain, making use of up to 64 processors. The choice of a “best” method for a given problem depends in general upon: (a) the domain geometry, (b) the variability of the operator, and (c) machine characteristics such as the number of processors available and their interconnection scheme, the memory available per processor, and c...

Abstract: The major costs of large implicit finite element calculations, particularly in three dimensions, arise from computing solutions to systems of linear equations. Direct methods, i.e., those based upon Gaussian elimination, can easily require prohibitively large amounts of both CPU time and storage, even on current supercomputers. Iterative procedures avoiding the formation and factorization of a global system of equations can circumvent these difficulties. The element-by-element (EBE) preconditioned conjugate gradients (PCG) algorithm is presented in the context of a vectorized implementation within the production nonlinear stress analysis code nike 3 d . Due to continued confusion as to the ease of vectorizing finite element procedures, we include examples of the main EBE subroutines in their entirety. The concept of a fractal dimension of a finite element mesh is introduced, and proves useful in characterizing the efficiency of this iterative algorithm with respect to a variable band, active column direct method. Sample calculations on a Cray X-MP/48 with solid-state storage device (SSD) illustrate the economy and range of applicability of EBE/PCG. Asymptotic cost formulae derived for two linear problems underscore differences between the direct and iterative algorithms for large problems and lead to predictions of problem size limitations imposed by the computing environment.

Abstract: We investigate the reconstruction of band-limited 2-D functions from nonuniformly spaced samples. Three algorithms which iterate between the space and frequency domains are developed. The algorithms differ in the manner in which the known value of the function at sample points is used to correct the reconstruction at each iteration. Convergence of the algorithms is speeded by choosing the largest possible gain for the error correction. The performance of the three methods is compared to that of thin-plate spline interpolation in terms of signal to error energy ratio.

Abstract: It is shown that Berlekamp's iterative algorithm can be derived from a normalized version of Euclid's extended algorithm. Simple proofs of the results given recently by Cheng are also presented.

Abstract: We present an iterative scheme for obstacle problems described by variational inequalities. At each step of the iteration the method leads to a reduced linear (resp. nonlinear) algebraic system whi...

Abstract: In this paper, we give a state-of-the-art review of many iterative methods for solving large convex quadratic programs. We attempt to classify several of the more basic methods in two categories, within each of which a unified iterative scheme will be introduced and its convergence analyzed. Hybrid iterative methods (such as the proximal point algorithm and a diagonalization scheme) that make use of the more basic schemes will also be described. The results of an extensive computer experimentation which is aimed at comparing the relative performance of the various methods will be reported and discussed. Finally, several important topics which require future research will be highlighted.

Abstract: This chapter discusses the convergence rate of iterative methods and the convergence rate of the method of conjugate gradients (cg-method) when applied to ill-posed problems of the kind K x = g , where K: H → H 1 is a linear bounded operator between Hilbert spaces and g ∈ H 1 . For the methods of steepest descent and of conjugate gradients the first convergence proofs for ill-posed problems were given by Kammerer–Nashed in 1971 and 1972. For the cases of the Landweber–Fridman method and the method of steepest descent, the exponent v is best-possible for the class P [ v ]. Iterative methods exist for P [ v ] with convergence rates ∥ e n ∥ = 0((1/ n ) 2 v ), and the cg-method has this property for all v > 0. The chapter presents a family of iterative methods depending on a parameter v > 0, where the sequence S n ( v ) = 1 - λ R n ( v ) of associated polynomials has the property that ω( v, S n ( v ) ) = 0((1/ n ) 2 v ).

Abstract: Optimum Coding in the Frequency domain (OCF) uses entropy coding of quantized spectral coefficients to efficiently code high quality sound signals with 3 bits/sample. In an iterative algorithm psychoacoustic weigthing is used to get the quantization noise to be masked in every critical band. The coder itself uses iterative quantizer control to get each data block to be coded with a fixed number of bits. Details about the OCF-Coder are presented together with information about the codebook needed and the training for the entropy coder. An algorithm for calculating a noise-to-mask ratio is presented which helps to identify, where quantization noise could be audible.

Abstract: It is shown that, under a mild condition, H\infty -optimization and robust stabilization are equivalent to a directional interpolation problem which is a matrix extension of the classical Pick-Nevanlinna interpolation problem. A classical iterative method, which is an extension of the Schur-Nevanlinna algorithm, is given for solving the problem. This method does not require the inner-outer factorization nor the balanced realization of the original plant. A circuit theoretical parameterization of all solutions is derived that is expected to enhance the physical insight to the H\infty -optimal control and robust stabilization. This parameterization has the degree much less than the one obtained previously.

Abstract: An efficient iterative reconstruction method for positron emission tomography (PET) is presented. The algorithm is basically an enhanced EM (expectation maximization) algorithm with improved frequency response. High-frequency components of the ratio of measured to calculated projections are extracted and are taken into account for the iterative correction of image density in such a way that the correction is performed with a uniform efficiency over the image plane and with a flat frequency response. As a result, the convergence speed is not so sensitive to the image pattern or matrix size as the standard EM algorithm, and nonuniformity of the spatial resolution is significantly improved. Nonnegativity of the reconstructed image is preserved. Simulation studies have been made assuming two PET systems: a scanning PET with ideal sampling and a stationary PET with sparse sampling. In the latter, a "bank array" of detectors is employed to improve the sampling in the object plane. The new algorithm provides satisfactory images by two or three iterations starting from a flat image in either case. The behavior of convergence is monitored by evaluating the root mean square of C(b)-1 where C(b) is the correction factor for pixel b in the EM algorithm. The value decreases rapidly and monotonically with iteration number. Although the theory is not accurate enough to assure the stability of convergence, the algorithm is promising to achieve significant saving in computation compared to the standard EM algorithm.

Abstract: Regular mesh-connected arrays are shown to be isomorphic to a class of so-called regular iterative algorithms. For a wide variety of problems it is shown how to obtain appropriate iterative algorithms and then how to translate these algorithms into arrays in a systematic fashion. Several "systolic" arrays presented in the literature are shown to be specific cases of the variety of architectures that can be derived by the techniques presented here. These include arrays for Fourier Transform, Matrix Multiplication, and Sorting.

Abstract: In this paper we consider a symmetric partial algebraic eigenvalue problem. In Section 1 we present several estimates for the rates of convergence of some classical algorithms of vector iterations. Estimates of the accuracy of the Rayleigh-Ritz method and of the subspace iterations are considered in Section 2. The convergence rates of several methods of solving generalized eigenvalue problems by means of a preconditioner are analysed in Section 3. Finally, Section 4 deals with the Temple-Lehmann two-sided estimates for eigenvalues. The paper constitutes a systematic review of recent results mainly due to the author. Consider in a Euclidean space H the problem of computing p maximal eigenvalues A! > · · · > λρ and the corresponding eigenvectors for a generalized eigenvalue problem Mu = XLu, M = M*, L = L*>0 (0.1) (to simplify the notation, all the eigenvalues λί > ··· > λρ are taken to be simple). To calculate the minimal eigenvalues of (0.1), we need only to replace Μ by — Μ throughout. In computational practice the problem is traditionally tackled by implicit reduction of the generalized eigenproblem (0.1) to the ordinary one, i.e. ΑΗ = λΐΛ, A = A* (0.2) where, for example, A = L\"M in the space HL equipped with the scalar product (Λ *)*. = (£·,*). In Section 1 we present several estimates for the convergence rates of some classical methods of vector iterations for problem (0.2) with ρ = 1. In Section 2 we consider some estimates for the accuracy of the Rayleigh-Ritz method and of the subspace iterations for problem (0.2) with ρ > 1. In Section 3 we investigate the convergence rates of several methods of solving eigenvalue problem (0.1) using a preconditioner Β = Β* > 0 such that the system Bu = f can be efficiently solved, and the ratio δ, δ = δ0/δΐ9 0<δ0Β*ζΙιζδ1Β (0.3) is as close to 1 as possible. Finally, Section 4 deals with the Temple-Lehmann approach to obtaining two-sided estimates for the eigenvalues of (0.1). The results of this paper can be useful in evaluating the efficiency of various iterative techniques for solving problems of type (0.1), which can occur from the finite difference or finite element discretization of differential eigenvalue problems. Our choice of Originally published in Russian in Numerical Methods and Mathematical Modelling, Transactions of the Department of Numerical Mathematics of the USSR Academy of Sciences, Moscow, 1986.

Abstract: We propose an algorithm for computing a class of least squares polynomials on polygonal regions of the complex plane. An important application of this technique to solving large sparse linear systems is considered. The advantage of using general polygonal regions instead of ellipses as was done in previous work, is that elliptic regions may fail to accurately represent the convex hull of the spectrum of the matrix A. An attractive feature of the algorithm is that it does not explicitly require any numerical integration. Numerical experiments show that the least-squares based methods for solving linear systems are competitive with the Chebyshev based methods and are more reliable.