Abstract: This paper studies a difficult and fundamental problem that arises throughout electrical engineering, applied mathematics, and statistics. Suppose that one forms a short linear combination of elementary signals drawn from a large, fixed collection. Given an observation of the linear combination that has been contaminated with additive noise, the goal is to identify which elementary signals participated and to approximate their coefficients. Although many algorithms have been proposed, there is little theory which guarantees that these algorithms can accurately and efficiently solve the problem. This paper studies a method called convex relaxation, which attempts to recover the ideal sparse signal by solving a convex program. This approach is powerful because the optimization can be completed in polynomial time with standard scientific software. The paper provides general conditions which ensure that convex relaxation succeeds. As evidence of the broad impact of these results, the paper describes how convex relaxation can be used for several concrete signal recovery problems. It also describes applications to channel coding, linear regression, and numerical analysis

Abstract: We present a method for learning a low-dimensional representation which is shared across a set of multiple related tasks. The method builds upon the well-known 1-norm regularization problem using a new regularizer which controls the number of learned features common for all the tasks. We show that this problem is equivalent to a convex optimization problem and develop an iterative algorithm for solving it. The algorithm has a simple interpretation: it alternately performs a supervised and an unsupervised step, where in the latter step we learn commonacross-tasks representations and in the former step we learn task-specific functions using these representations. We report experiments on a simulated and a real data set which demonstrate that the proposed method dramatically improves the performance relative to learning each task independently. Our algorithm can also be used, as a special case, to simply select – not learn – a few common features across the tasks.

Abstract: We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The "COSSO" is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

Abstract: The quality of images reconstructed by statistical iterative methods depends on an accurate model of the relationship between image space and projection space through the system matrix. The elements of the system matrix for the clinical Hi-Rez scanner were derived by processing the data measured for a point source at different positions in a portion of the field of view. These measured data included axial compression and azimuthal interleaving of adjacent projections. Measured data were corrected for crystal and geometrical efficiency. Then, a whole system matrix was derived by processing the responses in projection space. Such responses included both geometrical and detection physics components of the system matrix. The response was parameterized to correct for point source location and to smooth for projection noise. The model also accounts for axial compression (span) used on the scanner. The forward projector for iterative reconstruction was constructed using the estimated response parameters. This paper extends our previous work to fully three-dimensional. Experimental data were used to compare images reconstructed by the standard iterative reconstruction software and the one modeling the response function. The results showed that the modeling of the response function improves both spatial resolution and noise properties

Abstract: INTRODUCTION AND PROGRAMMING PRELIMINARIES Introduction Running Programs Hardware External Fortran Sub-Program Libraries A Simple Fortran Program Some Simple Fortran Constructs Intrinsic Functions User-Supplied Functions and Subroutines Errors and Accuracy Graphical Output Conclusions LINEAR ALGEBRAIC EQUATIONS Introduction Gaussian Elimination Equation Solution Using Factorization Equations with a Symmetrical Coefficient Matrix Banded Equations Compact Storage for Variable Bandwidths Pivoting Equations with Prescribed Solutions Iterative Methods Gradient Methods Unsymmetrical Systems Preconditioning Comparison of Direct and Iterative Methods Exercises NONLINEAR EQUATIONS Introduction Iterative Substitution Multiple Roots and Other Difficulties Interpolation Methods Extrapolation Methods Acceleration of Convergence Systems of Nonlinear Equations Exercises EIGENVALUE EQUATIONS Introduction Vector Iteration Intermediate Eigenvalues by Deflation The Generalized Eigenvalue Problem [K] {x} = ?[M] {x} Transformation Methods Characteristic Polynomial Methods Exercises INTERPOLATION AND CURVE FITTING Introduction Interpolating Polynomials Interpolation Using Cubic Spline Functions Numerical Differentiation Curve Fitting Exercises NUMERICAL INTEGRATION Introduction Newton-Cotes Rules Gauss-Legendre Rules Adaptive Integration Rules Special Integration Rules Multiple Integrals Exercises NUMERICAL SOLUTION OF ORDINARY DIFFERENTIAL EQUATIONS Introduction Definitions and Types of ODE Initial Value Problems Boundary Value Problems Exercises INTRODUCTION TO PARTIAL DIFFERENTIAL EQUATIONS Introduction Definitions and Types of PDE First Order Equations Second Order Equations Finite Difference Method Finite Element Method Exercises APPENDIX A: Descriptions of Library Subprograms APPENDIX B: Fortran 95 Listings of Library Subprograms APPENDIX C: References and Additional Reading

Abstract: Confocal laser scanning microscopy is a powerful and popular technique for 3D imaging of biological specimens. Although confocal microscopy images are much sharper than standard epifluorescence ones, they are still degraded by residual out-of-focus light and by Poisson noise due to photon-limited detection. Several deconvolution methods have been proposed to reduce these degradations, including the Richardson-Lucy iterative algorithm, which computes maximum likelihood estimation adapted to Poisson statistics. As this algorithm tends to amplify noise, regularization constraints based on some prior knowledge on the data have to be applied to stabilize the solution. Here, we propose to combine the Richardson-Lucy algorithm with a regularization constraint based on Total Variation, which suppresses unstable oscillations while preserving object edges. We show on simulated and real images that this constraint improves the deconvolution results as compared with the unregularized Richardson-Lucy algorithm, both visually and quantitatively.

Abstract: Iterative compiler optimization has been shown to outperform static approaches. This, however, is at the cost of large numbers of evaluations of the program. This paper develops a new methodology to reduce this number and hence speed up iterative optimization. It uses predictive modelling from the domain of machine learning to automatically focus search on those areas likely to give greatest performance. This approach is independent of search algorithm, search space or compiler infrastructure and scales gracefully with the compiler optimization space size. Off-line, a training set of programs is iteratively evaluated and the shape of the spaces and program features are modelled. These models are learnt and used to focus the iterative optimization of a new program. We evaluate two learnt models, an independent and Markov model, and evaluate their worth on two embedded platforms, the Texas Instrument C67I3 and the AMD Au1500. We show that such learnt models can speed up iterative search on large spaces by an order of magnitude. This translates into an average speedup of 1.22 on the TI C6713 and 1.27 on the AMD Au1500 in just 2 evaluations.

Abstract: A geometry optimization method using an energy-represented direct inversion in the iterative subspace algorithm, GEDIIS, is introduced and compared with another DIIS formulation (controlled GDIIS) and the quasi-Newton rational function optimization (RFO) method. A hybrid technique that uses different methods at various stages of convergence is presented. A set of test molecules is optimized using the hybrid, GEDIIS, controlled GDIIS, and RFO methods. The hybrid method presented in this paper results in smooth, well-behaved optimization processes. The optimization speed is the fastest among the methods considered.

Abstract: In emission tomography statistically based iterative methods can improve image quality relative to analytic image reconstruction through more accurate physical and statistical modelling of high-energy photon production and detection processes. Continued exponential improvements in computing power, coupled with the development of fast algorithms, have made routine use of iterative techniques practical, resulting in their increasing popularity in both clinical and research environments. Here we review recent progress in developing statistically based iterative techniques for emission computed tomography. We describe the different formulations of the emission image reconstruction problem and their properties. We then describe the numerical algorithms that are used for optimizing these functions and illustrate their behaviour using small scale simulations.

Abstract: Seismic surveys generally have irregular areas where data cannot be acquired. These data should often be interpolated. A projection onto convex sets (POCS) algorithm using Fourier transforms allows interpolation of irregularly populated grids of seismic data with a simple iterative method that produces high-quality results. The original 2D image restoration method, the Gerchberg-Saxton algorithm, is extended easily to higher dimensions, and the 3D version of the process used here produces much better interpolations than typical 2D methods. The only parameter that makes a substantial difference in the results is the number of iterations used, and this number can be overestimated without degrading the quality of the results. This simplicity is a significant advantage because it relieves the user of extensive parameter testing. Although the cost of the algorithm is several times the cost of typical 2D methods, the method is easily parallelized and still completely practical.

Abstract: State-of-the-art finite-element methods for time-harmonic acoustics governed by the Helmholtz equation are reviewed. Four major current challenges in the field are specifically addressed: the effective treatment of acoustic scattering in unbounded domains, including local and nonlocal absorbing boundary conditions, infinite elements, and absorbing layers; numerical dispersion errors that arise in the approximation of short unresolved waves, polluting resolved scales, and requiring a large computational effort; efficient algebraic equation solving methods for the resulting complex-symmetric (non-Hermitian) matrix systems including sparse iterative and domain decomposition methods; and a posteriori error estimates for the Helmholtz operator required for adaptive methods. Mesh resolution to control phase error and bound dispersion or pollution errors measured in global norms for large wave numbers in finite-element methods are described. Stabilized, multiscale, and other wave-based discretization methods developed to reduce this error are reviewed. A review of finite-element methods for acoustic inverse problems and shape optimization is also given.

Abstract: We propose an image hashing paradigm using visually significant feature points. The feature points should be largely invariant under perceptually insignificant distortions. To satisfy this, we propose an iterative feature detector to extract significant geometry preserving feature points. We apply probabilistic quantization on the derived features to introduce randomness, which, in turn, reduces vulnerability to adversarial attacks. The proposed hash algorithm withstands standard benchmark (e.g., Stirmark) attacks, including compression, geometric distortions of scaling and small-angle rotation, and common signal-processing operations. Content changing (malicious) manipulations of image data are also accurately detected. Detailed statistical analysis in the form of receiver operating characteristic (ROC) curves is presented and reveals the success of the proposed scheme in achieving perceptual robustness while avoiding misclassification

Abstract: In this paper we present a general framework for performing constrained mesh deformation tasks with gradient domain techniques. We present a gradient domain technique that works well with a wide variety of linear and nonlinear constraints. The constraints we introduce include the nonlinear volume constraint for volume preservation, the nonlinear skeleton constraint for maintaining the rigidity of limb segments of articulated figures, and the projection constraint for easy manipulation of the mesh without having to frequently switch between multiple viewpoints. To handle nonlinear constraints, we cast mesh deformation as a nonlinear energy minimization problem and solve the problem using an iterative algorithm. The main challenges in solving this nonlinear problem are the slow convergence and numerical instability of the iterative solver. To address these issues, we develop a subspace technique that builds a coarse control mesh around the original mesh and projects the deformation energy and constraints onto the control mesh vertices using the mean value interpolation. The energy minimization is then carried out in the subspace formed by the control mesh vertices. Running in this subspace, our energy minimization solver is both fast and stable and it provides interactive responses. We demonstrate our deformation constraints and subspace deformation technique with a variety of constrained deformation examples.

Abstract: We describe an effective solver for the discrete Oseen problem based on an augmented Lagrangian formulation of the corresponding saddle point system The proposed method is a block triangular preconditioner used with a Krylov subspace iteration like BiCGStab The crucial ingredient is a novel multigrid approach for the (1,1) block, which extends a technique introduced by Schoberl for elasticity problems to nonsymmetric problems Our analysis indicates that this approach results in fast convergence, independent of the mesh size and largely insensitive to the viscosity We present experimental evidence for both isoP2-P0 and isoP2-P1 finite elements in support of our conclusions We also show results of a comparison with two state-of-the-art preconditioners, showing the competitiveness of our approach

Abstract: The aim of this paper is twofold First, several basic mathematical concepts involved in the construction and study of Bregman type iterative algorithms are presented from a unified analytic perspective Also, some gaps in the current knowledge about those concepts are filled in Second, we employ existing results on total convexity, sequential consistency, uniform convexity and relative projections in order to define and study the convergence of a new Bregman type iterative method of solving operator equations

Abstract: We characterize the capacity-achieving input covariance for multi-antenna channels known instantaneously at the receiver and in distribution at the transmitter. Our characterization, valid for arbitrary numbers of antennas, encompasses both the eigenvectors and the eigenvalues. The eigenvectors are found for zero-mean channels with arbitrary fading profiles and a wide range of correlation and keyhole structures. For the eigenvalues, in turn, we present necessary and sufficient conditions as well as an iterative algorithm that exhibits remarkable properties: universal applicability, robustness and rapid convergence. In addition, we identify channel structures for which an isotropic input achieves capacity.

Abstract: Dual-primal FETI methods are nonoverlapping domain decomposition methods where some of the continuity constraints across subdomain boundaries are required to hold throughout the iterations, as in primal iterative substructuring methods, while most of the constraints are enforced by Lagrange multipliers, as in one-level FETI methods. These methods are used to solve the large algebraic systems of equations that arise in elliptic finite element problems. The purpose of this article is to develop strategies for selecting these constraints, which are enforced throughout the iterations, such that good convergence bounds are obtained that are independent of even large changes in the stiffness of the subdomains across the interface between them. The algorithms are described in terms of a change of basis that has proven to be quite robust in practice. A theoretical analysis is provided for the case of linear elasticity, and condition number bounds are established that are uniform with respect to arbitrarily large jumps in the Young's modulus of the material and otherwise depend only polylogarithmically on the number of unknowns of a single subdomain. The strategies have already proven quite successful in large-scale implementations of these iterative methods. © 2006 Wiley Periodicals, Inc.

Abstract: The hybrid cellular automaton (HCA) algorithm is a methodology developed to simulate the process of structural adaptation in bones. This methodology incorporates a distributed control loop within a structure in which ideally localized sensor cells activate local processes of the formation and resorption of material. With a proper control strategy, this process drives the overall structure to an optimal configuration. The controllers developed in this investigation include two-position, proportional, integral and derivative strategies. The HCA algorithm combines elements of the cellular automaton (CA) paradigm with finite element analysis (FEA). This methodology has proved to be computationally efficient to solve topology optimization problems. The resulting optimal structures are free of numerical instabilities such as the checkerboarding effect. This investigation presents the main features of the HCA algorithm and the influence of different parameters applied during the iterative optimization process. DOI: 10.1115/1.2336251

Abstract: This paper introduces an efficient domain decomposition algorithm for the solution of time-harmonic electromagnetic fields arising in three dimensional, finite-size photonic band gap and electromagnetic band gap structures. The method is based on the finite element approximation and a nonoverlapping domain decomposition method. A set of "cement" unknowns on the inter-domain interfaces has been explicitly introduced to enforce the appropriate field continuities. The introduction of these extra unknowns allows for nonconforming/nonmatching triangulations across domain, eliminating the need for periodic mesh. To ensure and improve the convergence of the outer iteration loop, Robin transmission condition is used to communicate information across interfaces. The resulting system of equations is solved with a fast algorithm that loosely resembles the well known finite element tearing and interconnecting algorithm. In this algorithm, the method solves, in the preprocessing step, for the Robin primal subdomain problems multiple times, by exciting one dual unknown at a time. This step generates an iteration matrix that is then used to update the dual unknowns in the outer-loop iteration. The present method becomes extremely efficient for problems with geometric repetitions, such as, photonic and electromagnetic band gap structures.

Abstract: The well-known shrinkage technique is still relevant for contemporary signal processing problems over redundant dictionaries. We present theoretical and empirical analyses for two iterative algorithms for sparse approximation that use shrinkage. The GENERAL IT algorithm amounts to a Landweber iteration with nonlinear shrinkage at each iteration step. The BLOCK IT algorithm arises in morphological components analysis. A sufficient condition for which General IT exactly recovers a sparse signal is presented, in which the cumulative coherence function naturally arises. This analysis extends previous results concerning the Orthogonal Matching Pursuit (OMP) and Basis Pursuit (BP) algorithms to IT algorithms.

Abstract: This paper addresses the scan matching problem for mobile robot displacement estimation. The contribution is a new metric distance and all the tools necessary to be used within the iterative closest point framework. The metric distance is defined in the configuration space of the sensor, and takes into account both translation and rotation error of the sensor. The new scan matching technique ameliorates previous methods in terms of robustness, precision, convergence, and computational load. Furthermore, it has been extensively tested to validate and compare this technique with existing methods

Abstract: We combine the main ideas introduced in Part I with adaptive techniques to arrive at a powerful algorithm that estimates missing data in nonstationary signals. The proposed approach operates automatically based on a chosen linear transform that is expected to provide sparse decompositions over missing regions such that a portion of the transform coefficients over missing regions are zero or close to zero. Unlike prevalent algorithms, our method does not necessitate any complex preconditioning, segmentation, or edge detection steps, and it can be written as a progression of denoising operations. We show that constructing estimates based on nonlinear approximants is fundamentally a nonconvex problem and we propose a progressive algorithm that is designed to deal with this issue directly. The algorithm is applied to images through an extensive set of simulation examples, primarily on missing regions containing textures, edges, and other image features that are not readily handled by established estimation and recovery methods. We discuss the properties required of good transforms, and in conjunction, show the types of regions over which well-known transforms provide good predictors. We further discuss extensions of the algorithm where the utilized transforms are also chosen adaptively, where unpredictable signal components in the progressions are identified and not predicted, and where the prediction scenario is more general.

Abstract: Variational methods are among the most accurate techniques for estimating the optic flow. They yield dense flow fields and can be designed such that they preserve discontinuities, estimate large displacements correctly and perform well under noise and varying illumination. However, such adaptations render the minimisation of the underlying energy functional very expensive in terms of computational costs: Typically one or more large linear or nonlinear equation systems have to be solved in order to obtain the desired solution. Consequently, variational methods are considered to be too slow for real-time performance. In our paper we address this problem in two ways: (i) We present a numerical framework based on bidirectional multigrid methods for accelerating a broad class of variational optic flow methods with different constancy and smoothness assumptions. Thereby, our work focuses particularly on regularisation strategies that preserve discontinuities. (ii) We show by the examples of five classical and two recent variational techniques that real-time performance is possible in all cases¯even for very complex optic flow models that offer high accuracy. Experiments show that frame rates up to 63 dense flow fields per second for image sequences of size 160 x 120 can be achieved on a standard PC. Compared to classical iterative methods this constitutes a speedup of two to four orders of magnitude.