Abstract: We present a method for learning sparse representations shared across multiple tasks. This method is a generalization of the well-known single-task 1-norm regularization. It is based on a novel non-convex regularizer which controls the number of learned features common across the tasks. We prove that the method is equivalent to solving a convex optimization problem for which there is an iterative algorithm which converges to an optimal solution. The algorithm has a simple interpretation: it alternately performs a supervised and an unsupervised step, where in the former step it learns task-specific functions and in the latter step it learns common-across-tasks sparse representations for these functions. We also provide an extension of the algorithm which learns sparse nonlinear representations using kernels. We report experiments on simulated and real data sets which demonstrate that the proposed method can both improve the performance relative to learning each task independently and lead to a few learned features common across related tasks. Our algorithm can also be used, as a special case, to simply select--not learn--a few common variables across the tasks.

Abstract: We propose a network flow based optimization method for data association needed for multiple object tracking. The maximum-a-posteriori (MAP) data association problem is mapped into a cost-flow network with a non-overlap constraint on trajectories. The optimal data association is found by a min-cost flow algorithm in the network. The network is augmented to include an explicit occlusion model(EOM) to track with long-term inter-object occlusions. A solution to the EOM-based network is found by an iterative approach built upon the original algorithm. Initialization and termination of trajectories and potential false observations are modeled by the formulation intrinsically. The method is efficient and does not require hypotheses pruning. Performance is compared with previous results on two public pedestrian datasets to show its improvement.

Abstract: In this paper we investigate the usage of persistent point feature histograms for the problem of aligning point cloud data views into a consistent global model. Given a collection of noisy point clouds, our algorithm estimates a set of robust 16D features which describe the geometry of each point locally. By analyzing the persistence of the features at different scales, we extract an optimal set which best characterizes a given point cloud. The resulted persistent features are used in an initial alignment algorithm to estimate a rigid transformation that approximately registers the input datasets. The algorithm provides good starting points for iterative registration algorithms such as ICP (Iterative Closest Point), by transforming the datasets to its convergence basin. We show that our approach is invariant to pose and sampling density, and can cope well with noisy data coming from both indoor and outdoor laser scans.

Abstract: Finding sparse approximate solutions to large underdetermined linear systems of equations is a common problem in signal/image processing and statistics. Basis pursuit, the least absolute shrinkage and selection operator (LASSO), wavelet-based deconvolution and reconstruction, and compressed sensing (CS) are a few well-known areas in which problems of this type appear. One standard approach is to minimize an objective function that includes a quadratic (pound 2) error term added to a sparsity-inducing (usually pound 1) regularizer. We present an algorithmic framework for the more general problem of minimizing the sum of a smooth convex function and a nonsmooth, possibly nonconvex, sparsity-inducing function. We propose iterative methods in which each step is an optimization subproblem involving a separable quadratic term (diagonal Hessian) plus the original sparsity-inducing term. Our approach is suitable for cases in which this subproblem can be solved much more rapidly than the original problem. In addition to solving the standard pound 2 - pound 1 case, our approach handles other problems, e.g., pound p regularizers with p ne 1, or group-separable (GS) regularizers. Experiments with CS problems show that our approach provides state-of-the-art speed for the standard pound 2 - pound 1 problem, and is also efficient on problems with GS regularizers.

Abstract: Convergence of the value-iteration-based heuristic dynamic programming (HDP) algorithm is proven in the case of general nonlinear systems. That is, it is shown that HDP converges to the optimal control and the optimal value function that solves the Hamilton-Jacobi-Bellman equation appearing in infinite-horizon discrete-time (DT) nonlinear optimal control. It is assumed that, at each iteration, the value and action update equations can be exactly solved. The following two standard neural networks (NN) are used: a critic NN is used to approximate the value function, whereas an action network is used to approximate the optimal control policy. It is stressed that this approach allows the implementation of HDP without knowing the internal dynamics of the system. The exact solution assumption holds for some classes of nonlinear systems and, specifically, in the specific case of the DT linear quadratic regulator (LQR), where the action is linear and the value quadratic in the states and NNs have zero approximation error. It is stressed that, for the LQR, HDP may be implemented without knowing the system A matrix by using two NNs. This fact is not generally appreciated in the folklore of HDP for the DT LQR, where only one critic NN is generally used.

Abstract: The minimum lscr1-norm solution to an underdetermined system of linear equations y=Ax is often, remarkably, also the sparsest solution to that system. This sparsity-seeking property is of interest in signal processing and information transmission. However, general-purpose optimizers are much too slow for lscr1 minimization in many large-scale applications.In this paper, the Homotopy method, originally proposed by Osborne et al. and Efron et al., is applied to the underdetermined lscr1-minimization problem min parxpar1 subject to y=Ax. Homotopy is shown to run much more rapidly than general-purpose LP solvers when sufficient sparsity is present. Indeed, the method often has the following k-step solution property: if the underlying solution has only k nonzeros, the Homotopy method reaches that solution in only k iterative steps. This k-step solution property is demonstrated for several ensembles of matrices, including incoherent matrices, uniform spherical matrices, and partial orthogonal matrices. These results imply that Homotopy may be used to rapidly decode error-correcting codes in a stylized communication system with a computational budget constraint. The approach also sheds light on the evident parallelism in results on lscr1 minimization and orthogonal matching pursuit (OMP), and aids in explaining the inherent relations between Homotopy, least angle regression (LARS), OMP, and polytope faces pursuit.

Abstract: The paper proposes a method for constructing a sparse estimator for the inverse covariance (concentration) matrix in high-dimensional settings. The estimator uses a penalized normal likelihood approach and forces sparsity by using a lasso-type penalty. We establish a rate of convergence in the Frobenius norm as both data dimension $p$ and sample size $n$ are allowed to grow, and show that the rate depends explicitly on how sparse the true concentration matrix is. We also show that a correlation-based version of the method exhibits better rates in the operator norm. We also derive a fast iterative algorithm for computing the estimator, which relies on the popular Cholesky decomposition of the inverse but produces a permutation-invariant estimator. The method is compared to other estimators on simulated data and on a real data example of tumor tissue classification using gene expression data.

Abstract: In this work we propose the use of a modified version of the correlation coefficient as a performance criterion for the image alignment problem. The proposed modification has the desirable characteristic of being invariant with respect to photometric distortions. Since the resulting similarity measure is a nonlinear function of the warp parameters, we develop two iterative schemes for its maximization, one based on the forward additive approach and the second on the inverse compositional method. As it is customary in iterative optimization, in each iteration the nonlinear objective function is approximated by an alternative expression for which the corresponding optimization is simple. In our case we propose an efficient approximation that leads to a closed form solution (per iteration) which is of low computational complexity, the latter property being particularly strong in our inverse version. The proposed schemes are tested against the forward additive Lucas-Kanade and the simultaneous inverse compositional algorithm through simulations. Under noisy conditions and photometric distortions our forward version achieves more accurate alignments and exhibits faster convergence whereas our inverse version has similar performance as the simultaneous inverse compositional algorithm but at a lower computational complexity.

Abstract: This paper describes PLICP, an ICP (iterative closest/corresponding point) variant that uses a point-to-line metric, and an exact closed-form for minimizing such metric. The resulting algorithm has some interesting properties: it converges quadratically, and in a finite number of steps. The method is validated against vanilla ICP, IDC (iterative dual correspondences), and MBICP (Metric-Based ICP) by reproducing the experiments performed in Minguez et al. (2006). The experiments suggest that PLICP is more precise, and requires less iterations. However, it is less robust to very large initial displacement errors. The last part of the paper is devoted to purely algorithmic optimization of the correspondence search; this allows for a significant speed-up of the computation. The source code is available for download.

Abstract: This paper presents a novel iterative greedy reconstruction algorithm for practical compressed sensing (CS), called the sparsity adaptive matching pursuit (SAMP). Compared with other state-of-the-art greedy algorithms, the most innovative feature of the SAMP is its capability of signal reconstruction without prior information of the sparsity. This makes it a promising candidate for many practical applications when the number of non-zero (significant) coefficients of a signal is not available. The proposed algorithm adopts a similar flavor of the EM algorithm, which alternatively estimates the sparsity and the true support set of the target signals. In fact, SAMP provides a generalized greedy reconstruction framework in which the orthogonal matching pursuit and the subspace pursuit can be viewed as its special cases. Such a connection also gives us an intuitive justification of trade-offs between computational complexity and reconstruction performance. While the SAMP offers a comparably theoretical guarantees as the best optimization-based approach, simulation results show that it outperforms many existing iterative algorithms, especially for compressible signals.

Abstract: We consider distributed iterative algorithms for the averaging problem over time-varying topologies. Our focus is on the convergence time of such algorithms when complete (unquantized) information is available, and on the degradation of performance when only quantized information is available. We study a large and natural class of averaging algorithms, which includes the vast majority of algorithms proposed to date, and provide tight polynomial bounds on their convergence time. We also describe an algorithm within this class whose convergence time is the best among currently available averaging algorithms for time-varying topologies. We then propose and analyze distributed averaging algorithms under the additional constraint that agents can only store and communicate quantized information, so that they can only converge to the average of the initial values of the agents within some error. We establish bounds on the error and tight bounds on the convergence time, as a function of the number of quantization levels.

Abstract: In this paper, we aim to solve the infinite-time optimal tracking control problem for a class of discrete-time nonlinear systems using the greedy heuristic dynamic programming (HDP) iteration algorithm. A new type of performance index is defined because the existing performance indexes are very difficult in solving this kind of tracking problem, if not impossible. Via system transformation, the optimal tracking problem is transformed into an optimal regulation problem, and then, the greedy HDP iteration algorithm is introduced to deal with the regulation problem with rigorous convergence analysis. Three neural networks are used to approximate the performance index, compute the optimal control policy, and model the nonlinear system for facilitating the implementation of the greedy HDP iteration algorithm. An example is given to demonstrate the validity of the proposed optimal tracking control scheme.

Abstract: Recently, a new adaptive scheme [Conte (1995), Gini (1997)] has been introduced for covariance structure matrix estimation in the context of adaptive radar detection under non-Gaussian noise. This latter has been modeled by compound-Gaussian noise, which is the product c of the square root of a positive unknown variable tau (deterministic or random) and an independent Gaussian vector x, c=radictaux. Because of the implicit algebraic structure of the equation to solve, we called the corresponding solution, the fixed point (FP) estimate. When tau is assumed deterministic and unknown, the FP is the exact maximum-likelihood (ML) estimate of the noise covariance structure, while when tau is a positive random variable, the FP is an approximate maximum likelihood (AML). This estimate has been already used for its excellent statistical properties without proofs of its existence and uniqueness. The major contribution of this paper is to fill these gaps. Our derivation is based on some likelihood functions general properties like homogeneity and can be easily adapted to other recursive contexts. Moreover, the corresponding iterative algorithm used for the FP estimate practical determination is also analyzed and we show the convergence of this recursive scheme, ensured whatever the initialization.

Abstract: We introduce an iterative method for finding a common element of the set of solutions of a generalized equilibrium problem and the set of fixed points of a nonexpansive mapping in a Hilbert space and then obtain that the sequence converges strongly to a common element of two sets. Using this result, we prove three new strong convergence theorems in fixed point problems, variational inequalities and equilibrium problems.

Abstract: In this paper we consider iterative methods for stochastic variational inequalities (s.v.i.) with monotone operators. Our basic assumption is that the operator possesses both smooth and nonsmooth components. Further, only noisy observations of the problem data are available. We develop a novel Stochastic Mirror-Prox (SMP) algorithm for solving s.v.i. and show that with the convenient stepsize strategy it attains the optimal rates of convergence with respect to the problem parameters. We apply the SMP algorithm to Stochastic composite minimization and describe particular applications to Stochastic Semidefinite Feasability problem and Eigenvalue minimization.

Abstract: Decomposition Frameworks.- Schwarz Iterative Algorithms.- Schur Complement and Iterative Substructuring Algorithms.- Lagrange Multiplier Based Substructuring: FETI Method.- Computational Issues and Parallelization.- Least Squares-Control Theory: Iterative Algorithms.- Multilevel and Local Grid Refinement Methods.- Non-Self Adjoint Elliptic Equations: Iterative Methods.- Parabolic Equations.- Saddle Point Problems.- Non-Matching Grid Discretizations.- Heterogeneous Domain Decomposition Methods.- Fictitious Domain and Domain Imbedding Methods.- Variational Inequalities and Obstacle Problems.- Maximum Norm Theory.- Eigenvalue Problems.- Optimization Problems.- Helmholtz Scattering Problem.

Abstract: Using an efficient iterative method, we have developed a distance-dependent knowledge-based scoring function to predict protein-protein interactions. The function, referred to as ITScore-PP, was derived using the crystal structures of a training set of 851 protein-protein dimeric complexes containing true biological interfaces. The key idea of the iterative method for deriving ITScore-PP is to improve the interatomic pair potentials by iteration, until the pair potentials can distinguish true binding modes from decoy modes for the protein-protein complexes in the training set. The iterative method circumvents the challenging reference state problem in deriving knowledge-based potentials. The derived scoring function was used to evaluate the ligand orientations generated by ZDOCK 2.1 and the native ligand structures on a diverse set of 91 protein-protein complexes. For the bound test cases, ITScore-PP yielded a success rate of 98.9% if the top 10 ranked orientations were considered. For the more realistic unbound test cases, the corresponding success rate was 40.7%. Furthermore, for faster orientational sampling purpose, several residue-level knowledge-based scoring functions were also derived following the similar iterative procedure. Among them, the scoring function that uses the side-chain center of mass (SCM) to represent a residue, referred to as ITScore-PP(SCM), showed the best performance and yielded success rates of 71.4% and 30.8% for the bound and unbound cases, respectively, when the top 10 orientations were considered. ITScore-PP was further tested using two other published protein-protein docking decoy sets, the ZDOCK decoy set and the RosettaDock decoy set. In addition to binding mode prediction, the binding scores predicted by ITScore-PP also correlated well with the experimentally determined binding affinities, yielding a correlation coefficient of R = 0.71 on a test set of 74 protein-protein complexes with known affinities. ITScore-PP is computationally efficient. The average run time for ITScore-PP was about 0.03 second per orientation (including optimization) on a personal computer with 3.2 GHz Pentium IV CPU and 3.0 GB RAM. The computational speed of ITScore-PP(SCM) is about an order of magnitude faster than that of ITScore-PP. ITScore-PP and/or ITScore-PP(SCM) can be combined with efficient protein docking software to study protein-protein recognition.

Abstract: This paper deals with an iterative method, in a real Hilbert space, for approximating a common element of the set of fixed points of a demicontractive operator (possibly quasi-nonexpansive or strictly pseudocontractive) and the set of solutions of a variational inequality problem for a monotone, Lipschitz continuous mapping. The considered algorithm can be regarded as a combination of a variation of the hybrid steepest descent method and the so-called extragradient method. Under classical conditions, we prove the strong convergence of the sequences of iterates given by the considered scheme.

Abstract: We present IDR($s$), a new family of efficient, short-recurrence methods for large nonsymmetric systems of linear equations. The new methods are based on the induced dimension reduction (IDR) method proposed by Sonneveld in 1980. IDR($s$) generates residuals that are forced to be in a sequence of nested subspaces. Although IDR($s$) behaves like an iterative method, in exact arithmetic it computes the true solution using at most $N + N/s$ matrix-vector products, with $N$ the problem size and $s$ the codimension of a fixed subspace. We describe the algorithm and the underlying theory and present numerical experiments to illustrate the theoretical properties of the method and its performance for systems arising from different applications. Our experiments show that IDR($s$) is competitive with or superior to most Bi-CG-based methods and outperforms Bi-CGSTAB when $s > 1$.

Abstract: This paper proposes a new perspective for solving systems of complex nonlinear equations by simply viewing them as a multiobjective optimization problem. Every equation in the system represents an objective function whose goal is to minimize the difference between the right and left terms of the corresponding equation. An evolutionary computation technique is applied to solve the problem obtained by transforming the system into a multiobjective optimization problem. The results obtained are compared with a very new technique that is considered as efficient and is also compared with some of the standard techniques that are used for solving nonlinear equations systems. Several well-known and difficult applications (such as interval arithmetic benchmark, kinematic application, neuropsychology application, combustion application, and chemical equilibrium application) are considered for testing the performance of the new approach. Empirical results reveal that the proposed approach is able to deal with high-dimensional equations systems.

Abstract: This paper contributes two novel techniques in the context of image restoration by nonlocal filtering. First, we introduce an efficient implementation of the nonlocal means filter based on arranging the data in a cluster tree. The structuring of data allows for a fast and accurate preselection of similar patches. In contrast to previous approaches, the preselection is based on the same distance measure as used by the filter itself. It allows for large speedups, especially when the search for similar patches covers the whole image domain, i.e., when the filter is truly nonlocal. However, also in the windowed version of the filter, the cluster tree approach compares favorably to previous techniques in respect of quality versus computational cost. Second, we suggest an iterative version of the filter that is derived from a variational principle and is designed to yield nontrivial steady states. It reveals to be particularly useful in order to restore regular, textured patterns.

Abstract: We introduce a new class of exponential integrators for the numerical integration of large-scale systems of stiff differential equations. These so-called Rosenbrock-type methods linearize the flow in each time step and make use of the matrix exponential and related functions of the Jacobian. In contrast to standard integrators, the methods are fully explicit and do not require the numerical solution of linear systems. We analyze the convergence properties of these integrators in a semigroup framework of semilinear evolution equations in Banach spaces. In particular, we derive an abstract stability and convergence result for variable step sizes. This analysis further provides the required order conditions and thus allows us to construct pairs of embedded methods. We present a third-order method with two stages, and a fourth-order method with three stages, respectively. The application of the required matrix functions to vectors are computed by Krylov subspace approximations. We briefly discuss these implementation issues, and we give numerical examples that demonstrate the efficiency of the new integrators.

Abstract: We describe a numerically efficient strategy for solving a linear system of equations arising in the Method of Moments for solving electromagnetic scattering problems. This novel approach, termed as the characteristic basis function method (CBFM), is based on utilizing characteristic basis functions (CBFs)-special functions defined on macro domains (blocks)-that include a relatively large number of conventional sub-domains discretized by using triangular or rectangular patches. Use of these basis functions leads to a significant reduction in the number of unknowns, and results in a substantial size reduction of the MoM matrix; this, in turn, enables us to handle the reduced matrix by using a direct solver, without the need to iterate. In addition, the paper shows that the CBFs can be generated by using a sparse representation of the impedance matrix-resulting in lower computational cost-and that, in contrast to the iterative techniques, multiple excitations can be handled with only a small overhead. Another important attribute of the CBFM is that it is readily parallelized. Numerical results that demonstrate the accuracy and time efficiency of the CBFM for several representative scattering problems are included in the paper.

Abstract: We study preconditioners for the iterative solution of the linear systems arising in the implicit time integration of the compressible Navier-Stokes equations. The spatial discretization is carried out using a discontinuous Galerkin method with fourth order polynomial interpolations on triangular elements. The time integration is based on backward difference formulas resulting in a nonlinear system of equations which is solved at each timestep. This is accomplished using Newton's method. The resulting linear systems are solved using a preconditioned GMRES iterative algorithm. We consider several existing preconditioners such as block Jacobi and Gauss-Seidel combined with multilevel schemes which have been developed and tested for specific applications. While our results are consistent with the claims reported, we find that these preconditioners lack robustness when used in more challenging situations involving low Mach numbers, stretched grids, or high Reynolds number turbulent flows. We propose a preconditioner based on a coarse scale correction with postsmoothing based on a block incomplete LU factorization with zero fill-in (ILU0) of the Jacobian matrix. The performance of the ILU0 smoother is found to depend critically on the element numbering. We propose a numbering strategy based on minimizing the discarded fill-in in a greedy fashion. The coarse scale correction scheme is found to be important for diffusion dominated problems, whereas the ILU0 preconditioner with the proposed ordering is effective at handling the convection dominated case. While little can be said in the way of theoretical results, the proposed preconditioner is shown to perform remarkably well for a broad range of representative test problems. These include compressible flows ranging from very low Reynolds numbers to fully turbulent flows using the Reynolds averaged Navier-Stokes equations discretized on highly stretched grids. For low Mach number flows, the proposed preconditioner is more than one order of magnitude more efficient than the other preconditioners considered.

Abstract: This paper considers the minimization of transmit power in Gaussian parallel interference channels, subject to a rate constraint for each user. To derive decentralized solutions that do not require any cooperation among the users, we formulate this power control problem as a (generalized) Nash equilibrium (NE) game. We obtain sufficient conditions that guarantee the existence and nonemptiness of the solution set to our problem. Then, to compute the solutions of the game, we propose two distributed algorithms based on the single user water-filling solution: The sequential and the simultaneous iterative water-filling algorithms, wherein the users update their own strategies sequentially and simultaneously, respectively. We derive a unified set of sufficient conditions that guarantee the uniqueness of the solution and global convergence of both algorithms. Our results are applicable to all practical distributed multipoint-to-multipoint interference systems, either wired or wireless, where a quality of service in (QoS) terms of information rate must be guaranteed for each link.

Abstract: This paper proposes a novel two-stage optimization method for robust model predictive control (RMPC) with Gaussian disturbance and state estimation error. Since the disturbance is unbounded, it is impossible to achieve zero probability of violating constraints. Our goal is to optimize the expected value of an objective function while limiting the probability of violating any constraints over the planning horizon (joint chance constraint). Prior arts include ellipsoidal relaxation approach [1] and particle control [2], but the former yields very conservative result and the latter is computationally intensive. Our approach divide the optimization problem into two stages; the upper-stage that optimizes risk allocation, and the lower-stage that optimizes control sequence with tightened constraints. The lower-stage is a regular convex optimization, such as linear programming or quadratic programming. The upper-stage is also convex, but the objective function is not always differentiable. We developed a fast descent algorithm for the upper-stage called iterative risk allocation (IRA), which yield much smaller suboptimality than ellipsoidal relaxation method while achieving a substantial speedup compared to and particle control.