Abstract: Accelerating the data acquisition of dynamic magnetic resonance imaging leads to a challenging ill-posed inverse problem, which has received great interest from both the signal processing and machine learning communities over the last decades. The key ingredient to the problem is how to exploit the temporal correlations of the MR sequence to resolve aliasing artifacts. Traditionally, such observation led to a formulation of an optimization problem, which was solved using iterative algorithms. Recently, however, deep learning-based approaches have gained significant popularity due to their ability to solve general inverse problems. In this paper, we propose a unique, novel convolutional recurrent neural network architecture which reconstructs high quality cardiac MR images from highly undersampled k-space data by jointly exploiting the dependencies of the temporal sequences as well as the iterative nature of the traditional optimization algorithms. In particular, the proposed architecture embeds the structure of the traditional iterative algorithms, efficiently modeling the recurrence of the iterative reconstruction stages by using recurrent hidden connections over such iterations. In addition, spatio–temporal dependencies are simultaneously learnt by exploiting bidirectional recurrent hidden connections across time sequences. The proposed method is able to learn both the temporal dependence and the iterative reconstruction process effectively with only a very small number of parameters, while outperforming current MR reconstruction methods in terms of reconstruction accuracy and speed.

Abstract: Point cloud registration is a key problem for computer vision applied to robotics, medical imaging, and other applications. This problem involves finding a rigid transformation from one point cloud into another so that they align. Iterative Closest Point (ICP) and its variants provide simple and easily-implemented iterative methods for this task, but these algorithms can converge to spurious local optima. To address local optima and other difficulties in the ICP pipeline, we propose a learning-based method, titled Deep Closest Point (DCP), inspired by recent techniques in computer vision and natural language processing. Our model consists of three parts: a point cloud embedding network, an attention-based module combined with a pointer generation layer, to approximate combinatorial matching, and a differentiable singular value decomposition (SVD) layer to extract the final rigid transformation. We train our model end-to-end on the ModelNet40 dataset and show in several settings that it performs better than ICP, its variants (e.g., Go-ICP, FGR), and the recently-proposed learning-based method PointNetLK. Beyond providing a state-of-the-art registration technique, we evaluate the suitability of our learned features transferred to unseen objects. We also provide preliminary analysis of our learned model to help understand whether domain-specific and/or global features facilitate rigid registration.

Abstract: Substantial progress has been made recently on developing provably accurate and efficient algorithms for low-rank matrix factorization via nonconvex optimization. While conventional wisdom often takes a dim view of nonconvex optimization algorithms due to their susceptibility to spurious local minima, simple iterative methods such as gradient descent have been remarkably successful in practice. The theoretical footings, however, had been largely lacking until recently. In this tutorial-style overview, we highlight the important role of statistical models in enabling efficient nonconvex optimization with performance guarantees. We review two contrasting approaches: (1) two-stage algorithms, which consist of a tailored initialization step followed by successive refinement; and (2) global landscape analysis and initialization-free algorithms. Several canonical matrix factorization problems are discussed, including but not limited to matrix sensing, phase retrieval, matrix completion, blind deconvolution, and robust principal component analysis. Special care is taken to illustrate the key technical insights underlying their analyses. This article serves as a testament that the integrated consideration of optimization and statistics leads to fruitful research findings.

Abstract: This paper is devoted to solving a full-wave inverse scattering problem (ISP), which is aimed at retrieving permittivities of dielectric scatterers from the knowledge of measured scattering data. ISPs are highly nonlinear due to multiple scattering, and iterative algorithms with regularizations are often used to solve such problems. However, they are associated with heavy computational cost, and consequently, they are often time-consuming. This paper proposes the convolutional neural network (CNN) technique to solve full-wave ISPs. We introduce and compare three training schemes based on U-Net CNN, including direct inversion, backpropagation, and dominant current schemes (DCS). Several representative tests are carried out, including both synthetic and experimental data, to evaluate the performances of the proposed methods. It is demonstrated that the proposed DCS outperforms the other two schemes in terms of accuracy and is able to solve typical ISPs quickly within 1 s. The proposed deep-learning inversion scheme is promising in providing quantitative images in real time.

Abstract: Nonlinear electromagnetic (EM) inverse scattering is a quantitative and super-resolution imaging technique, in which more realistic interactions between the internal structure of scene and EM wavefield are taken into account in the imaging procedure, in contrast to conventional tomography. However, it poses important challenges arising from its intrinsic strong nonlinearity, ill-posedness, and expensive computational costs. To tackle these difficulties, we, for the first time to our best knowledge, exploit a connection between the deep neural network (DNN) architecture and the iterative method of nonlinear EM inverse scattering. This enables the development of a novel DNN-based methodology for nonlinear EM inverse problems (termed here DeepNIS). The proposed DeepNIS consists of a cascade of multilayer complex-valued residual convolutional neural network modules. We numerically and experimentally demonstrate that the DeepNIS outperforms remarkably conventional nonlinear inverse scattering methods in terms of both the image quality and computational time. We show that DeepNIS can learn a general model approximating the underlying EM inverse scattering system. It is expected that the DeepNIS will serve as powerful tool in treating highly nonlinear EM inverse scattering problems over different frequency bands, which are extremely hard and impractical to solve using conventional inverse scattering methods.

Abstract: In this paper, to bridge the gap between physical knowledge and learning approaches, we propose an induced current learning method (ICLM) by incorporating merits in traditional iterative algorithms into the architecture of convolutional neural network (CNN). The main contributions of the proposed method are threefold. First, to the best of our knowledge, it is the first time that the contrast source is learned to solve full-wave inverse scattering problems (ISPs). Second, inspired by the basis-expansion strategy in the traditional iterative approach for solving ISPs, a combined loss function with multiple labels is defined in a cascaded end-to-end CNN (CEE-CNN) architecture to decrease the nonlinearity of objective function, where no additional computational cost is introduced in generating extra labels. Third, to accelerate the convergence speed and decrease the difficulties of the learning process, the proposed CEE-CNN is designed to focus on learning the minor part of the induced current by introducing several skip connections and to bypass the major part of induced current to the output layers. The proposed method is compared with the state-of-the-art of deep learning scheme and a well-known iterative ISP solver, where numerical and experimental tests are conducted to verify the proposed ICLM.

Abstract: A cell-free massive multiple-input multiple-output system is considered using a max-min approach to maximize the minimum user rate with per-user power constraints. First, an approximated uplink user rate is derived based on channel statistics. Then, the original max-min signal-to-interference-plus-noise ratio problem is formulated for the optimization of receiver filter coefficients at a central processing unit and user power allocation. To solve this max-min non-convex problem, we decouple the original problem into two sub-problems, namely, receiver filter coefficient design and power allocation. The receiver filter coefficient design is formulated as a generalized Eigenvalue problem, whereas the geometric programming (GP) is used to solve the user power allocation problem. Based on these two sub-problems, an iterative algorithm is proposed, in which both problems are alternately solved while one of the design variables is fixed. This iterative algorithm obtains a globally optimum solution, whose optimality is proved through establishing an uplink-downlink duality. Moreover, we present a novel sub-optimal scheme which provides a GP formulation to efficiently and globally maximize the minimum uplink user rate. The numerical results demonstrate that the proposed scheme substantially outperforms the existing schemes in the literature.

Abstract: Plug-and-play priors (PnP) is a powerful framework for regularizing imaging inverse problems by using advanced denoisers within an iterative algorithm. Recent experimental evidence suggests that PnP algorithms achieve the state-of-the-art performance in a range of imaging applications. In this paper, we introduce a new online PnP algorithm based on the proximal gradient method (PGM). The proposed algorithm uses only a subset of measurements at every iteration, which makes it scalable to very large datasets. We present a new theoretical convergence analysis, for both batch and online variants of PnP-PGM, for denoisers that do not necessarily correspond to proximal operators. We also present simulations illustrating the applicability of the algorithm to image reconstruction in diffraction tomography. The results in this paper have the potential to expand the applicability of the PnP framework to very large datasets.

Abstract: Near-eye displays using holographic projection are emerging as an exciting display approach for virtual and augmented reality at high-resolution without complex optical setups --- shifting optical complexity to computation. While precise phase modulation hardware is becoming available, phase retrieval algorithms are still in their infancy, and holographic display approaches resort to heuristic encoding methods or iterative methods relying on various relaxations. In this work, we depart from such existing approximations and solve the phase retrieval problem for a hologram of a scene at a single depth at a given time by revisiting complex Wirtinger derivatives, also extending our framework to render 3D volumetric scenes. Using Wirtinger derivatives allows us to pose the phase retrieval problem as a quadratic problem which can be minimized with first-order optimization methods. The proposed Wirtinger Holography is flexible and facilitates the use of different loss functions, including learned perceptual losses parametrized by deep neural networks, as well as stochastic optimization methods. We validate this framework by demonstrating holographic reconstructions with an order of magnitude lower error, both in simulation and on an experimental hardware prototype.

Abstract: This study considers the parameter estimation of a multi-variable output-error-like system with autoregressive moving average noise. In order to solve the problem of the information vector containing unknown variables, a least squares-based iterative algorithm is proposed by using the iterative search. The original system is divided into several subsystems by using the decomposition technique. However, the subsystems contain the same parameter vector, which poses a challenge for the identification problem, the approach taken here is to use the coupling identification concept to cut down the redundant parameter estimates. In addition, the recursive least squares algorithm is provided for comparison. The simulation results indicate that the proposed algorithms are effective.

Abstract: This paper describes a new MATLAB software package of iterative regularization methods and test problems for large-scale linear inverse problems. The software package, called IR TOOLS, serves two related purposes: we provide implementations of a range of iterative solvers, including several recently proposed methods that are not available elsewhere, and we provide a set of large-scale test problems in the form of discretizations of 2D linear inverse problems. The solvers include iterative regularization methods where the regularization is due to the semi-convergence of the iterations, Tikhonov-type formulations where the regularization is explicitly formulated in the form of a regularization term, and methods that can impose bound constraints on the computed solutions. All the iterative methods are implemented in a very flexible fashion that allows the problem’s coefficient matrix to be available as a (sparse) matrix, a function handle, or an object. The most basic call to all of the various iterative methods requires only this matrix and the right hand side vector; if the method uses any special stopping criteria, regularization parameters, etc., then default values are set automatically by the code. Moreover, through the use of an optional input structure, the user can also have full control of any of the algorithm parameters. The test problems represent realistic large-scale problems found in image reconstruction and several other applications. Numerical examples illustrate the various algorithms and test problems available in this package.

Abstract: Objective: Deep learning has recently been applied to electrical impedance tomography (EIT) imaging. Nevertheless, there are still many challenges that this approach has to face, e.g., targets with sharp corners or edges cannot be well recovered when using circular inclusion training data. This paper proposes an iterative-based inversion method and a convolutional neural network (CNN) based inversion method to recover some challenging inclusions such as triangular, rectangular, or lung shapes, where the CNN-based method uses only random circle or ellipse training data. Methods: First, the iterative method, i.e., bases-expansion subspace optimization method (BE-SOM), is proposed based on a concept of induced contrast current (ICC) with total variation regularization. Second, the theoretical analysis of BE-SOM and the physical concepts introduced there motivate us to propose a dominant-current deep learning scheme for EIT imaging, in which dominant parts of ICC are utilized to generate multi-channel inputs of CNN. Results: The proposed methods are tested with both numerical and experimental data, where several realistic phantoms including simulated pneumothorax and pleural effusion pathologies are also considered. Conclusions and Significance: Significant performance improvements of the proposed methods are shown in reconstructing targets with sharp corners or edges. It is also demonstrated that the proposed methods are capable of fast, stable, and high-quality EIT imaging, which is promising in providing quantitative images for potential clinical applications.

Abstract: The field of medical image reconstruction has seen roughly four types of methods. The first type tended to be analytical methods, such as filtered back-projection (FBP) for X-ray computed tomography (CT) and the inverse Fourier transform for magnetic resonance imaging (MRI), based on simple mathematical models for the imaging systems. These methods are typically fast, but have suboptimal properties such as poor resolution-noise trade-off for CT. A second type is iterative reconstruction methods based on more complete models for the imaging system physics and, where appropriate, models for the sensor statistics. These iterative methods improved image quality by reducing noise and artifacts. The FDA-approved methods among these have been based on relatively simple regularization models. A third type of methods has been designed to accommodate modified data acquisition methods, such as reduced sampling in MRI and CT to reduce scan time or radiation dose. These methods typically involve mathematical image models involving assumptions such as sparsity or low-rank. A fourth type of methods replaces mathematically designed models of signals and systems with data-driven or adaptive models inspired by the field of machine learning. This paper focuses on the two most recent trends in medical image reconstruction: methods based on sparsity or low-rank models, and data-driven methods based on machine learning techniques.

Abstract: Fast and accurate frequency estimation of a complex sinusoidal plays an important role in many applications, including communications, radar, and sonar. This paper presents two methods for the frequency estimation of a complex sinusoidal in noisy environments. The proposed methods interpolate on shifted DFT coefficients to acquire the frequency estimates iteratively. The first method interpolates on the $q$ -shifted DFT coefficients of the signal, whose optimum iteration number is found to be a logarithmic function of the signal length. Furthermore, we also show that by appropriately selecting the value of the shift $q$ , the proposed method becomes asymptotically efficient. In the second method, we use hybrid half-shifted and $q$ -shifted DFT interpolators together, which converges in only two iterations. It is shown that both of the estimators are asymptotically unbiased with their mean square errors performing on the Cramer–Rao lower bound. Theoretical results are validated by extensive computer simulations.

Abstract: Optimization problems in disciplines such as machine learning are commonly solved with iterative methods. Gradient descent algorithms find local minima by moving along the direction of steepest descent while Newton's method takes into account curvature information and thereby often improves convergence. Here, we develop quantum versions of these iterative optimization algorithms and apply them to polynomial optimization with a unit norm constraint. In each step, multiple copies of the current candidate are used to improve the candidate using quantum phase estimation, an adapted quantum state exponentiation scheme, as well as quantum matrix multiplications and inversions. The required operations perform polylogarithmically in the dimension of the solution vector and exponentially in the number of iterations. Therefore, the quantum algorithm can be useful for high-dimensional problems where a small number of iterations is sufficient.

Abstract: Two novel nonlinearly activated recurrent neural networks (RNNs) with finite-time convergence [called finite-time RNNs (FTRNNs)] are proposed and analyzed to solve efficiently time-varying systems of nonlinear equations (SoNEs). Compared with previously presented neural networks for solving such a SoNE, the FTRNNs are activated by new nonlinear activation functions and thus possess a better finite-time convergence property. In addition, theoretical analyses about FTRNNs are presented to determine the upper bounds of convergence time under the context of using such two novel nonlinear activation functions. Computer simulations based on a numerical example validate the preponderance of the proposed FTRNNs for time-varying SoNE, as compared to the recently proposed Zhang neural network and its improved version. Finally, an engineering practical example to motion tracking of a robot manipulator demonstrates the feasibility and applicability of the FTRNNs.

Abstract: A cell-free Massive multiple-input multiple-output (MIMO) uplink is considered, where the access points (APs) are connected to a central processing unit (CPU) through limited-capacity wireless microwave links. The quantized version of the weighted signals are available at the CPU, by exploiting the Bussgang decomposition to model the effect of quantization. A closed-form expression for spectral efficiency is derived taking into account the effects of channel estimation error and quantization distortion. The energy efficiency maximization problem is considered with per-user power, backhaul capacity and throughput requirement constraints. To solve this non-convex problem, we decouple the original problem into two sub-problems, namely, receiver filter coefficient design, and power allocation. The receiver filter coefficient design is formulated as a generalized eigenvalue problem whereas a successive convex approximation (SCA) and a heuristic sub-optimal scheme are exploited to convert the power allocation problem into a standard geometric programming (GP) problem. An iterative algorithm is proposed to alternately solve each sub-problem. Complexity analysis and convergence of the proposed schemes are investigated. Numerical results indicate the superiority of the proposed algorithms over the case of equal power allocation.

Abstract: Of late, there are many studies on the robust discriminant analysis, which adopt L1-norm as the distance metric, but their results are not robust enough to gain universal acceptance. To overcome this problem, the authors of this article present a nonpeaked discriminant analysis (NPDA) technique, in which cutting L1-norm is adopted as the distance metric. As this kind of norm can better eliminate heavy outliers in learning models, the proposed algorithm is expected to be stronger in performing feature extraction tasks for data representation than the existing robust discriminant analysis techniques, which are based on the L1-norm distance metric. The authors also present a comprehensive analysis to show that cutting L1-norm distance can be computed equally well, using the difference between two special convex functions. Against this background, an efficient iterative algorithm is designed for the optimization of the proposed objective. Theoretical proofs on the convergence of the algorithm are also presented. Theoretical insights and effectiveness of the proposed method are validated by experimental tests on several real data sets.

Abstract: In this paper, based on the projection strategy, we propose a derivative-free iterative method for large-scale nonlinear monotone equations with convex constraints, which can generate a sufficient descent direction at each iteration. Due to its lower storage and derivative-free information, the proposed method can be used to solve large-scale non-smooth problems. The global convergence of the proposed method is proved under the Lipschitz continuity assumption. Moreover, if the local error bound condition holds, the proposed method is shown to be linearly convergent. Preliminary numerical comparison shows that the proposed method is efficient and promising.

Abstract: We propose a new approach to image fusion, inspired by the recent plug-and-play (PnP) framework In PnP, a denoiser is treated as a black box and plugged into an iterative algorithm, taking the place of the proximity operator of some convex regularizer, which is formally equivalent to a denoising operation This approach offers flexibility and excellent performance, but convergence may be hard to analyze, as most state-of-the-art denoisers lack an explicit underlying objective function Here, we propose using a scene-adapted denoiser (ie, targeted to the specific scene being imaged) plugged into the iterations of the alternating direction method of multipliers This approach, which is a natural choice for image fusion problems, not only yields state-of-the-art results but it also allows proving convergence of the resulting algorithm The proposed method is tested on two different problems: hyperspectral fusion/sharpening and fusion of blurred-noisy image pairs

Abstract: In this paper, we consider the parameter estimation problem of dual-frequency signals disturbed by stochastic noise. The signal model is a highly nonlinear function with respect to the frequencies and phases, and the gradient method cannot obtain the accurate parameter estimates. Based on the Newton search, we derive an iterative algorithm for estimating all parameters, including the unknown amplitudes, frequencies, and phases. Furthermore, by using the parameter decomposition, a hierarchical least squares and gradient-based iterative algorithm is proposed for improving the computational efficiency. A gradient-based iterative algorithm is given for comparisons. The numerical examples are provided to demonstrate the validity of the proposed algorithms.

Abstract: This paper studies a joint pricing, replenishment and preservation technology investment problem for non-instantaneous deteriorating items. Preservation technology affects both the length of non-deterioration period and deterioration rate. Shortages are allowed and partially backlogged. We use price-dependent demand, time-varying deterioration and waiting-time-dependent backlog rates in a general framework to formulate the model. We consider two cases: shortages happen after or before the non-deterioration period. We analytically show the existence and uniqueness of the optimal replenishment schedule, price or preservation investment for any given two of them in two cases. We also prove that there exists a global replenishment policy for any given pricing and preservation investment policies. We then provide an iterative algorithm to search for the optimal solution. Finally, we use numerical examples to illustrate the algorithm, and conduct sensitivity analysis to derive more managerial insights.

Abstract: Low-rank modeling has many important applications in computer vision and machine learning. While the matrix rank is often approximated by the convex nuclear norm, the use of nonconvex low-rank regularizers has demonstrated better empirical performance. However, the resulting optimization problem is much more challenging. Recent state-of-the-art requires an expensive full SVD in each iteration. In this paper, we show that for many commonly-used nonconvex low-rank regularizers, the singular values obtained from the proximal operator can be automatically threshold. This allows the proximal operator to be efficiently approximated by the power method. We then develop a fast proximal algorithm and its accelerated variant with inexact proximal step. It can be guaranteed that the squared distance between consecutive iterates converges at a rate of $O(1/T)$O(1/T), where $T$T is the number of iterations. Furthermore, we show the proposed algorithm can be parallelized, and the resultant algorithm achieves nearly linear speedup w.r.t. the number of threads. Extensive experiments are performed on matrix completion and robust principal component analysis. Significant speedup over the state-of-the-art is observed.