Abstract: This letter describes algorithms for nonnegative matrix factorization (NMF) with the β-divergence (β-NMF). The β-divergence is a family of cost functions parameterized by a single shape parameter β that takes the Euclidean distance, the Kullback-Leibler divergence, and the Itakura-Saito divergence as special cases (β = 2, 1, 0 respectively). The proposed algorithms are based on a surrogate auxiliary function (a local majorization of the criterion function). We first describe a majorization-minimization algorithm that leads to multiplicative updates, which differ from standard heuristic multiplicative updates by a β-dependent power exponent. The monotonicity of the heuristic algorithm can, however, be proven for β ∈ (0, 1) using the proposed auxiliary function. Then we introduce the concept of the majorization-equalization (ME) algorithm, which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than MM. Simulations on synthetic and real data illustrate the faster convergence of the ME approach. The letter also describes how the proposed algorithms can be adapted to two common variants of NMF: penalized NMF (when a penalty function of the factors is added to the criterion function) and convex NMF (when the dictionary is assumed to belong to a known subspace).

Abstract: A number of variable selection methods have been proposed involving nonconvex penalty functions. These methods, which include the smoothly clipped absolute deviation (SCAD) penalty and the minimax concave penalty (MCP), have been demonstrated to have attractive theoretical properties, but model fitting is not a straightforward task, and the resulting solutions may be unstable. Here, we demonstrate the potential of coordinate descent algorithms for fitting these models, establishing theoretical convergence properties and demonstrating that they are significantly faster than competing approaches. In addition, we demonstrate the utility of convexity diagnostics to determine regions of the parameter space in which the objective function is locally convex, even though the penalty is not. Our simulation study and data examples indicate that nonconvex penalties like MCP and SCAD are worthwhile alternatives to the lasso in many applications. In particular, our numerical results suggest that MCP is the preferred approach among the three methods.

Abstract: Penalized likelihood methods are fundamental to ultrahigh dimensional variable selection. How high dimensionality such methods can handle remains largely unknown. In this paper, we show that in the context of generalized linear models, such methods possess model selection consistency with oracle properties even for dimensionality of nonpolynomial (NP) order of sample size, for a class of penalized likelihood approaches using folded-concave penalty functions, which were introduced to ameliorate the bias problems of convex penalty functions. This fills a long-standing gap in the literature where the dimensionality is allowed to grow slowly with the sample size. Our results are also applicable to penalized likelihood with the L1-penalty, which is a convex function at the boundary of the class of folded-concave penalty functions under consideration. The coordinate optimization is implemented for finding the solution paths, whose performance is evaluated by a few simulation examples and the real data analysis.

Abstract: The main goal of the structural optimization is to minimize the weight of structures while satisfying all design requirements imposed by design codes. In this paper, the Artificial Bee Colony algorithm with an adaptive penalty function approach (ABC-AP) is proposed to minimize the weight of truss structures. The ABC algorithm is swarm intelligence based optimization technique inspired by the intelligent foraging behavior of honeybees. Five truss examples with fixed-geometry and up to 200 elements were studied to verify that the ABC algorithm is an effective optimization algorithm in the creation of an optimal design for truss structures. The results of the ABC-AP compared with results of other optimization algorithms from the literature show that this algorithm is a powerful search and optimization technique for structural design.

Abstract: In high-dimensional model selection problems, penalized least-square approaches have been extensively used. This paper addresses the question of both robustness and efficiency of penalized model selection methods, and proposes a data-driven weighted linear combination of convex loss functions, together with weighted L1-penalty. It is completely data-adaptive and does not require prior knowledge of the error distribution. The weighted L1-penalty is used both to ensure the convexity of the penalty term and to ameliorate the bias caused by the L1-penalty. In the setting with dimensionality much larger than the sample size, we establish a strong oracle property of the proposed method that possesses both the model selection consistency and estimation efficiency for the true non-zero coefficients. As specific examples, we introduce a robust method of composite L1-L2, and optimal composite quantile method and evaluate their performance in both simulated and real data examples.

Abstract: Using the $\ell_1$-norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, called the trace Lasso, uses the trace norm, which is a convex surrogate of the rank, of the selected covariates as the criterion of model complexity. We analyze the properties of our norm, describe an optimization algorithm based on reweighted least-squares, and illustrate the behavior of this norm on synthetic data, showing that it is more adapted to strong correlations than competing methods such as the elastic net.

Abstract: We estimate the worst-case complexity of minimizing an unconstrained, nonconvex composite objective with a structured nonsmooth term by means of some first-order methods. We find that it is unaffected by the nonsmoothness of the objective in that a first-order trust-region or quadratic regularization method applied to it takes at most $\mathcal{O}(\epsilon^{-2})$ function evaluations to reduce the size of a first-order criticality measure below $\epsilon$. Specializing this result to the case when the composite objective is an exact penalty function allows us to consider the objective- and constraint-evaluation worst-case complexity of nonconvex equality-constrained optimization when the solution is computed using a first-order exact penalty method. We obtain that in the reasonable case when the penalty parameters are bounded, the complexity of reaching within $\epsilon$ of a KKT point is at most $\mathcal{O}(\epsilon^{-2})$ problem evaluations, which is the same in order as the function-evaluation complexity of steepest-descent methods applied to unconstrained, nonconvex smooth optimization.

Abstract: We propose a new penalized method for variable selection and estimation that explicitly incorporates the correlation patterns among predictors. This method is based on a combination of the minimax concave penalty and Laplacian quadratic associated with a graph as the penalty function. We call it the sparse Laplacian shrinkage (SLS) method. The SLS uses the minimax concave penalty for encouraging sparsity and Laplacian quadratic penalty for promoting smoothness among coefficients associated with the correlated predictors. The SLS has a generalized grouping property with respect to the graph represented by the Laplacian quadratic. We show that the SLS possesses an oracle property in the sense that it is selection consistent and equal to the oracle Laplacian shrinkage estimator with high probability. This result holds in sparse, high-dimensional settings with p >> n under reasonable conditions. We derive a coordinate descent algorithm for computing the SLS estimates. Simulation studies are conducted to evaluate the performance of the SLS method and a real data example is used to illustrate its application.

Abstract: Specific Gaussian mixtures are considered to solve simultaneously variable selection and clustering problems. A non asymptotic penalized criterion is proposed to choose the number of mixture components and the relevant variable subset. Because of the non linearity of the associated Kullback-Leibler contrast on Gaussian mixtures, a general model selection theorem for MLE proposed by Massart (2007) is used to obtain the penalty function form. This theorem requires to control the bracketing entropy of Gaussian mixture families. The ordered and non-ordered variable selection cases are both addressed in this paper.

Abstract: We propose a new penalized method for variable selection and estimation that explicitly incorporates the correlation patterns among predictors. This method is based on a combination of the minimax concave penalty and Laplacian quadratic associated with a graph as the penalty function. We call it the sparse Laplacian shrinkage (SLS) method. The SLS uses the minimax concave penalty for encouraging sparsity and Laplacian quadratic penalty for promoting smoothness among coefficients associated with the correlated predictors. The SLS has a generalized grouping property with respect to the graph represented by the Laplacian quadratic. We show that the SLS possesses an oracle property in the sense that it is selection consistent and equal to the oracle Laplacian shrinkage estimator with high probability. This result holds in sparse, high-dimensional settings with p ≫ n under reasonable conditions. We derive a coordinate descent algorithm for computing the SLS estimates. Simulation studies are conducted to evaluate the performance of the SLS method and a real data example is used to illustrate its application.

Abstract: This paper deals with the economically optimized design and sensitivity of two of the most widely used systems in geotechnical engineering: spread footing and retaining wall. Several recent advanced optimization methods have been developed, but very few of these methods have been applied to geotechnical problems. The current research develops a modified particle swarm optimization (MPSO) approach to obtain the optimum design of spread footing and retaining wall. The algorithm handles the problem-specific constraints using a penalty function approach. The optimization procedure controls all geotechnical and structural design constraints while reducing the overall cost of the structures. To verify the effectiveness and robustness of the proposed algorithm, three case studies of spread footing and retaining wall are illustrated. Comparison of the results of the present method, standard PSO, and other selected methods employed in previous studies shows the reliability and accuracy of the algorithm. Moreover, the parametric performance is investigated in order to examine the effect of relevant variables on the optimum design of the footing and the retaining structure utilizing the proposed method.

Abstract: In this paper, we consider a class of optimal control problems subject to equality terminal state constraints and continuous state and control inequality constraints. By using the control parametrization technique and a time scaling transformation, the constrained optimal control problem is approximated by a sequence of optimal parameter selection problems with equality terminal state constraints and continuous state inequality constraints. Each of these constrained optimal parameter selection problems can be regarded as an optimization problem subject to equality constraints and continuous inequality constraints. On this basis, an exact penalty function method is used to devise a computational method to solve these optimization problems with equality constraints and continuous inequality constraints. The main idea is to augment the exact penalty function constructed from the equality constraints and continuous inequality constraints to the objective function, forming a new one. This gives rise to a sequence of unconstrained optimization problems. It is shown that, for sufficiently large penalty parameter value, any local minimizer of the unconstrained optimization problem is a local minimizer of the optimization problem with equality constraints and continuous inequality constraints. The convergent properties of the optimal parameter selection problems with equality constraints and continuous inequality constraints to the original optimal control problem are also discussed. For illustration, three examples are solved showing the effectiveness and applicability of the approach proposed.

Abstract: Classification and variable selection play an important role in knowledge discovery in high-dimensional data. Although Support Vector Machine (SVM) algorithms are among the most powerful classification and prediction methods with a wide range of scientific applications, the SVM does not include automatic feature selection and therefore a number of feature selection procedures have been developed. Regularisation approaches extend SVM to a feature selection method in a flexible way using penalty functions like LASSO, SCAD and Elastic Net. We propose a novel penalty function for SVM classification tasks, Elastic SCAD, a combination of SCAD and ridge penalties which overcomes the limitations of each penalty alone. Since SVM models are extremely sensitive to the choice of tuning parameters, we adopted an interval search algorithm, which in comparison to a fixed grid search finds rapidly and more precisely a global optimal solution. Feature selection methods with combined penalties (Elastic Net and Elastic SCAD SVMs) are more robust to a change of the model complexity than methods using single penalties. Our simulation study showed that Elastic SCAD SVM outperformed LASSO (L1) and SCAD SVMs. Moreover, Elastic SCAD SVM provided sparser classifiers in terms of median number of features selected than Elastic Net SVM and often better predicted than Elastic Net in terms of misclassification error. Finally, we applied the penalization methods described above on four publicly available breast cancer data sets. Elastic SCAD SVM was the only method providing robust classifiers in sparse and non-sparse situations. The proposed Elastic SCAD SVM algorithm provides the advantages of the SCAD penalty and at the same time avoids sparsity limitations for non-sparse data. We were first to demonstrate that the integration of the interval search algorithm and penalized SVM classification techniques provides fast solutions on the optimization of tuning parameters. The penalized SVM classification algorithms as well as fixed grid and interval search for finding appropriate tuning parameters were implemented in our freely available R package 'penalizedSVM'. We conclude that the Elastic SCAD SVM is a flexible and robust tool for classification and feature selection tasks for high-dimensional data such as microarray data sets.

Abstract: An efficient branch-bound algorithm is presented for solving the n-job, sequence-independent, single machine scheduling problem where the goal is to minimize the total penalty costs resulting from tardiness of jobs. The algorithm and computational results are given for the case of linear penalty functions. The modifications needed to handle the case of nonlinear penalty functions are also presented.

Abstract: C 0 interior penalty methods are discontinuous Galerkin methods for fourth order problems. In this article we discuss various aspects of such methods including a priori error analysis, a posteriori error analysis and fast solution techniques.

Abstract: This paper concerns an optimal dividend distribution problem for an insurance company whose risk process evolves as a spectrally negative L\'{e}vy process (in the absence of dividend payments). The management of the company is assumed to control timing and size of dividend payments. The objective is to maximize the sum of the expected cumulative discounted dividend payments received until the moment of ruin and a penalty payment at the moment of ruin, which is an increasing function of the size of the shortfall at ruin; in addition, there may be a fixed cost for taking out dividends. A complete solution is presented to the corresponding stochastic control problem. It is established that the value-function is the unique stochastic solution and the pointwise smallest stochastic supersolution of the associated HJB equation. Furthermore, a necessary and sufficient condition is identified for optimality of a single dividend-band strategy, in terms of a particular Gerber-Shiu function. A number of concrete examples are analyzed.

Abstract: Purpose: In comparison with conventional filtered backprojection (FBP) algorithms for x-ray computed tomography (CT) image reconstruction, statistical algorithms directly incorporate the random nature of the data and do not assume CT data are linear, noiseless functions of the attenuation line integral. Thus, it has been hypothesized that statistical image reconstruction may support a more favorable tradeoff than FBP between image noise and spatial resolution in dose-limited applications. The purpose of this study is to evaluate the noise-resolution tradeoff for the alternating minimization (AM) algorithm regularized using a nonquadratic penalty function. Methods: Idealized monoenergetic CT projection data with Poisson noise were simulated for two phantoms with inserts of varying contrast (7%-238%) and distance from the field-of-view (FOV) center (2-6.5 cm). Images were reconstructed for the simulated projection data by the FBP algorithm and two penalty function parameter values of the penalized AM algorithm. Each algorithm was run with a range of smoothing strengths to allow quantification of the noise-resolution tradeoff curve. Image noise is quantified as the standard deviation in the water background around each contrast insert. Modulation transfer functions (MTFs) were calculated from six-parameter model fits to oversampled edge-spread functions defined by the circular contrast-insert edges as a metric ofmore » local resolution. The integral of the MTF up to 0.5 lp/mm was adopted as a single-parameter measure of local spatial resolution. Results: The penalized AM algorithm noise-resolution tradeoff curve was always more favorable than that of the FBP algorithm. While resolution and noise are found to vary as a function of distance from the FOV center differently for the two algorithms, the ratio of noises when matching the resolution metric is relatively uniform over the image. The ratio of AM-to-FBP image variances, a predictor of dose-reduction potential, was strongly dependent on the shape of the AM's nonquadratic penalty function and was also strongly influenced by the contrast of the insert for which resolution is quantified. Dose-reduction potential, reported here as the fraction (%) of FBP dose necessary for AM to reconstruct an image with comparable noise and resolution, for one penalty parameter value of the AM algorithm was found to vary from 70% to 50% for low-contrast and high-contrast structures, respectively, and from 70% to 10% for the second AM penalty parameter value. However, the second penalty, AM-700, was found to suffer from poor low-contrast resolution when matching the high-contrast resolution metric with FBP. Conclusions: The results of this simulation study imply that penalized AM has the potential to reconstruct images with similar noise and resolution using a fraction (10%-70%) of the FBP dose. However, this dose-reduction potential depends strongly on the AM penalty parameter and the contrast magnitude of the structures of interest. In addition, the authors' results imply that the advantage of AM can be maximized by optimizing the nonquadratic penalty function to the specific imaging task of interest. Future work will extend the methods used here to quantify noise and resolution in images reconstructed from real CT data.« less

Abstract: We derive and study a C(0) interior penalty method for a sixth-order elliptic equation on polygonal domains. The method uses the cubic Lagrange finite-element space, which is simple to implement and is readily available in commercial software. After introducing some notation and preliminary results, we provide a detailed derivation of the method. We then prove the well-posedness of the method as well as derive quasi-optimal error estimates in the energy norm. The proof is based on replacing Galerkin orthogonality with a posteriori analysis techniques. Using this approach, we are able to obtain a Cea-like lemma with minimal regularity assumptions on the solution. Numerical experiments are presented that support the theoretical findings.

Abstract: A fast and efficient multiobjective optimization design method is developed for induction machines, which requires much fewer design iterations than the traditional design methods. In this new method, the number of prime variables that define the optimization is reduced to only six. A canonical particle swarm optimization (PSO) method with penalty function for design constraints is developed to find the optimal solution for a user-defined objective function. After several trial solutions with the PSO, the optimal regions for both the design variables and the performance indexes can be estimated. The results will provide useful information for both a drive system designer and a machine designer at an early stage of the design process. A comparison study of PSO and genetic algorithm (GA) is also performed in this paper, and the comparison shows that PSO is more successful in finding the global optima and also has better computational efficiency than GA. The original contributions of this paper are a novel induction machine design method, consideration of winding turn selection limitation, and a machine-design-focused comparison.

Abstract: Finite gaussian mixture models are widely used in statistics thanks to their great flexibility. However, parameter estimation for gaussian mixture models with high dimensionality can be challenging because of the large number of parameters that need to be estimated. In this letter, we propose a penalized likelihood estimator to address this difficulty. The -type penalty we impose on the inverse covariance matrices encourages sparsity on its entries and therefore helps to reduce the effective dimensionality of the problem. We show that the proposed estimate can be efficiently computed using an expectation-maximization algorithm. To illustrate the practical merits of the proposed method, we consider its applications in model-based clustering and mixture discriminant analysis. Numerical experiments with both simulated and real data show that the new method is a valuable tool for high-dimensional data analysis.

Abstract: Methods, systems, and apparatuses are provided for controlling an environmental maintenance system that includes a plurality of sensors and a plurality of actuators. The operation levels of the actuators can be determined by optimizing a penalty function. As part of the penalty function, the sensor values can be compared to reference values. The optimized values of the operation levels can account for energy use of actuators at various operation levels and predicted differences of the sensor values relative to the reference values at various operation levels. The predicted difference can be determined using a transfer model. An accuracy of the transfer model can be determined by comparing predicted values to measured values. This accuracy can be used in determining new operational levels from an output of the transfer model (e.g., attenuating the output of the transfer model based on the accuracy).

Abstract: We present a sequential quadratic programming method without using a penalty function or a filter for solving nonlinear equality constrained optimization. In each iteration, the linearized constraints of the quadratic programming are relaxed to satisfy two mild conditions; the step-size is selected such that either the value of the objective function or the measure of the constraint violations is sufficiently reduced. As a result, our method has two nice properties. First, we do not need to assume the boundedness of the iterative sequence. Second, we do not need any restoration phase which is necessary for filter methods. We prove that the algorithm will terminate at either an approximate Karush–Kuhn–Tucker point, an approximate Fritz–John point, or an approximate infeasible stationary point which is an approximate stationary point for minimizing the l2 norm of the constraint violations. By controlling the exactness of the linearized constraints and introducing a second-order correction technique, without...

Abstract: This paper deals with implementation of the multistage Adomian decomposition method (MADM) to solve a class of nonlinear programming (NLP) problems, which are reformulated with a nonlinear system of fractional differential equations. The multistage strategy is used to investigate the relation between an equilibrium point of the fractional order dynamical system and an optimal solution of the NLP problem. The preference of the method lies in the fact that the multistage strategy gives this relation in an arbitrary longtime interval, while the Adomian decomposition method (ADM) gives the optimal solution just only in the neighborhood of the initial time. The numerical results taken by the fractional order MADM show that these results are compatible with the solution of NLP problem rather than the ADM. Furthermore, in some cases the fractional order MADM can perform more rapid convergency to the optimal solution of optimization problem than the integer order ones.

Abstract: In this paper, the problem of short-term load forecasting is redefined and solved with a new metric, which is the extension of the conventional sum of squared error (SSE) metric The proposed metric is a nonsymmetric penalty function with different penalties for over-forecasting and under-forecasting Therefore, a large family of approaches that utilize gradient-based methods such as artificial neural networks with back propagation learning and regressions method with least squares estimate are not useful in this case To solve this problem, a modified radial basis function (RBF) network, which uses the genetic algorithm to estimate the weights of the network is presented This network has the ability to handle the new penalty function In addition, a fuzzy inference system is combined with the modified RBF network to incorporate the impact of temperature on load As a real case study, we tried to forecast the electric power load of Mazandaran area in Iran The comparison between the proposed method and the well-known RBF network demonstrates the efficiency of the proposed method with the new forecasting metric

Abstract: We propose and analyze several two-level additive Schwarz preconditioners for a weakly over-penalized symmetric interior penalty method for second order elliptic boundary value problems. We also report numerical results that illustrate the parallel performance of these preconditioners.