Abstract: In this paper, new strategies are devised for joint load balancing and interference management in the downlink of a heterogeneous network, where small cells are densely deployed within the coverage area of a traditional macrocell. Unlike existing work, the limited backhaul capacity at each base station (BS) is taken into account. Here, users (UEs) cannot be offloaded to any arbitrary BS, but only to ones with sufficient backhaul capacity remaining. Jointly designed with traffic offload, transmit power allocation mitigates the intercell interference to further support the quality of service of each UE. The objective here is either: 1) to maximize the network sum rate subject to minimum throughput requirements at individual UEs, or 2) to maximize the minimum UE throughput. Both formulated problems belong to the difficult class of mixed-integer nonconvex optimization problems. The inherently binary BS-UE association variables are strongly coupled with the transmit power variables, making the problems even more challenging to solve. New iterative algorithms are developed based on an exact penalty method combined with successive convex programming, where the binary BS-UE association problem and the nonconvex power allocation problem are dealt with one at a time. At each iteration of the proposed algorithms, only two simple convex problems need to be solved at the same time scale. It is proven that the algorithms improve the objective functions at each iteration and converge eventually. Numerical results demonstrate the efficiency of the proposed algorithms in both traffic offloading and interference mitigation.

Abstract: In this paper, joint resource allocation and power control for energy efficient device-to-device (D2D) communications underlaying cellular networks are investigated. The resource and power are optimized for maximization of the energy efficiency (EE) of D2D communications. Exploiting the properties of fractional programming, we transform the original nonconvex optimization problem in fractional form into an equivalent optimization problem in subtractive form. Then, an efficient iterative resource allocation and power control scheme is proposed. In each iteration, part of the constraints of the EE optimization problem is removed by exploiting the penalty function approach. We further propose a novel two-layer approach which allows to find the optimum at each iteration by decoupling the EE optimization problem of joint resource allocation and power control into two separate steps. In the first layer, the optimal power values are obtained by solving a series of maximization problems through root-finding with or without considering the loss of cellular users' rates. In the second layer, the formulated optimization problem belongs to a classical resource allocation problem with single allocation format which admits a network flow formulation so that it can be solved to optimality. Simulation results demonstrate the remarkable improvements in terms of EE by using the proposed iterative resource allocation and power control scheme.

Abstract: We study a new class of elliptic variational-hemivariational inequalities in reflexive Banach spaces. An inequality in the class is governed by a nonlinear operator, a convex set of constraints and two nondifferentiable functionals, among which at least one is convex. We deliver a result on existence and uniqueness of a solution to the inequality. Next, we show the continuous dependence of the solution on the data of the problem and we introduce a penalty method, for which we state and prove a convergence result. Finally, we consider a mathematical model which describes the equilibrium of an elastic body in unilateral contact with a foundation. The model leads to a variational-hemivariational inequality for the displacement field, that we analyse by using our abstract results.

Abstract: This paper addresses the issue of robust sparse recovery in compressive sensing (CS) in the presence of impulsive measurement noise. Recently, robust data-fitting models, such as $\ell _1$ -norm, Lorentzian-norm, and Huber penalty function, have been employed to replace the popular $\ell _2$ -norm loss model to gain more robust performance. In this paper, we propose a robust formulation for sparse recovery using the generalized $\ell _p$ -norm with $0\leq p as the metric for the residual error. To solve this formulation efficiently, we develop an alternating direction method (ADM) via incorporating the proximity operator of $\ell _p$ -norm functions into the framework of augmented Lagrangian methods. Furthermore, to derive a convergent method for the nonconvex case of $p 1, a smoothing strategy has been employed. The convergence conditions of the proposed algorithm have been analyzed for both the convex and nonconvex cases. The new algorithm has been compared with some state-of-the-art robust algorithms via numerical simulations to show its improved performance in highly impulsive noise.

Abstract: We propose an algorithm for solving nonsmooth, nonconvex, constrained optimization problems as well as a new set of visualization tools for comparing the performance of optimization algorithms. Our algorithm is a sequential quadratic optimization method that employs Broyden-Fletcher-Goldfarb-Shanno BFGS quasi-Newton Hessian approximations and an exact penalty function whose parameter is controlled using a steering strategy. While our method has no convergence guarantees, we have found it to perform very well in practice on challenging test problems in controller design involving both locally Lipschitz and non-locally-Lipschitz objective and constraint functions with constraints that are typically active at local minimizers. In order to empirically validate and compare our method with available alternatives—on a new test set of 200 problems of varying sizes—we employ new visualization tools which we call relative minimization profiles. Such profiles are designed to simultaneously assess the relative performance of several algorithms with respect to objective quality, feasibility, and speed of progress, highlighting the trade-offs between these measures when comparing algorithm performance.

Abstract: In the multiple changepoint setting, various search methods have been proposed, which involve optimizing either a constrained or penalized cost function over possible numbers and locations of changepoints using dynamic programming. Recent work in the penalized optimization setting has focused on developing an exact pruning-based approach that, under certain conditions, is linear in the number of data points. Such an approach naturally requires the specification of a penalty to avoid under/over-fitting. Work has been undertaken to identify the appropriate penalty choice for data-generating processes with known distributional form, but in many applications the model assumed for the data is not correct and these penalty choices are not always appropriate. To this end, we present a method that enables us to find the solution path for all choices of penalty values across a continuous range. This permits an evaluation of the various segmentations to identify a suitable penalty choice. The computational ...

Abstract: This chapter introduces two very important concepts in constrained nonlinear optimization. These are penalty and augmented Lagrangian concepts. The idea is that both these concepts replace the original problem by a sequence of sub-problems in which the constraints are expressed by terms added to the objective function. The penalty concept is implemented in two different methods. The quadratic penalty method adds to the objective function a multiple of the square of the violation of each constraint and solves a sequence of unconstrained optimization sub-problems. Simple and enough intuitive, this approach has some important deficiencies. The nonsmooth exact penalty method, on the other hand, solves a single unconstrained optimization problem. In this approach, a popular function is the l1 penalty function. The problem with this method is that the nonsmoothness may create complications in numerical implementations. Finally, the second concept is the multiplier method or the augmented Lagrangian method, which explicitly uses Lagrange multiplier estimates in order to avoid the ill-conditioning of the quadratic penalty method.

Abstract: Unconstrained optimization is the search for the maximum or minimum of a function with no restriction on the values of the variables. At the same time, it forms the basis for methods of constrained optimization in the next chapter. Zero-order methods use only function values, progress made in the previous step pointing the way to the next step. The Hooke and Jeeves method is one such method, suitable for small problems with little programming effort. First-order methods employ the gradient of the function, usually obtained by finite difference, to derive a search direction. This is followed by a line search along this direction for the current maximum or minimum, performed either by progressive reduction of the region in which the maximum or minimum is to be found or by polynomial interpolation. In its simplest form, this is the steepest descent method. However, by the use of gradient data from the previous iteration, an improved search direction can be found, with faster convergence. This is the Fletcher–Reeves method. A more general formulation is based on a quadratic approximation to the objective function, referred to as a second-order method or quasi-Newton method. This involves progressively building up an approximation to the inverse of the Hessian matrix of second derivatives to deduce a search direction. A spreadsheet program for the Hooke and Jeeves method is also used in the next chapter for the penalty function method for constrained optimization.

Abstract: Many optimization problems in communications and signal processing can be formulated as rank-one constrained optimization problems. This has motivated the development of methods to solve such problem in specific scenarios. However, due to the non-convex nature of the rank-one constraint, limited progress has been made in solving generic rank-one constrained optimization problems. In particular, the problem of efficiently finding a locally optimal solution to a generic rank-one constrained problem remains open. This paper focuses on solving general rank-one constrained problems via relaxation techniques. However, instead of dropping the rank-one constraint completely as is done in traditional rank-one relaxation methods, a novel algorithm that gradually relaxes the rank-one constraint, termed the sequential rank-one constraint relaxation (SROCR) algorithm, is proposed. Compared with previous algorithms, the SROCR algorithm can solve general rank-one constrained problems, and can find feasible solutions with favorable complexity.

Abstract: The sensitivity of the initial guess in terms of optimizer based on an hp-adaptive pseudospectral method for solving a space maneuver vehicle's (SMV) trajectory optimization problem has long been recognized as a difficult problem. Because of the sensitivity with regard to the initial guess, it may cost the solver a large amount of time to do the Newton iteration and get the optimal solution or even the local optimal solution. In this paper, to provide the optimizer a better initial guess and solve the SMV trajectory optimization problem, an initial guess generator using a violation learning differential evolution algorithm is introduced. A new constraint-handling strategy without using penalty function is presented to modify the fitness values so that the performance of each candidate can be generalized. In addition, a learning strategy is designed to add diversity for the population in order to improve the convergency speed and avoid local optima. Several simulation results are conducted by using the combination algorithm; simulation results indicated that using limited computational efforts, the method proposed to generate initial guess can have better performance in terms of convergence ability and convergence speed compared with other approaches. By using the initial guess, the combinational method can also enhance the quality of the solution and reduce the number of Newton iteration and computational time. Therefore, the method is potentially feasible for solving the SMV trajectory optimization problem.

Abstract: This paper presents a Successive Convexification ($ \texttt{SCvx} $) algorithm to solve a class of non-convex optimal control problems with certain types of state constraints. Sources of non-convexity may include nonlinear dynamics and non-convex state/control constraints. To tackle the challenge posed by non-convexity, first we utilize exact penalty function to handle the nonlinear dynamics. Then the proposed algorithm successively convexifies the problem via a $ \textit{project-and-linearize} $ procedure. Thus a finite dimensional convex programming subproblem is solved at each succession, which can be done efficiently with fast Interior Point Method (IPM) solvers. Global convergence to a local optimum is demonstrated with certain convexity assumptions, which are satisfied in a broad range of optimal control problems. The proposed algorithm is particularly suitable for solving trajectory planning problems with collision avoidance constraints. Through numerical simulations, we demonstrate that the algorithm converges reliably after only a few successions. Thus with powerful IPM based custom solvers, the algorithm can be implemented onboard for real-time autonomous control applications.

Abstract: Differential Dynamic Programming (DDP) has become a popular approach to performing trajectory optimization for complex, underactuated robots. However, DDP presents two practical challenges. First, the evaluation of dynamics derivatives during optimization creates a computational bottleneck, particularly in implementations that capture second-order dynamic effects. Second, constraints on the states (e.g., boundary conditions, collision constraints, etc.) require additional care since the state trajectory is implicitly defined from the inputs and dynamics. This paper addresses both of these problems by building on recent work on Unscented Dynamic Programming (UDP) — which eliminates dynamics derivative computations in DDP—to support general nonlinear state and input constraints using an augmented Lagrangian. The resulting algorithm has the same computational cost as first-order penalty-based DDP variants, but can achieve constraint satisfaction to high precision without the numerical ill-conditioning associated with penalty methods. We present results demonstrating its favorable performance on several simulated robot systems including a quadrotor and 7-DoF robot arm.

Abstract: This paper explores the performance of three evolutionary optimization methods, differential evolution (DE), evolutionary strategy (ES) and biogeography based optimization algorithm (BBO), for nonlinear constrained optimum design of a cantilever retaining wall. These algorithms are based on biological contests for survival and reproduction. The retaining wall optimization problem consists of two criteria, geotechnical stability and structural strength, while the final design minimizes an objective function. The objective function is defined in terms of both cost and weight. Constraints are applied using the penalty function method. The efficiency of the proposed method is examined by means of two numerical retaining wall design examples, one with a base shear key and one without a base shear key. The final designs are compared to the ones determined by genetic algorithms as classical metaheuristic optimization methods. The design results and convergence rate of the BBO algorithm show a significantly better performance than the other algorithms in both design cases.

Abstract: We investigate the augmented Lagrangian dual (ALD) for mixed integer linear programming (MIP) problems. ALD modifies the classical Lagrangian dual by appending a nonlinear penalty function on the violation of the dualized constraints in order to reduce the duality gap. We first provide a primal characterization for ALD for MIPs and prove that ALD is able to asymptotically achieve zero duality gap when the weight on the penalty function is allowed to go to infinity. This provides an alternative characterization and proof of a recent result in Boland and Eberhard (Math Program 150(2):491---509, 2015, Proposition 3). We further show that, under some mild conditions, ALD using any norm as the augmenting function is able to close the duality gap of an MIP with a finite penalty coefficient. This generalizes the result in Boland and Eberhard (2015, Corollary 1) from pure integer programming problems with bounded feasible region to general MIPs. We also present an example where ALD with a quadratic augmenting function is not able to close the duality gap for any finite penalty coefficient.

Abstract: In this paper, a modified version of spider monkey optimization (SMO) algorithm for solving constrained optimization problems has been proposed. To the best of author's knowledge, this is the first attempt to develop a version of SMO which can solve constrained continuous optimization problems by using the Deb's technique for handling constraints. The proposed algorithm is named constrained spider monkey optimization (CSMO) algorithm. The performance of CSMO is investigated on the well-defined constrained optimization problems of CEC2006 and CEC2010 benchmark sets. The results of the proposed algorithm are compared with constrained versions of particle swarm optimization, artificial bee colony and differential evolution. Outcome of the experiment and the discussion of results demonstrate that CSMO handles the global optimization task very well for constrained optimization problems and shows better performance in comparison with compared algorithms. Such an outcome will be an encouragement for the research community to further explore the potential of SMO in solving benchmarks as well as real-world problems, which are often constrained in nature.

Abstract: Maximum consensus estimation plays a critically important role in computer vision. Currently, the most prevalent approach draws from the class of non-deterministic hypothesize-and-verify algorithms, which are cheap but do not guarantee solution quality. On the other extreme, there are global algorithms which are exhaustive search in nature and can be costly for practical-sized inputs. This paper aims to fill the gap between the two extremes by proposing a locally convergent maximum consensus algorithm. Our method is based on a formulating the problem with linear complementarity constraints, then defining a penalized version which is provably equivalent to the original problem. Based on the penalty problem, we develop a Frank-Wolfe algorithm that can deterministically solve the maximum consensus problem. Compared to the randomized techniques, our method is deterministic and locally convergent, relative to the global algorithms, our method is much more practical on realistic input sizes. Further, our approach is naturally applicable to problems with geometric residuals.

Abstract: Extended waveform inversion globalizes the convergence of seismic waveform inversion by adding nonphysical degrees of freedom to the model, thus permitting it to fit the data well throughout the inversion process. These extra degrees of freedom must be curtailed at the solution, for example, by penalizing them as part of an optimization formulation. For separable (partly linear) models, a natural objective function combines a mean square data residual and a quadratic regularization term penalizing the nonphysical (linear) degrees of freedom. The linear variables are eliminated in an inner optimization step, leaving a function of the outer (nonlinear) variables to be optimized. This variable projection method is convenient for computation, but it requires that the penalty weight be increased as the estimated model tends to the (physical) solution. We describe an algorithm based on discrepancy, that is, maintaining the data residual at the inner optimum within a prescribed range, to control the pena...

Abstract: Beetle antennae search (BAS) is an efficient meta-heuristic algorithm inspired by foraging behaviors of beetles. This algorithm includes several parameters for tuning and the existing results are limited to solve single objective optimization. This work pushes forward the research on BAS by providing one variant that releases the tuning parameters and is able to handle multi-objective optimization. This new approach applies normalization to simplify the original algorithm and uses a penalty function to exploit infeasible solutions with low constraint violation to solve the constraint optimization problem. Extensive experimental studies are carried out and the results reveal efficacy of the proposed approach to constraint handling.

Abstract: Feasibility pumps are highly effective primal heuristics for mixed-integer linear and nonlinear optimization. However, despite their success in practice there are only a few works considering their theoretical properties. We show that feasibility pumps can be seen as alternating direction methods applied to special reformulations of the original problem, inheriting the convergence theory of these methods. Moreover, we propose a novel penalty framework that encompasses this alternating direction method, which allows us to refrain from random perturbations that are applied in standard versions of feasibility pumps in case of failure. We present a convergence theory for the new penalty based alternating direction method and compare the new variant of the feasibility pump with existing versions in an extensive numerical study for mixed-integer linear and nonlinear problems.

Abstract: Recently, several researchers within the evolutionary and swarm computing community have been interested in solving dynamic multi-objective problems where the objective functions, the problem's parameters, and/or the constraints may change over time. According to the related literature, most works have focused on the dynamicity of objective functions, which is insufficient since also constraints may change over time along with the objectives. For instance, a feasible solution could become infeasible after a change occurrence, and vice versa. Besides, a non-dominated solution may become dominated, and vice versa. Motivated by these observations, we devote this paper to focus on the dynamicity of both: (1) problem's constraints and (2) objective functions. To achieve our goal, we propose a new self-adaptive penalty function and a new feasibility driven strategy that are embedded within the NSGA-II and that are applied whenever a change is detected. The feasibility driven strategy is able to guide the search towards the new feasible directions according to the environment changes. The empirical results have shown that our proposal is able to handle various challenges raised by the problematic of dynamic constrained multi-objective optimization. Moreover, we have compared our new dynamic constrained NSGA-II version, denoted as DC-MOEA, against two existent dynamic constrained evolutionary algorithms. The obtained results have demonstrated the competitiveness and the superiority of our algorithm on both aspects of convergence and diversity.