Abstract: The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

Abstract: The state-of-the-art hardware platforms for training deep neural networks are moving from traditional single precision (32-bit) computations towards 16 bits of precision - in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation Here we demonstrate, for the first time, the successful training of deep neural networks using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of deep learning models and datasets In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4 times improved throughput over today's systems

Abstract: Approximate computing forms a design alternative that exploits the intrinsic error resilience of various applications and produces energy-efficient circuits with small accuracy loss. In this paper, we propose an approximate hybrid high radix encoding for generating the partial products in signed multiplications that encodes the most significant bits with the accurate radix-4 encoding and the least significant bits with an approximate higher radix encoding. The approximations are performed by rounding the high radix values to their nearest power of two. The proposed technique can be configured to achieve the desired energy-accuracy tradeoffs. Compared with the accurate radix-4 multiplier, the proposed multipliers deliver up to 56% energy and 55% area savings, when operating at the same frequency, while the imposed error is bounded by a Gaussian distribution with near-zero average. Moreover, the proposed multipliers are compared with state-of-the-art inexact multipliers, outperforming them by up to 40% in energy consumption, for similar error values. Finally, we demonstrate the scalability of our technique.

Abstract: In this paper, we present a new iterative rounding framework for many clustering problems. Using this, we obtain an (α1 + є ≤ 7.081 + є)-approximation algorithm for k-median with outliers, greatly improving upon the large implicit constant approximation ratio of Chen. For k-means with outliers, we give an (α2+є ≤ 53.002 + є)-approximation, which is the first O(1)-approximation for this problem. The iterative algorithm framework is very versatile; we show how it can be used to give α1- and (α1 + є)-approximation algorithms for matroid and knapsack median problems respectively, improving upon the previous best approximations ratios of 8 due to Swamy and 17.46 due to Byrka et al. The natural LP relaxation for the k-median/k-means with outliers problem has an unbounded integrality gap. In spite of this negative result, our iterative rounding framework shows that we can round an LP solution to an almost-integral solution of small cost, in which we have at most two fractionally open facilities. Thus, the LP integrality gap arises due to the gap between almost-integral and fully-integral solutions. Then, using a pre-processing procedure, we show how to convert an almost-integral solution to a fully-integral solution losing only a constant-factor in the approximation ratio. By further using a sparsification technique, the additive factor loss incurred by the conversion can be reduced to any є > 0.

Abstract: HPC applications use floating point arithmetic operations extensively to solve computational problems. Mixed-precision computing seeks to use the lowest precision data type that is sufficient to achieve a desired accuracy, improving performance and reducing power consumption. Manually optimizing a program to use mixed precision is challenging as it not only requires extensive knowledge about the numerical behavior of the algorithm but also estimates of the rounding errors. In this work, we present ADAPT, a scalable approach for mixed-precision analysis on HPC workloads using algorithmic differentiation to provide accurate estimates about the final output error. ADAPT provides a floating-point precision sensitivity profile while incurring an overhead of only a constant multiple of the original computation irrespective of the number of variables analyzed. The sensitivity profile can be used to make algorithmic choices and to develop mixed-precision configurations of a program. We evaluate ADAPT on six benchmarks and a proxy application (LULESH) and show that we are able to achieve a speedup of 1.2× on the proxy application.

Abstract: In this article, we target approximate computing for arithmetic circuits, focusing on the most complex and power-hungry units: hardware multipliers. Driven by the lack of a clear solution on the energy-error efficiency of existing approximate multiplication techniques, we present a new, efficient, and easily applied approximation design, as well as explore the current state-of-the-art design space. We show that the proposed approximation scheme can be equally applied at design time to enable synthesis of customized approximate multiplier circuits and at runtime to support dynamic approximation tuning scenarios. We achieve significant gains-up to 69-percent energy and 64-percent area savings with respect to accurate designs-by proposing hybrid approximation performed by two independent techniques that reduce both the depth (through perforation) and the width (through rounding) of the partial products accumulation tree. The corresponding runtime approximation solution delivers energy gains of up to 47 percent, introducing negligible area. More importantly, we show that design solutions configured through the proposed approach form the Pareto frontier of the energy-error space when considering direct quantitative comparisons with existing state of the art.

Abstract: We experimentally study the fundamental problem of computing the volume of a convex polytope given as an intersection of linear halfspaces. We implement and evaluate randomized polynomial-time algorithms for accurately approximating the polytope’s volume in high dimensions (e.g., few hundreds) based onhit-and-run random walks. To carry out this efficiently, we experimentally correlate the effect of parameters, such as random walk length and number of sample points, with accuracy and runtime. Our method is based on Monte Carlo algorithms with guaranteed speed and provably high probability of success for arbitrarily high precision. We exploit the problem’s features in implementing a practical rounding procedure of polytopes, in computing only partial “generations” of random points, and in designing fast polytope boundary oracles. Our publicly available software is significantly faster than exact computation and more accurate than existing approximation methods. For illustration, volume approximations of Birkhoff polytopes B11,…,B15 are computed, in dimensions up to 196, whereas exact methods have only computed volumes of up to B10.

Abstract: We study the Ordered k-Median problem, in which the solution is evaluated by first sorting the client connection costs and then multiplying them with a predefined non-increasing weight vector (higher connection costs are taken with larger weights). Since the 1990s, this problem has been studied extensively in the discrete optimization and operations research communities and has emerged as a framework unifying many fundamental clustering and location problems such as k-Median and k-Center. Obtaining non-trivial approximation algorithms was an open problem even for simple topologies such as trees. Recently, Aouad and Segev (2017) were able to obtain an O(log n) approximation algorithm for Ordered k-Median using a sophisticated local-search approach. The existence of a constant-factor approximation algorithm, however, remained open even for the rectangular weight vector. In this paper, we provide an LP-rounding constant-factor approximation algorithm for the Ordered k-Median problem. We achieve this result by revealing an interesting connection to the classic k-Median problem. In particular, we propose a novel LP relaxation that uses the constraints of the natural LP relaxation for k-Median but minimizes over a non-metric, distorted cost vector. This cost function (approximately) emulates the weighting of distances in an optimum solution and can be guessed by means of a clever enumeration scheme of Aouad and Segev. Although the resulting LP has an unbounded integrality gap, we can show that the LP rounding process by Charikar and Li (2012) for k-Median, operating on the original, metric space, gives a constant-factor approximation when relating not only to the LP value but also to a combinatorial bound derived from the guessing phase. To analyze the rounding process under the non-linear, ranking-based objective of Ordered k-Median, we employ several new ideas and technical ingredients that we believe could be of interest in some of the numerous other settings related to ordered, weighted cost functions.

Abstract: Operating distributed cloudlets at optimal cost is nontrivial when facing not only the dynamic and unpredictable resource prices and user requests, but also the low efficiency of today's immature cloudlet infrastructures. We propose to control cloudlet networks at multiple granularities: fine-grained control of servers inside cloudlets and coarse-grained control of cloudlets themselves. We model this problem as a mixed-integer nonlinear program with the switching cost over time. To solve this problem online, we firstly linearize, "regularize", and decouple it into a series of one-shot subproblems that we solve at each corresponding time slot, and afterwards we design an iterative, dependent rounding framework using our proposed randomized pairwise rounding algorithm to convert the fractional control decisions into the integral ones at each time slot. Via rigorous theoretical analysis, we exhibit our approach's performance guarantee in terms of the competitive ratio and the multiplicative integrality gap towards the offline optimal integral decisions. Extensive evaluations with real-world data confirm the empirical superiority of our approach over the single granularity server control and the state-of-the-art algorithms.

Abstract: We propose a modular framework for representing the real numbers that generalizes ieee, posits, and related floating-point number systems, and which has its roots in universal codes for the positive integers such as the Elias codes. This framework unifies several known but seemingly unrelated representations within a single schema while also introducing new representations. We particularly focus on variable-length encoding of the binary exponent and on the manner in which fraction bits are mapped to values. Our framework builds upon and shares many of the attractive properties of posits but allows for independent experimentation with exponent codes, fraction mappings, reciprocal closure, rounding modes, handling of under- and overflow, and underlying precision.

Abstract: It has been shown that natural interval extensions (NIE) can be used to calculate the largest positive Lyapunov exponent (LLE). However, the elaboration of NIE are not always possible for some dynamical systems, such as those modelled by simple equations or by Simulink-type blocks. In this paper, we use rounding mode of floating-point numbers to compute the LLE. We have exhibited how to produce two pseudo-orbits by means of different rounding modes; these pseudo-orbits are used to calculate the Lower Bound Error (LBE). The LLE is the slope of the line gotten from the logarithm of the LBE, which is estimated by means of a recursive least square algorithm (RLS). The main contribution of this paper is to develop a procedure to compute the LLE based on the LBE without using the NIE. Additionally, with the aid of RLS the number of required points has been decreased. Eight numerical examples are given to show the effectiveness of the proposed technique.

Abstract: We propose a unifying framework based on configuration linear programs and ran-domized rounding, for different energy optimization problems in the dynamic speed-scaling setting. We apply our framework to various scheduling and routing problems in heterogeneous computing and networking environments. We first consider the energy minimization problem of scheduling a set of jobs on a set of parallel speed scalable processors in a fully heterogeneous setting. For both the preemptive-non-migratory and the preemptive-migratory variants, our approach allows us to obtain solutions of almost the same quality as for the homogeneous environment. By exploiting the result for the preemptive-non-migratory variant, we are able to improve the best known approximation ratio for the single processor non-preemptive problem. Furthermore, we show that our approach allows to obtain a constant-factor approximation algorithm for the power-aware preemptive job shop scheduling problem. Finally, we consider the min-power routing problem where we are given a network modeled by an undirected graph and a set of uniform demands that have to be routed on integral routes from their sources to their destinations so that the energy consumption is minimized. We improve the best known approximation ratio for this problem.

Abstract: We are concerned with the small ball behavior of the smallest singular value of random matrices. Often, establishing such results involves, in some capacity, a discretization of the unit sphere. This requires bounds on the norm of the matrix, and the latter bounds require strong assumptions on the distribution of the entries, such as bounded fourth moments (for a weak estimate), sub-gaussian tails (for a strong estimate), and structural assumptions such as mean zero and variance one. Recently, Rebrova and Tikhomirov developed a discretization procedure which does not rely on strong tail assumptions for the entries. However, their argument still required the structural assumptions of mean zero, variance one i.i.d. entries. In this paper, we discuss an efficient discretization of the unit sphere, which works with exponentially high probability, does not require any such structural assumptions, and, furthermore, does not require independence of the rows of the matrix. We show the existence of nets near the sphere, which compare values of any (deterministic) random matrix on the sphere and on the net via the regularized the Hilbert-Schmidt norm, which we introduce. Such refinement is a form of averaging, and enjoys strong large deviation properties. As a consequence we show sharp small ball estimates for the smallest singular value of square random matrices under mild assumptions, and for the random matrices with arbitrary aspect ratio.

Abstract: A long-standing problem related to floating-point implementation of numerical programs is to provide efficient yet precise analysis of output errors.We present a framework to compute lower bounds on largest absolute roundoff errors, for a particular rounding model. This method applies to numerical programs implementing polynomial functions with box constrained input variables. Our study is based on three different hierarchies, relying respectively on generalized eigenvalue problems, elementary computations, and semidefinite programming (SDP) relaxations. This is complementary of over-approximation frameworks, consisting of obtaining upper bounds on the largest absolute roundoff error. Combining the results of both frameworks allows one to get enclosures for upper bounds on roundoff errors.The under-approximation framework provided by the third hierarchy is based on a new sequence of convergent robust SDP approximations for certain classes of polynomial optimization problems. Each problem in this hierarchy can be solved exactly via SDP. By using this hierarchy, one can provide a monotone nondecreasing sequence of lower bounds converging to the absolute roundoff error of a program implementing a polynomial function, applying for a particular rounding model.We investigate the efficiency and precision of our method on nontrivial polynomial programs coming from space control, optimization, and computational biology.

Abstract: Linear segments are widely used to describe the tool path in five-axis NC machining. However, the geometrical discontinuity of adjacent segments will inevitably result in the feedrate fluctuation and excessive acceleration, thus deteriorating the machining accuracy and machining quality. In this paper, a new circumscribed corner rounding method for a five-axis linear tool path is proposed based on double cubic B-splines. Compared with commonly used inscribed corner rounding method, the transition curves generated by the circumscribed method have smaller curvatures, which is benefit of improving the feedrate of the corner regions of the tool path. A configuration of control points for the circumscribed corner transition spline is first given, and double cubic B-splines are subsequently employed to smooth the trajectories of the tool tip point and the second point on the tool axis according to the maximum approximation error. Then, based on the parametric synchronization between the bottom and top B-splines, the first and second geometrical derivative continuous conditions of the tool orientation are derived at the junctions between the remaining linear tool path and the transition curves. The proposed corner rounding method is capable of achieving tool orientation continuous change without any iteration, which can be used in a high-efficiency way. Simulations are performed to show the feasibility and efficiency of the proposed method.

Abstract: We consider a generalized version of the correlation clustering problem, defined as follows. Given a complete graph $G$ whose edges are labeled with + or −, we wish to partition the graph into clusters while trying to avoid errors: + edges between clusters or − edges within clusters. Classically, one seeks to minimize the total number of such errors. We introduce a new framework that allows the objective to be a more general function of the number of errors at each vertex (for example, we may wish to minimize the number of errors at the worst vertex) and provides a rounding algorithm which converts “fractional clusterings” into discrete clusterings while causing only a constant-factor blowup in the number of errors at each vertex. This rounding algorithm yields constant-factor approximation algorithms for the discrete problem under a wide variety of objective functions.

Abstract: Semidefinite programming is a powerful tool in the design and analysis of approximation algorithms for combinatorial optimization problems. In particular, the random hyperplane rounding method of Goemans and Williamson has been extensively studied for more than two decades, resulting in various extensions to the original technique and beautiful algorithms for a wide range of applications. Despite the fact that this approach yields tight approximation guarantees for some problems, e.g., Max-Cut, for many others, e.g., Max-SAT and Max-DiCut, the tight approximation ratio is still unknown. One of the main reasons for this is the fact that very few techniques for rounding semidefinite relaxations are known. In this work, we present a new general and simple method for rounding semi-definite programs, based on Brownian motion. Our approach is inspired by recent results in algorithmic discrepancy theory. We develop and present tools for analyzing our new rounding algorithms, utilizing mathematical machinery from the theory of Brownian motion, complex analysis, and partial differential equations. Focusing on constraint satisfaction problems, we apply our method to several classical problems, including Max-Cut, Max-2SAT, and MaxDiCut, and derive new algorithms that are competitive with the best known results. To illustrate the versatility and general applicability of our approach, we give new approximation algorithms for the Max-Cut problem with side constraints that crucially utilizes measure concentration results for the Sticky Brownian Motion, a feature missing from hyperplane rounding and its generalizations

Abstract: The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of the 12-bit fixed-point arithmetic when training a convolutional neural network with the CIFAR-10 dataset is analysed, showing that such arithmetic is not the most appropriate for the training phase. After that, the paper presents and evaluates, under the same conditions, alternative low-precision arithmetics, starting with the 12-bit floating-point arithmetic. These two representations are then leveraged using local scaling in order to increase accuracy and get closer to the baseline 32-bit floating-point arithmetic. Finally, the paper introduces a simplified model in which both the outputs and the gradients of the neural networks are constrained to power-of-two values, just using 7 bits for their representation. The evaluation demonstrates a minimal loss in accuracy for the proposed Power-of-Two neural network, avoiding the use of multiplications and divisions and thereby, significantly reducing the training time as well as the energy consumption and memory requirements during the training and inference phases.

Abstract: The Time-Invariant Incremental Knapsack problem (IIK) is a generalization of Maximum Knapsack to a discrete multi-period setting At each time, capacity increases and items can be added, but not removed from the knapsack The goal is to maximize the sum of profits over all times IIK models various applications including specific financial markets and governmental decision processes IIK is strongly NP-Hard [2] and there has been work [2, 3, 6, 13, 15] on giving approximation algorithms for some special cases In this paper, we settle the complexity of IIK by designing a PTAS based on rounding a disjunctive formulation, and provide several extensions of the technique

Abstract: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

Abstract: We propose a simple iterative (SI) algorithm for the maxcut problem through fully using an equivalent continuous formulation. It does not need rounding at all and has advantages that all subproblems have explicit analytic solutions, the cut values are monotonically updated and the iteration points converge to a local optima in finite steps via an appropriate subgradient selection. Numerical experiments on G-set demonstrate the performance. In particular, the ratios between the best cut values achieved by SI and the best known ones are at least $0.986$ and can be further improved to at least $0.997$ by a preliminary attempt to break out of local optima.

Abstract: A growing number of surveys elicit respondents’ expectations for future events on a 0-100 scale of percent chance. These data reveal substantial heaping at multiples of 10 and 5 percent, suggesting that respondents round their reports. This paper studies the nature of rounding by analyzing response patterns across expectations questions and waves of the Health and Retirement Study. We discover a tendency by about half of the respondents to provide more refined responses in the tails of the scale than the center. Only about five percent provide more refined responses in the center than the tails. We find that rounding varies across question domains, which range from personal health to personal finances to macroeconomic events. We develop a two-stage framework to characterize person-specific rounding. The first stage uses observed responses to infer respondents’ rounding practice in each question domain and scale segment. The second stage replaces each original point response with an interval, representing the range of possible values of the respondent’s true latent belief implied by the degree of rounding inferred in the first stage. We study how the inferred rounding types in the first stage vary with respondent characteristics, including age and cognitive abilities.

Abstract: In a second seminal paper on the application of semidefinite programming to graph partitioning problems, Goemans and Williamson showed in 2004 how to formulate and round a complex semidefinite program to give what is to date still the best-known approximation guarantee of .836008 for Max-3-Cut. (This approximation ratio was also achieved independently around the same time by De Klerk et al..) Goemans and Williamson left open the problem of how to apply their techniques to Max-k-Cut for general k. They point out that it does not seem straightforward or even possible to formulate a good quality complex semidefinite program for the general Max-k-Cut problem, which presents a barrier for the further application of their techniques. We present a simple rounding algorithm for the standard semidefinite programmming relaxation of Max-k-Cut and show that it is equivalent to the rounding of Goemans and Williamson in the case of Max-3-Cut. This allows us to transfer the elegant analysis of Goemans and Williamson for Max-3-Cut to Max-k-Cut. For k > 3, the resulting approximation ratios are about .01 worse than the best known guarantees. Finally, we present a generalization of our rounding algorithm and conjecture (based on computational observations) that it matches the best-known guarantees of De Klerk et al.

Abstract: Many service systems provide queue length information to customers to aid their decisions of what queue to join. One example is at Walt Disney World (WDW), where waiting times are posted to customers via an app. However, it has been observed that the real waiting times are not posted in the app. In fact, WDW rounds the waiting time up to nearest five minute interval. In this paper, we build a simulation model to study the impact of rounding this information and when the information is lagged. We show that rounding or delaying information can result in oscillations in the queue length process. Moreover, increasing the rounding parameter or the delay in information causes oscillations to increase. We also demonstrate that our queueing model can mimic the observed dynamics in the data seen in WDW. Thus, we show the importance of understanding the impact of rounding or delaying information.

Abstract: We give a general method for rounding linear programs that combines the commonly used iterated rounding and randomized rounding techniques. In particular, we show that whenever iterated rounding can be applied to a problem with some slack, there is a randomized procedure that returns an integral solution that satisfies the guarantees of iterated rounding and also has concentration properties. We use this to give new results for several classic problems where iterated rounding has been useful.