Golomb coding

Abstract: LOCO-I (LOw COmplexity LOssless COmpression for Images) is the algorithm at the core of the new ISO/ITU standard for lossless and near-lossless compression of continuous-tone images, JPEG-LS. It is conceived as a "low complexity projection" of the universal context modeling paradigm, matching its modeling unit to a simple coding unit. By combining simplicity with the compression potential of context models, the algorithm "enjoys the best of both worlds." It is based on a simple fixed context model, which approaches the capability of the more complex universal techniques for capturing high-order dependencies. The model is tuned for efficient performance in conjunction with an extended family of Golomb (1966) type codes, which are adaptively chosen, and an embedded alphabet extension for coding of low-entropy image regions. LOCO-I attains compression ratios similar or superior to those obtained with state-of-the-art schemes based on arithmetic coding. Moreover, it is within a few percentage points of the best available compression ratios, at a much lower complexity level. We discuss the principles underlying the design of LOCO-I, and its standardization into JPEC-LS.

Abstract: Federated learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods, however, are only of limited utility in the federated learning setting, as they either only compress the upstream communication from the clients to the server (leaving the downstream communication uncompressed) or only perform well under idealized conditions, such as i.i.d. distribution of the client data, which typically cannot be found in federated learning. In this article, we propose sparse ternary compression (STC), a new compression framework that is specifically designed to meet the requirements of the federated learning environment. STC extends the existing compression technique of top- $k$ gradient sparsification with a novel mechanism to enable downstream compression as well as ternarization and optimal Golomb encoding of the weight updates. Our experiments on four different learning tasks demonstrate that STC distinctively outperforms federated averaging in common federated learning scenarios. These results advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.

Abstract: explicitly evaluable functions. For example, the M-ary error probability is expressed as a quadrature in Lindsey's equation (17), PE(M) = 1 [I-2 lrn Qi(h, $;) exp (-g) dz] z/d eeL s m =22/;;moe-(1+d)s~41-'2@3(1, 1 + M, s, sL) d.s, (5) where, following Lindsey, h2/2 has been replaced by L to simplify the notation. From the series form of @3, it is obvious that the integral gives an additional double series numerator parameter: PE(M) = di eeL z ~1 (1 + d)-\"-1'2 i (61 A complete set of recursion relations for F1 when one parameter at a time changes has been given by Le Vavasseur [S]. It is a simple matter to derive the necessary change for this two-parameter case but Le Vavasseur has included this as one of several examples, so that we have at once (8) (9)-I, e (a \"-' a \" '-'_ [eLP,(l)] = (1 + &)Y,(l), which is equivalent to a result of Price [9], who has derived a number of expressions for these and related integrals. Note that the derivation above is, thus far, much simpler and more straightforward than the admirably executed tours de force of previous derivations. However, the last step, viz., recognizing the form of the result, is automatically accomplished in the other derivations, and is much the harder part in the hypergeometric case. To obtain the reduction, we use operational relations [lo] to get The integral with the special parameters of (11) has been previously recognized as a Q function [12]-[14] so that the reduction is essentially complete.

Abstract: The applications of digital data compression and the major components of compression systems are described. Data modeling is discussed, and the role of entropy and data statistics is examined. Gray-scale image modeling is used to illustrate some of these mechanisms. The coding mechanisms are examined, and prefix codes are explained. Arithmetic coding is considered. >