Abstract: Quantization is considered as one of the most effective methods to optimize the inference cost of neural network models for their deployment to mobile and embedded systems, which have tight resource constraints. In such approaches, it is critical to provide low-cost quantization under a tight accuracy loss constraint (e.g., 1%). In this paper, we propose a novel method for quantizing weights and activations based on the concept of weighted entropy. Unlike recent work on binary-weight neural networks, our approach is multi-bit quantization, in which weights and activations can be quantized by any number of bits depending on the target accuracy. This facilitates much more flexible exploitation of accuracy-performance trade-off provided by different levels of quantization. Moreover, our scheme provides an automated quantization flow based on conventional training algorithms, which greatly reduces the design-time effort to quantize the network. According to our extensive evaluations based on practical neural network models for image classification (AlexNet, GoogLeNet and ResNet-50/101), object detection (R-FCN with 50-layer ResNet), and language modeling (an LSTM network), our method achieves significant reductions in both the model size and the amount of computation with minimal accuracy loss. Also, compared to existing quantization schemes, ours provides higher accuracy with a similar resource constraint and requires much lower design effort.

Abstract: Today's HPC applications are producing extremely large amounts of data, such that data storage and analysis are becoming more challenging for scientific research. In this work, we design a new error-controlled lossy compression algorithm for large-scale scientific data. Our key contribution is significantly improving the prediction hitting rate (or prediction accuracy) for each data point based on its nearby data values along multiple dimensions. We derive a series of multilayer prediction formulas and their unified formula in the context of data compression. One serious challenge is that the data prediction has to be performed based on the preceding decompressed values during the compression in order to guarantee the error bounds, which may degrade the prediction accuracy in turn. We explore the best layer for the prediction by considering the impact of compression errors on the prediction accuracy. Moreover, we propose an adaptive error-controlled quantization encoder, which can further improve the prediction hitting rate considerably. The data size can be reduced significantly after performing the variable-length encoding because of the uneven distribution produced by our quantization encoder. We evaluate the new compressor on production scientific data sets and compare it with many other state-of-the-art compressors: GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA. Experiments show that our compressor is the best in class, especially with regard to compression factors (or bit-rates) and compression errors (including RMSE, NRMSE, and PSNR). Our solution is better than the second-best solution by more than a 2x increase in the compression factor and 3.8x reduction in the normalized root mean squared error on average, with reasonable error bounds and user-desired bit-rates.

Abstract: We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.

Abstract: This paper presents an effective image retrieval method by combining high-level features from convolutional neural network (CNN) model and low-level features from dot-diffused block truncation coding (DDBTC). The low-level features, e.g., texture and color, are constructed by vector quantization -indexed histogram from DDBTC bitmap, maximum, and minimum quantizers. Conversely, high-level features from CNN can effectively capture human perception. With the fusion of the DDBTC and CNN features, the extended deep learning two-layer codebook features is generated using the proposed two-layer codebook, dimension reduction, and similarity reweighting to improve the overall retrieval rate. Two metrics, average precision rate and average recall rate (ARR), are employed to examine various data sets. As documented in the experimental results, the proposed schemes can achieve superior performance compared with the state-of-the-art methods with either low-or high-level features in terms of the retrieval rate. Thus, it can be a strong candidate for various image retrieval related applications.

Abstract: The goal of this paper is to propose an accurate method for estimating quantization steps from an image that has been previously JPEG-compressed and stored in lossless format. The method is based on the combination of the quantization effect and the statistics of discrete cosine transform (DCT) coefficient characterized by the statistical model that has been proposed in our previous works. The analysis of quantization effect is performed within a mathematical framework, which justifies the relation of local maxima of the number of integer quantized forward coefficients with the true quantization step. From the candidate set of the true quantization step given by the previous analysis, the statistical model of DCT coefficients is used to provide the optimal quantization step candidate. The proposed method can also be exploited to estimate the secondary quantization table in a double-JPEG compressed image stored in lossless format and detect the presence of JPEG compression. Numerical experiments on large image databases with different image sizes and quality factors highlight the high accuracy of the proposed method.

Abstract: Power spectral density (PSD) maps providing the distribution of RF power across space and frequency are constructed using power measurements collected by a network of low-cost sensors. By introducing linear compression and quantization to a small number of bits, sensor measurements can be communicated to the fusion center with minimal bandwidth requirements. Strengths of data- and model-driven approaches are combined to develop estimators capable of incorporating multiple forms of spectral and propagation prior information while fitting the rapid variations of shadow fading across space. To this end, novel nonparametric and semiparametric formulations are investigated. It is shown that PSD maps can be obtained using support vector machine-type solvers. In addition to batch approaches, an online algorithm attuned to real-time operation is developed. Numerical tests assess the performance of the novel algorithms.

Abstract: The Quantization table in JPEG, which specifies the quantization scale for each discrete cosine transform (DCT) coefficient, plays an important role in image codec optimization. However, the generic quantization table design that is based on the characteristics of human visual system (HVS) cannot adapt to the variations of image content. In this letter, we propose a just-noticeable difference (JND) based quantization table derivation method for JPEG by optimizing the rate-distortion costs for all the frequency bands. To achieve better perceptual quality, the DCT domain JND-based distortion metric is utilized to model the stair distortion perceived by HVS. The rate-distortion cost for each band is derived by estimating the rate with the first-order entropy of quantized coefficients. Subsequently, the optimal quantization table is obtained by minimizing the total rate-distortion costs of all the bands. Extensive experimental results show that the quantization table generated by the proposed method achieves significant bit-rate savings compared with JPEG recommended quantization table and specifically developed quantization tables in terms of both objective and subjective evaluations.

Abstract: Steganalysis of the quantization index modulation (QIM) steganography in a low-bit-rate encoded speech stream is conducted in this research. According to the speech generation theory and the phoneme distribution properties in language, we first point out that the correlation characteristics of split vector quantization (VQ) codewords of linear predictive coding filter coefficients are changed after the QIM steganography. Based on this observation, we construct a model called the Quantization codeword correlation network (QCCN) based on split VQ codeword from adjacent speech frames. Furthermore, the QCCN model is pruned to yield a stronger correlation network. After quantifying the correlation characteristics of vertices in the pruned correlation network, we obtain feature vectors that are sensitive to steganalysis. Finally, we build a high-performance detector using the support vector machine (SVM) classifier. It is shown by experimental results that the proposed QCCN steganalysis method can effectively detect the QIM steganography in encoded speech stream when it is applied to low-bit-rate speech codec such as G.723.1 and G.729.

Abstract: The aim of the article is to present a novel method for fuzzy medical image retrieval (FMIR) using vector quantization (VQ) with fuzzy signatures in conjunction with fuzzy S-trees. In past times, a task of similar pictures searching was not based on searching for similar content (e.g. shapes, colour) of the pictures but on the picture name. There exist some methods for the same purpose, but there is still some space for development of more efficient methods. The proposed image retrieval system is used for finding similar images, in our case in the medical area --- in mammography, in addition to the creation of the list of similar images --- cases. The created list is used for assessing the nature of the finding --- whether the medical finding is malignant or benign. The suggested method is compared to the method using Normalized Compression Distance (NCD) instead of fuzzy signatures and fuzzy S-tree. The method with NCD is useful for the creation of the list of similar cases for malignancy assessment, but it is not able to capture the area of interest in the image. The proposed method is going to be added to the complex decision support system to help to determine appropriate healthcare according to the experiences of similar, previous cases.

Abstract: Based on the knowledge that input data distribution is important for learning, a data density-dependent quantization scheme (DQS) is proposed for sparse input data representation. The usefulness of the representation scheme is demonstrated by using it as a data preprocessing unit attached to the well-known least squares support vector machine (LS-SVM) for application on big data sets. Essentially, the proposed DQS adopts a single shrinkage threshold to obtain a simple quantization scheme, which adapts its outputs to input data density. With this quantization scheme, a large data set is quantized to a small subset where considerable sample size reduction is generally obtained. In particular, the sample size reduction can save significant computational cost when using the quantized subset for feature approximation via the Nystrom method. Based on the quantized subset, the approximated features are incorporated into LS-SVM to develop a data density-dependent quantized LS-SVM (DQLS-SVM), where an analytic solution is obtained in the primal solution space. The developed DQLS-SVM is evaluated on synthetic and benchmark data with particular emphasis on large data sets. Extensive experimental results show that the learning machine incorporating DQS attains not only high computational efficiency but also good generalization performance.

Abstract: In the present work, a novel image watermarking algorithm using vector quantization (VQ) approach is presented for digital image authentication Watermarks are embedded in two successive stages for image integrity verification and authentication In the first stage, a key based approach is used to embed robust zero level watermark using properties of indices of vector quantized image In the second stage, semifragile watermark is embedded by using modified index key based (MIKB) method Random keys are used to improve the integrity and security of the designed system Further, to classify an attack quantitatively as acceptable or as a malicious attack, pixel neighbourhood clustering approach is introduced Proposed approach is evaluated on 250 standard test images using performance measures such as peak signal to noise ratio (PSNR) and normalized hamming similarity (NHS) The experimental results shows that propose approach achieve average false positive rate 000024 and the average false negative rate 00012 Further, the average PSNR and tamper detection/localization accuracy of watermarked image is 42 dB and 998% respectively; while tamper localization sensitivity is very high The proposed model is found to be robust to common content preserving attacks while fragile to content altering attacks

Abstract: In this paper, we propose a self-embedding fragile watermarking scheme using vector quantization (VQ) and index sharing. First, the principle contents of original image are compactly represented by a series of VQ indices. Then, after permutation, the binary bits of VQ indices are extended to generate reference-bits by a random binary matrix, which can make all reference-bits share the information of VQ index bits from different regions of the whole image. The image is embedded with watermark-bits including hash-bits for tampering localization and reference-bits for content recovery, and is transmitted to receiver side. Tampered regions in the received, suspicious image can be accurately located and then be recovered by VQ index reconstruction. Experimental results demonstrate that the proposed scheme can achieve successful content recovery for larger tampering rate and obtain better visual quality of recovered results than the reported schemes.

Abstract: In this work we present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives state-of-the-art results for both.

Abstract: The growing neural gas (GNG) self-organizing neural network stands as one of the most successful examples of unsupervised learning of a graph of processing units. Despite its success, little attention has been devoted to its extension to a hierarchical model, unlike other models such as the self-organizing map, which has many hierarchical versions. Here, a hierarchical GNG is presented, which is designed to learn a tree of graphs. Moreover, the original GNG algorithm is improved by a distinction between a growth phase where more units are added until no significant improvement in the quantization error is obtained, and a convergence phase where no unit creation is allowed. This means that a principled mechanism is established to control the growth of the structure. Experiments are reported, which demonstrate the self-organization and hierarchy learning abilities of our approach and its performance for vector quantization applications.

Abstract: We consider the problem of deep neural net compression by quantization: given a large, reference net, we want to quantize its real-valued weights using a codebook with $K$ entries so that the training loss of the quantized net is minimal. The codebook can be optimally learned jointly with the net, or fixed, as for binarization or ternarization approaches. Previous work has quantized the weights of the reference net, or incorporated rounding operations in the backpropagation algorithm, but this has no guarantee of converging to a loss-optimal, quantized net. We describe a new approach based on the recently proposed framework of model compression as constrained optimization \citep{Carreir17a}. This results in a simple iterative "learning-compression" algorithm, which alternates a step that learns a net of continuous weights with a step that quantizes (or binarizes/ternarizes) the weights, and is guaranteed to converge to local optimum of the loss for quantized nets. We develop algorithms for an adaptive codebook or a (partially) fixed codebook. The latter includes binarization, ternarization, powers-of-two and other important particular cases. We show experimentally that we can achieve much higher compression rates than previous quantization work (even using just 1 bit per weight) with negligible loss degradation.

Abstract: The future wireless network, such as the Centralized Radio Access Network (C-RAN), will need to deliver data rate about 100–1000 times the current fourth-generation (4G) technology. For the C-RAN-based network architecture, there is a pressing need for tremendous enhancement of the effective data rate of the common public radio interface (CPRI). Compression of CPRI data is one of the potential enhancements. In this paper, we introduce a vector quantization based compression algorithm for CPRI links, utilizing the Lloyd algorithm. Methods to vectorize the I/Q samples and enhanced initialization of the Lloyd algorithm for codebook training are investigated for improved performance. Multistage vector quantization and unequally protected multigroup quantization are considered to reduce codebook search complexity and codebook size. Simulation results show that our solution can achieve compression of four times for uplink and 4.5 times for downlink, within $2\%$ error vector magnitude (EVM) distortion. Remarkably, vector quantization codebook proves to be quite robust against data modulation mismatch, fading, signal-to-noise ratio (SNR), and Doppler spread.

Abstract: This paper proposes a vector-quantization-based secret key generation (SKG) procedure to efficiently extract shared secret keys from correlated channel observations at two communicating terminals, Alice and Bob. Most existing SKG schemes utilize scalar quantization to extract secret key bits separately from each individual channel observation. This approach is simple to implement but yields higher key disagreement probability (or lower key entropy) compared with vector-quantization-based approaches. However, regardless of the quantizer design, quantization for SKG often suffers from the so-called cell-boundary problem, which occurs when the channel observations at Alice and Bob lie close to the quantization cell boundaries, resulting in high probability of key disagreement. In this paper, a general SKG procedure that utilizes sample and quantizer selection techniques to avoid this problem is first proposed. The vector quantizer adopted in the above procedure is designed by minimizing the quadratic distortion between the true channel vector and the noisy observation at Alice (or Bob). Then, by considering the case where the eavesdropper (Eve) may observe a channel vector that is correlated with that observed by Alice and Bob, a clustered key mapping scheme that assigns each secret key to multiple quantization cells in different clusters is also proposed to induce additional randomness at Eve and, thus, maintain high conditional key entropy. The effectiveness of the proposed schemes is demonstrated through computer simulations.

Abstract: Emotional identification is necessary for example in Brain Computer Interface (BCI) application and when emotional therapy and medical rehabilitation take place. Some emotional states can be characterized in the frequency of EEG signal, such excited, relax and sad. The signal extracted in certain frequency useful to distinguish the three emotional state. The classification of the EEG signal in real time depends on extraction methods to increase class distinction, and identification methods with fast computing. This paper proposed human emotion monitoring in real time using Wavelet and Learning Vector Quantization (LVQ). The process was done before the machine learning using training data from the 10 subjects, 10 trial, 3 classes and 16 segments (equal to 480 sets of data). Each data set processed in 10 seconds and extracted into Alpha, Beta, and Theta waves using Wavelet. Then they become input for the identification system using LVQ three emotional state that is excited, relax, and sad. The results showed that by using wavelet we can improve the accuracy of 72% to 87% and number of training data variation increased the accuracy. The system was integrated with wireless EEG to monitor emotion state in real time with change each 10 seconds. It takes 0.44 second, was not significant toward 10 seconds.

Abstract: In this paper, an optimized residual vector quantization-based approach is presented for improving the quality of vector quantization and approximate nearest neighbor search. The main contributions are as follows. Based on residual vector quantization (RVQ), a joint optimization process called enhanced RVQ (ERVQ) is introduced. Each stage codebook is iteratively optimized by the others aiming at minimizing the overall quantization errors. Thus, an input vector is approximated by its quantization outputs more accurately. Consequently, the precision of approximate nearest neighbor search is improved. To efficiently find nearest centroids when quantizing vectors, a non-linear vector quantization method is proposed. The vectors are embedded into 2-dimensional space where the lower bounds of Euclidean distances between the vectors and centroids are calculated. The lower bound is used to filter non-nearest centroids for the purpose of reducing computational costs. ERVQ is noticeably optimized in terms of time efficiency on quantizing vectors when combining with this method. To evaluate the accuracy that vectors are approximated by their quantization outputs, an ERVQ-based exhaustive method for approximate nearest neighbor search is implemented. Experimental results on three datasets demonstrate that our approaches outperform the state-of-the-art methods over vector quantization and approximate nearest neighbor search.

Abstract: Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction of the communication bottleneck in distributed DNN training and faster neural network implementations on hardware accelerators like FPGAs. Several quantization methods have been proposed to map the original 32-bit floating point problem to low-bit representations. While most related publications validate the proposed approach on a single DNN topology, it appears to be evident, that the optimal choice of the quantization method and number of coding bits is topology dependent. To this end, there is no general theory available, which would allow users to derive the optimal quantization during the design of a DNN topology.In this paper, we present a quantization tool box for the TensorFlow framework. TensorQuant allows a transparent quantization simulation of existing DNN topologies during training and inference. TensorQuant supports generic quantization methods and allows experimental evaluation of the impact of the quantization on single layers as well as on the full topology. In a first series of experiments with TensorQuant, we show an analysis of fix-point quantizations of popular CNN topologies.

Abstract: This paper presents a compact vector quantizer based on the self-organizing map (SOM), which can fulfill the data compression task for high-speed image sequence. In this vector quantizer, we solve the most severe computational demands in the codebook learning mode and the image encoding mode by a reconfigurable complete-binary-adder-tree (RCBAT), where the arithmetic units are thoroughly reused. In this way, the hardware efficiency of our proposed vector quantizer is greatly improved. In addition, by distributing the codebook into the multi-parallel processing sub-blocks, our design obtains a high compression speed successfully. Furthermore, a mechanism of partial vector-component storage (PVCS) is adopted to make the compression ratio adjustable. Finally, the proposed vector quantizer has been implemented on the field programmable gate array (FPGA). The experimental results indicate that it respectively achieves a compression speed of 500 frames/s and a million connections per second (MCPS) of 28,494 (compression ratio is 64) when working at 79.8 MHz. Besides, compared with the previous scheme, our proposed quantizer achieves a reduction of 8% in hardware usage and an increase of 33% in compression speed. This means the proposed quantizer is hardware-efficient and can be used for high-speed image compression.

Abstract: Product quantization (PQ) has been recognized as a useful technique to encode visual feature vectors into compact codes to reduce both the storage and computation cost. Recent advances in retrieval and vision tasks indicate that high-dimensional descriptors are critical to ensuring high accuracy on large-scale data sets. However, optimizing PQ codes with high-dimensional data is extremely time-consuming and memory-consuming. To solve this problem, in this paper, we present a novel PQ method based on bilinear projection, which can well exploit the natural data structure and reduce the computational complexity. Specifically, we learn a global bilinear projection for PQ, where we provide both non-parametric and parametric solutions. The non-parametric solution does not need any data distribution assumption. The parametric solution can avoid the problem of local optima caused by random initialization, and enjoys a theoretical error bound. Besides, we further extend this approach by learning locally bilinear projections to fit underlying data distributions. We show by extensive experiments that our proposed method, dubbed bilinear optimization product quantization, achieves competitive retrieval and classification accuracies while having significant lower time and space complexities.

Abstract: To address the challenging problem of vector quantization (VQ) for high dimensional vector using large coding bits, this work proposes a novel deep neural network (DNN) based VQ method. This method uses a k-means based vector quantizer as an encoder and a DNN as a decoder. The decoder is initialized by the decoder network of deep auto-encoder, fed with the codes provided by the k-means based vector quantizer, and trained to minimize the coding error of VQ system. Experiments on speech spectrogram coding demonstrate that, compared with the k-means based method and a recently introduced DNN-based method, the proposed method significantly reduces the coding error. Furthermore, in the experiments of coding multi-frame speech spectrogram, the proposed method achieves about 11% relative gain over the k-means based method in terms of segmental signal to noise ratio (SegSNR).

Abstract: In this brief, a class of weighted quantized kernel recursive least squares (WQKRLS) algorithms is proposed to efficiently improve the performance of online applications. In the proposed WQKRLS, an online vector quantization with weighted outputs is incorporated into quantized kernel recursive least squares. The resulting desired outputs are smoothed by exponential weights. In addition, the members of the dictionary are updated by the steepest descent method for further performance improvement. Simulations illustrate the superior performance of the proposed WQKRLS.

Abstract: This paper investigates the development of language identification based on Mel-Frequency Cepstral Coefficients (MFCC) and Vector Quantization (VQ) algorithm. In this study, a total of ten speakers were chosen randomly with different languages from online language database. A total of six males and four females were selected as subjects for this research and each of them spoke different languages, including Arabic, Chinese, English, Korean and Malay. The MFCC will be extracted to derive the related feature vector. Vector Quantization (VQ) algorithm is then used as classifier. The recognition rate is then calculated for each language. Several experiments were conducted to find the optimum parameters, in which we found that sampling frequency of 16000 Hz and codebook size of 75 provided good results. On average, the recognition rate for all five languages evaluated was 78%. The experimental results show that our proposed system provides a good recognition rate.

Abstract: This paper proposes a method based on the bag-of-words (BoW) and the softmax regression for microscopic image classification Essentially, the locality-constrained linear coding (LLC) is adopted for local feature encoding Compared with the traditionally adopted vector quantization (VQ) in the BoW framework, the LLC encodes local structures of microscopic images with lower quantization errors and generates a sparse image representation This enables the use of linear classifiers with low computational complexity A softmax regression classifier is then adopted to address the multi-categorical classification task where the confidence of categorical prediction is quantified by posterior probabilities Compared with other linear classifiers (such as the linear SVM) which only assign labels to images, such probabilistic outputs provide extra quantitative information to analyze misclassified images Our experiments on the 2D-Hela and the PAP smear data sets show significant performance improvement of the proposed method comparing with competing methods using different features and classifiers under the BoW framework

Abstract: Speech recognition is a process to identify the speaker on the basis of individual information within the speech wave Recent development has made the voice recognition in the security system In this paper the implementation of speech digit recognition system is discussed This technique is mainly used in person voice identification and control access like banking by telephone, voice dialing and database access services The zero to nine digit utterances for speech data was collected The speech digit recognition mainly involves two parts, one is the feature extraction and other one is the feature matching The main approach is to isolate the speech recognition by Cepstrum and vector quantization Cepstrum technique is used for feature extraction and vector quantization is used for feature matching The result show that all digit gives good performance The proposed speech digit recognition algorithm is implemented by using MATLAB software