Deep Error-Correcting Output Codes

TL;DR: This paper combines the ideas of ensemble learning and deep learning, and presents a novel deep learning framework called deep error-correcting output codes (DeepECOC), which performs not only better than traditional ECOC and feature learning algorithms, but also state-of-the-art deep learning models in most cases.

Abstract: Existing deep networks are generally initialized with unsupervised methods, such as random assignments and greedy layerwise pre-training. This may result in the whole training process (initialization/pre-training + fine-tuning) to be very time consuming. In this paper, we combine the ideas of ensemble learning and deep learning, and present a novel deep learning framework called deep error-correcting output codes (DeepECOC). DeepECOC are composed of multiple layers of the ECOC module, which combines multiple binary classifiers for feature learning. Here, the weights learned for the binary classifiers can be considered as weights between two successive layers, while the outputs of the combined binary classifiers as the outputs of a hidden layer. On the one hand, the ECOC modules can be learned using given supervisory information, and on the other hand, based on the ternary coding design, the weights can be learned only using part of the training data. Hence, the supervised pre-training of DeepECOC is in general very effective and efficient. We have conducted extensive experiments to compare DeepECOC with traditional ECOC, feature learning and deep learning algorithms on several benchmark data sets. The results demonstrate that DeepECOC perform not only better than traditional ECOC and feature learning algorithms, but also state-of the-art deep learning models in most cases.