Teacher-student collaborative knowledge distillation for image classification
TL;DR: Li et al. as discussed by the authors proposed a teacher-student collaborative knowledge distillation (TSKD) method, which consists of two parts: learning in the teacher network and self-teaching in the student network.
read more
Abstract: A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve model performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on knowledge distillation and self-distillation. The method consists of two parts: learning in the teacher network and self-teaching in the student network. Learning in the teacher network allows the student network to use knowledge from the teacher network. Self-teaching in the student network is to build a multi-exit network based on self-distillation and provide deep features as supervised information for training. In the inference stage, we use ensembles to vote on the classification results of multiple sub-models in the student network. The experimental results demonstrate the superior performance of our method compared with a traditional knowledge distillation method and a self-distillation-based multi-exit network.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Adversarial Attacks and Defenses in Machine Learning-Empowered Communication Systems and Networks: A Contemporary Survey
Yulong Wang,Tong Sun,Shenghong Li,Xinnan Yuan,W. Ni,Ekram Hossain,H. Vincent Poor +6 more
TL;DR: A comprehensive classification of recent adversarial attack methods and state-of-the-art adversarial defense techniques based on attack principles are conducted, and they are presented in visually appealing tables and tree diagrams.
43
A Cooperative Vehicle-Infrastructure System for Road Hazards Detection With Edge Intelligence
Chen Chen,Guo-chun Yao,Lei Liu,Qingqi Pei,Houbing Song,Schahram Dustdar +5 more
- 01 May 2023
TL;DR: In this article , a road hazards detection (RHD) solution based on cooperative vehicle-infrastructure systems (CVIS) is proposed, which uses a meta-learning paradigm, critical features are generalized from a few-shot RH data.
22
Efficient Image Classification through Collaborative Knowledge Distillation: A Novel AlexNet Modification Approach
Avazov Kuldashboy,Sabina Umirzakova,Sharofiddin Allaberdiev,Rashid Nasimov,Akmalbek Abdusalomov,Young Im Cho +5 more
11
Cervical Cell Image Classification-Based Knowledge Distillation
Vicky Ani,Muhammad Mohsin Waqas +1 more
TL;DR: Wang et al. as mentioned in this paper introduced a multi-exit classification network, where a global context module is embedded in each exit branch, and a self-distillation method is then proposed to fuse contextual information; deep classifiers in the student network guide shallow classifiers to learn, and multiple classifier outputs are fused using an average integration strategy to form a classifier with strong generalization performance.
6
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Posted Content
Deep Residual Learning for Image Recognition
TL;DR: This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
117.9K
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
102.6K
ImageNet classification with deep convolutional neural networks
TL;DR: A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
ImageNet: A large-scale hierarchical image database
Jia Deng,Wei Dong,Richard Socher,Li-Jia Li,Kai Li,Li Fei-Fei +5 more
- 20 Jun 2009
TL;DR: A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.