Essentials for Class Incremental Learning
Sudhanshu Mittal,Silvio Galesso,Thomas Brox +2 more
- 04 May 2021
- pp 3513-3522
TL;DR: In this paper, a combination of simple components and a loss that balances intra-task and inter-task learning can resolve forgetting to the same extent as more complex measures proposed in literature, and they identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL.
read more
Abstract: Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and intertask learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate regularizer. With these lessons learned, class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Efficient Test-Time Model Adaptation without Forgetting
Shuai Niu,Jiaxiang Wu,Yifan Zhang,Yaofo Chen,Shi Dong Zheng,Peilin Zhao,Mingkui Tan +6 more
- 06 Apr 2022
TL;DR: An active sample selection criterion is proposed to identify reliable and non-redundant samples, on which the model is updated to minimize the entropy loss for test-time adaptation, and a Fisher regularizer is introduced to constrain important model parameters from drastic changes.
194
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning
Yabin Wang,Zhiwu Huang,Xiaopeng Hong +2 more
- 26 Jul 2022
TL;DR: This paper proposes one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL).
125
Deep Learning in Cancer Diagnosis and Prognosis Prediction: A Minireview on Challenges, Recent Trends, and Future Directions.
Ahsan Bin Tufail,Ahsan Bin Tufail,Yong-Kui Ma,Mohammed K. A. Kaabar,Francisco Martínez,A R Junejo,Inam Ullah,Rahim Khan +7 more
TL;DR: Deep learning (DL) is a branch of machine learning and artificial intelligence that has been applied to many areas in different domains such as health care and drug design as mentioned in this paper, which has emerged as a technology of choice due to the availability of high computational resources.
Class-Incremental Continual Learning into the eXtended DER-verse
TL;DR: In this article , the authors propose an approach that combines rehearsal and knowledge distillation to learn a sequence of tasks incrementally, blending sequentially-gained knowledge into a comprehensive prediction.
52
•Proceedings Article
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration Without Forgetting
Anna Kukleva,Hilde Kuehne,Bernt Schiele +2 more
- 18 Aug 2021
TL;DR: In this article, a three-stage framework is proposed to explicitly and effectively address the challenges of generalized and incremental few-shot learning, where the first phase learns base classes with many samples, the second phase learns a calibrated classifier for novel classes from few samples, and the final phase, calibration is achieved across all classes.
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Posted Content
Deep Residual Learning for Image Recognition
TL;DR: This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
117.9K
ImageNet: A large-scale hierarchical image database
Jia Deng,Wei Dong,Richard Socher,Li-Jia Li,Kai Li,Li Fei-Fei +5 more
- 20 Jun 2009
TL;DR: A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
Rethinking the Inception Architecture for Computer Vision
Christian Szegedy,Vincent Vanhoucke,Sergey Ioffe,Jonathon Shlens,Zbigniew Wojna +4 more
- 27 Jun 2016
TL;DR: In this article, the authors explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization.
27.9K
•Dissertation
Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky
- 01 Jan 2009
TL;DR: In this paper, the authors describe how to train a multi-layer generative model of natural images, using a dataset of millions of tiny colour images, described in the next section.