Embracing Change: Continual Learning in Deep Neural Networks

TL;DR: In this article , the authors describe three fundamental types of continual learning: task-incremental, domain-increasing and class-increasing learning, and provide a comprehensive empirical comparison of currently used continual learning strategies.

TL;DR: DeBERTaV3 as mentioned in this paper improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task.

TL;DR: In this paper , the authors propose to dynamically prompt a pre-trained model to learn tasks sequen-tially under different task transitions using small learnable parameters, which are maintained in a memory space.

TL;DR: A comprehensive survey of continual learning can be found in this article , where the authors summarize the general objectives of continuous learning as ensuring a proper stability-plasticity trade-off and an adequate intra/inter-task generalizability in the context of resource efficiency.

TL;DR: A solution to the problem of catastrophic forgetting in neural networks is described, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself.

TL;DR: In this article, a data-dependent latent generative representation of model parameters is learned and a gradient-based meta-learning is performed in a low-dimensional latent space for few-shot learning.

TL;DR: This account helps to explain why sleep, alcohol, and benzodiazepines all improve memory for a recently learned list, and it is consistent with recent work on the variables that affect the induction and maintenance of long-term potentiation in the hippocampus.

TL;DR: An improved version of GEM is proposed, dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance as GEM, while being almost as computationally and memory efficient as EWC and other regularization-based methods.