Open AccessPosted Content
Continual Learning Through Synaptic Intelligence
TL;DR: In this paper, the authors introduce intelligent synapses that bring some of this biological complexity into artificial neural networks, and evaluate their approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.
read more
Abstract: While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Hyper-parameter Tuning for Progressive Learning and its Application to Network Cyber Security
Rupesh Raj Karn,Matthew Ziegler,Jin Wook Jung,Ibrahim M. Elfadel +3 more
- 28 May 2022
TL;DR: In this article , a hyper-parameter optimization framework is proposed that selects the best hyperparameter values on a task-by-task basis for each progressive learning task by adjusting the hyperparameters under which the neural architecture is incrementally grown.
2
Streaming LifeLong Learning With Any-Time Inference
Soumya Banerjee,Vinay Kumar Verma,Vinay P. Namboodiri +2 more
- 27 Jan 2023
TL;DR: In this article , the authors propose a single pass, class-incremental, and any-time inference approach, where a single input sample arrives in each time step, and the proposed approach is subject to be evaluated at any moment.
2
•Posted Content
Multilayer Neuromodulated Architectures for Memory-Constrained Online Continual Learning
TL;DR: An architecture where input processing over data streams and online learning are integrated in a single recurrent network architecture is developed, which allows metalearning optimization as a mixed-integer optimization problem, where different synaptic plasticity algorithms and feature extraction layers can be swapped out and their hyperparameters are optimized to identify optimal architectures for different sets of tasks.
Progressive Latent Replay for efficient Generative Rehearsal
Stanislaw Pawlak,Filip Szatkowski,Michał Bortkiewicz,Jan Dubi'nski,T. Trzci'nski +4 more
- 04 Jul 2022
TL;DR: A new method for internal replay is introduced that mod-ulates the frequency of rehearsal based on the depth of the network, motivated by the observation that earlier layers of neural networks forget less abruptly.
2
Incremental Learning with Differentiable Architecture and Forgetting Search
James Smith,Zachary Seymour,Han-Pang Chiu +2 more
- 19 May 2022
TL;DR: This paper creates a strong baseline approach for incremental learning based on Differentiable Architecture Search (DARTS) and state-of-the-art incremental learning strategies, and extends the idea of architecture search to regularize architecture forgetting, boosting performance past the proposed baseline.
2
References
•Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
- 01 Jan 2015
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
138.5K
•Posted Content
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: In this article, the adaptive estimates of lower-order moments are used for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimate of lowerorder moments.
82.5K
•Dissertation
Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky
- 01 Jan 2009
TL;DR: In this paper, the authors describe how to train a multi-layer generative model of natural images, using a dataset of millions of tiny colour images, described in the next section.
•Posted Content
Distilling the Knowledge in a Neural Network
TL;DR: This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse.
21.2K
The perceptron: a probabilistic model for information storage and organization in the brain.
TL;DR: This article will be concerned primarily with the second and third questions, which are still subject to a vast amount of speculation, and where the few relevant facts currently supplied by neurophysiology have not yet been integrated into an acceptable theory.
10.6K