Continual Learning and Catastrophic Forgetting

A lecture that discusses continual learning and catastrophic forgetting in deep neural networks. We discuss the context, methods for evaluating algorithms, and algorithms based on regularization, dynamic architectures, and Complementary Learning Systems. Specifically, we discuss data permutation tasks, incremental task learning, multimodal learning, the Learning without Forgetting algorithm, Elastic Weight Consolidation, Progressive Neural Networks, and Generative replay.

This lecture is from Northeastern University’s CS 7150 Summer 2020 class on Deep Learning, taught by Paul Hand.

The notes are available at: http://khoury.northeastern.edu/home/hand/teaching/cs7150-summer-2020/Continual_Learning_and_Catastrophic_Forgetting.pdf

References:

Parisi et al. 2019:

Parisi, German I., Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. "Continual lifelong learning with neural networks: A review." Neural Networks (2019).

Chen and Liu 2018:

Chen, Zhiyuan, and Bing Liu. "Lifelong machine learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 12, no. 3 (2018): 1-207. Chapter 4.

Kemker et al. 2017:

Kemker, Ronald, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. "Measuring catastrophic forgetting in neural networks." In Thirty-second AAAI conference on artificial intelligence. 2018.

van de Ven and Tolias 2019:

van de Ven, Gido M., and Andreas S. Tolias. "Three scenarios for continual learning." arXiv preprint arXiv:1904.07734 (2019).

Li and Hoiem 2018:

Li, Zhizhong, and Derek Hoiem. "Learning without forgetting." IEEE transactions on pattern analysis and machine intelligence 40, no. 12 (2017): 2935-2947.

Kirkpatrick et al. 2017:

Kirkpatrick, James, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114, no. 13 (2017): 3521-3526.

Rusu et al. 2016:

Rusu, Andrei A., Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016).

Shin et al. 2017:

Shin, Hanul, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. "Continual learning with deep generative replay." In Advances in Neural Information Processing Systems, pp. 2990-2999. 2017.