> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/continual-learning/icarl-incremental-classifier-and-representation-learning.md).

# iCaRL: Incremental Classifier and Representation Learning

## Summary

New training strategy to learning in class-incremental setting without catastrophic forgetting. Only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. 

It simultaneously learns classifiers and feature representations in the class-incremental setting. 

Conponents:

* classification by a nearest-mean-of-examplars rule
* prioritized exemplar selection based on herding.
* representation learning using knowledge distillation and prototype rehearsal. 

## Background

**Class Incremental Setting**: When stream of data is provided to train and different classes occur at different time. Computational requirement and memory footprint should remain bounded. 

**Catastrophic forgetting:** When naively use SGD for training in class incremental training, the classification accuracy deteriorates as more classed comes, because network forgets about previous classes seen during training. 

## Methodology

**Classification:** sets $$P1, ...P\_t$$of exemplar images, each set corresponding to each class of image. Total number of exemplar images never exceeds parameter K. 

**mean-of-exemplars classifier:**

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-MSQSJ2bUdU2p_76gOCm%2F-MSQh_q4Fc8Yx5eet661%2Fimage.png?alt=media\&token=06793cae-00f1-4bed-ab62-f9afb50a0a62)

**Training:** This procedure determines the update of network weights and exemplar sets, etc. 

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-MSQSJ2bUdU2p_76gOCm%2F-MSQiMyAn5yNSO1jnggD%2Fimage.png?alt=media\&token=5de8e699-822f-4841-b5bd-5e92555e224a)

**Architecture:** We interpret the network as a trainable feature extractor, ϕ : X → Rd, fol- lowed by a single classification layer with as many sigmoid output nodes as classes observed so far. All feature vectors are L2-normalized, and the results of any operation on feature vectors, e.g. averages, are also re-normalized, which we do not write explicitly to avoid a cluttered notation.