Research On Continual Learning Method Based On Generic Feature Representations

Most of the current deep neural network training methods follow a single-learning paradigm,which cannot accumulate knowledge incrementally like humans through continuously incoming data.In order to solve this problem and achieve more general artificial intelligence,continual learning has received widespread attention in recent years.Continual learning aims to continuously learn new classes while retaining the knowledge of old classes with sequentially incoming data.Existing methods for continual learning fail to learn discriminative representations for both old and new classes simultaneously due to the lack of old-class samples,resulting in severe performance degradation in old classes,known as the catastrophic forgetting phenomenon.Existing works have mainly focused on improving distillation losses or more precisely protecting important parameters.In contrast,this thesis proposes to facilitate continual learning from learning generic feature representations that maintain enough discrimination while being well shared by both seen and unseen classes,thus reducing forgetting.To this end,this thesis first explores from the perspective of singular value analysis what kind of feature representations benifits continual learning.Then,using widely adopted evaluation metrics in continual learning,including accuracy,forgetting rate,and average incremental accuracy,we propose and prove that favorable feature representations for continual learning should have a larger number of significant singular values to reduce forgetting,while avoiding excessively uniform singular value distributions to ensure classification discriminability.Moreover,this thesis proposes three mechanisms to obtain the expected singular value distribution,thereby improving the performance of continual learning and validating the proposed hypothesis.To be more specific,the following work has been done:(1)This thesis proposes and proves the hypothesis that feature representations beneficial for continual learning shoule possess a considerable number of significant singular values to mitigate forgetting,while avoiding an overly uniform singular value spectrum to maintain discrimination.Furthermore,the thesis proposes that a wider embedding vector would enlarge the number of significant singular values.(2)Considering the paradigm of classification models,this thesis expands the embedding vector in two ways,namely constructing a wider network(WidNet)or introducing a replacement pooling technique(generic feature representation regulation module,GFR).Both proposed mechanisms obtain the desired singular value distribution and improve performance,thus validating the core hypothesis.(3)The previous two proposed methods inevitably increase memory costs.To overcome this issue,this thesis proposes the generic feature representation regulation loss(GfrLoss),which improves performance at a much lower cost,while also providing validation for the core hypothesis from another perspective.Finally,the combination of WidNet and GfrLoss achieves better performance than either WidNet or GfrLoss alone.