ICAF:具有插值一致性和自适应筛选的无监督持续学习
(1.交通数据分析与挖掘北京市重点实验室(北京交通大学),北京 100044; 2.北京交通大学 计算机科学与技术学院,北京 100044)
摘要
无监督持续学习解决了在没有监督信息的情况下按顺序学习任务的挑战。然而,这种设置往往会导致灾难性的遗忘。为了克服这个问题,使用记忆模块来存储过去任务的样本。但是存储和重放合适的样本是一项复杂的任务,简单而常用的随机选择并不有效,尤其是对于具有大规模样本的长序列任务。为了解决这个问题,一种具有插值一致性和自适应记忆筛选的无监督持续学习(ICAF)被提出,它可以利用随机增强样本中学习的特征来有效地选择高质量样本,在训练过程中调整阈值,并通过专门设计的线性插值和数据组合策略来获取插值一致性,从而防止仅依赖于过去任务中的插值样本而导致的学习不足。在多个实验集上,该方法实现了最好的效果。
关键词
持续学习;无监督学习;表示学习;对比学习;自监督学习
参考文献
- [1] DELANGE M,ALJUNDI R,MASANA M,et al. A Continual Learning Survey: Defying Forgetting in Classification Tasks [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(7):3366-3385.
- [2] MICHIELI U,ZANUTTIGH P. Incremental Learning Techniques for Semantic Segmentation [C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).Seoul:IEEE,2019:3205-3212.
- [3] HADSELL R,RAO D,RUSU A A,et al. Embracing Change:Continual Learning in Deep Neural Networks [J].Trends in Cognitive Sciences,2020,24(12):1028-1040.
- [4] BIESIALSKA M,BIESIALSKA K,COSTA-JUSSA M R. Continual Lifelong Learning in Natural Language Processing: A Survey [C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona:ACL,2020:6523-6541.
- [5] ALJUNDI R,LIN M,GOUJAUD B,et al. Gradient Based Sample Selection for Online Continual Learning [C]//Advances in Neural Information Processing Systems.Red Hook:Curran Associates,Inc.,2019,32[2025-01-03].https://proceedings.neurips.cc/paper_files/paper/2019/file/e562cd9c0768d5464b64cf61da7fc6bb-Paper.pdf.
- [6] BUZZEGA P,BOSCHINI M,PORRELLI A,et al. Dark Experience for General Continual Learning: A Strong, Simple Baseline [C]//Advances in Neural Information Processing Systems.Red Hook:Curran Associates,Inc.,2020,33:15920-15930.
- [7] CHAUDHRY A,RANZATO M A,ROHRBACH M,et al. Efficient Lifelong Learning with A-GEM [C]//International Conference on Learning Representations.New Orleans:Cambridge University Press,2019:1-20.
- [8] RUSU A A,RABINOWITZ N C,DESJARDINS G,et al. Progressive Neural Networks [J/OL].arXiv:1606.04671 [cs.LG].(2022-10-22)https://arxiv.org/abs/1606.04671.
- [9] YU X,GUO Y,GAO S,et al. SCALE: Online Self-Supervised Lifelong Learning Without Prior Knowledge [J/OL].arXiv:2208.11266v5 [cs.LG].(2022-08-24).https://arxiv.org/pdf/2208.11266.
- [10] ZENKE F,POOLE B,GANGULI S. Continual Learning through Synaptic Intelligence [C]//International Conference on Machine Learning.Sydney:PMLR,2017:3987-3995.
- [11] HAO J,JI K,LIU M. Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm [EB/OL].(2023-11-03).https://openreview.net/forum?id=2dtU9ZbgSN.
- [12] MADAAN D,YOON J,LI Y,et al. Representational Continuity for Unsupervised Continual Learning [J/OL].arXiv:2110.06976 [cs.LG].(2021-10-13).https://arxiv.org/abs/2110.06976.
- [13] SOHN K,BERTHELOT D,CARLINI N,et al. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence [J/OL].arXiv:2001.07685 [cs.LG].[2024-12-23].https://doi.org/10.48550/arXiv.2001.07685.
- [14] NIU C,SHAN H,WANG G. SPICE: Semantic Pseudo-Labeling for Image Clustering [J].IEEE Transactions on Image Processing,2022,31:7264-7278.
- [15] CHEN X,HE K. Exploring Simple Siamese Representation Learning [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville:IEEE,2021:15750-15758.
- [16] ZHANG H,CISSE M,DAUPHIN Y N,et al. Mixup: Beyond Empirical Risk Minimization [J/OL].arXiv:1710.09412 [cs.LG].(2018-04-27).https://arxiv.org/abs/1710.09412.
- [17] WANG L,ZHANG X,SU H,et al. A Comprehensive Survey of Continual Learning: Theory, Method and Application [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2024,46(8):5362-5383.
- [18] ARAZO E,ORTEGO D,ALBERT P,et al. Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning [C]//2020 International Joint Conference on Neural Networks (IJCNN).Glasgow:IEEE,2020:1-8.
- [19] KRIZHEVSKY A,HINTON G. Learning Multiple Layers of Features from Tiny Images [EB/OL].[2024-09-16].https://api.semanticscholar.org/CorpusID:18268744.
- [20] DENG J,DONG W,SOCHER R,et al. ImageNet: A Large-Scale Hierarchical Image Database [C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami:IEEE,2009:248-255.
- [21] HE K,ZHANG X,REN S,et al. Deep Residual Learning for Image Recognition [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:770-778.
- [22] WU Z,XIONG Y,YU S X,et al. Unsupervised Feature Learning via Non-Parametric Instance Discrimination [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City:IEEE,2018:3733-3742.
- [23] BORSOS Z,MUTNY M,KRAUSE A. Coresets via Bilevel Optimization for Continual Learning and Streaming [J/OL].arXiv:2006.03875 [cs.LG].(2020-10-22).https://arxiv.org/abs/2006.03875.
- [24] TIWARI R,KILLAMSETTY K,IYER R,et al. GCR: Gradient Coreset Based Replay Buffer Selection for Continual Learning [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans:IEEE,2022:99-108.
- [25] ZHOU X,PI R,ZHANG W,et al. Probabilistic Bilevel Coreset Selection [C]//International Conference on Machine Learning.PMLR,2022:27287-27302.
- [26] YOON J,MADAAN D,YANG E,et al. Online Coreset Selection for Rehearsal-Based Continual Learning [J/OL].arXiv:2106.01085 [cs.LG].(2022-05-18).https://arxiv.org/abs/2106.01085.
基金项目
中央高校基本科研业务费专项资金(20119jbz110);国家自然科学基金项目(62176020);国家重点研发计划项目(2020AAA0106800);北京市自然科学基金项目L211016;中国科学院(OEIP-O-202004)
作者简介
叶根诚(2000—),男,汉族,河北邯郸人,研究生在读,研究方向:聚类、持续学习、无监督学习。