Article Details

Abstract

In few-shot image classification, single knowledge distillation strategies tend to introduce knowledge transfer bias, limiting student model performance. To address this issue, this paper proposes a joint distillation method based on Prototypical Networks. Specifically, ResNet18 serves as the teacher model, achieving a classification accuracy of 67.53% on the CIFAR-FS dataset under the 5-way 1-shot setting. The student model employs a lightweight 4-layer residual network with a baseline accuracy of 54.13%. Comparative experiments across three distillation strategies show that hard-label distillation with a confidence filtering mechanism improves the student model accuracy to 56.89%. Soft-label distillation further enhances performance to 59.36% by transferring inter-class similarity information. The proposed joint distillation strategy combines the strengths of both approaches and introduces dynamic weight adjustment, achieving an accuracy of 62.20%—an improvement of 8.07 percentage points over the baseline. Meanwhile, the performance gap between the teacher and student models narrows from 13.40% to 5.33%. In conclusion, the proposed joint distillation strategy effectively leverages the teacher model's knowledge potential, alleviates knowledge transfer bias, and demonstrates significant advantages in few-shot learning scenarios.

Keywords

Few-Shot Learning; knowledge distillation; distillation strategy comparison; prototypical network

References

1. [1] Zhao Kailin, Jin Xiaolong, Wang Yuanzhuo A Review of Small Sample Learning Research [J]. Journal of Software, 2021, 32 (2): 349-369
2. [2] Hinton G，Vinyals O，Dean J. Distilling the Knowledge in a Neural Network [J/OL].arXiv:1503.02531 [stat.ML].https://doi.org/10.48550/arXiv.1503.02531.
3. [3] Snell J，Swersky K，Zemel R S. Prototypical Networks for Few-shot Learning [C]//NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems，2017：4080-4090.
4. [4] Shi Ling, He Changle, Chang Baofang, etc A knowledge distillation based edge federated learning algorithm [J]. Journal of Zhengzhou University (Science Edition), 2025, 57 (2): 44-50
5. [5] Zhou H L，Song L C，Chen J J，et al. Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective [J/OL].arXiv:2102.00650 [cs.LG].https://doi.org/10.48550/arXiv.2102.00650.
6. [6] Bertinetto L，Henriques J F，Torr P H S，et al. Meta-learning with Differentiable Closed-form Solvers [J/OL].arXiv:1805.08136 [cs.CV].https://doi.org/10.48550/arXiv.1805.08136.
7. [7] Wang Shouxiang, Zhao Ning, Wang Tongxun, etc Lightweight identification method for power quality disturbances based on trajectory color coding and knowledge distillation [J]. Electric Power Automation Equipment, 2024, 44 (12): 85-91
8. [8] Wang Hang, Tian Shengzhao, Tang Qing, etc Small sample image classification based on multi-scale label propagation [J]. Computer Research and Development, 2022, 59 (7): 1486-1495
9. [9] Loshchilov I，Hutter F. Decoupled Weight Decay Regularization [J/OL].arXiv:1711.05101 [cs.LG].https://doi.org/10.48550/arXiv.1711.05101.
10. [10] Tang Zhenchao, Wei Wei, Luo Weiran, etc Fusion of cosine annealing and dilated convolution for semantic segmentation of remote sensing images [J]. Journal of Remote Sensing, 2023, 27 (11): 2579-2592

About the authors

Wang Yan (1994-), female, Han ethnicity, born in the year of Xiaoyi Shanxi, teaching assistant, master's student, research direction: computer vision, signal processing.