Article Details

Title · Authors · Abstract · Keywords · DOI · References

A Feature-Alignment-Driven CNN Lightweight Distillation Framework for Transformer-Based Text Processing

PAN Jiacong, CHAI Bencheng, YE Jiahui, DU Chenjie, CHEN Yi

(College of Big Data and Software Engineering, Zhejiang Wanli University, Ningbo 315100, China)

Section：Artificial Intelligence · Vol. 2026, Issue 15 · Pages：12-17 · 2096-4706（2026）15-0012-06 · CLC number：TP391 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2026.15.003

Abstract

With the rapid development of Natural Language Processing (NLP) technologies, improving model inference efficiency and reducing computational costs have drawn increasing attention. In text classification and analysis tasks, NLP models based on the Transformer framework often face challenges such as large model size, high computational demand, and expensive deployment. To address these issues, this study explores distillation and quantization methods for Transformer-based NLP models, specifically BERT, and proposes a feature-alignment-driven lightweight CNN distillation framework. A specially designed convolutional structure serves as the student model, while the widely used BERT-base acts as the teacher model. The framework conducts distillation training for text classification and analysis tasks. After training, the parameter size of the student model in this framework is reduced by approximately 77% compared to that of the teacher model, achieving an average accuracy of 79.6% of the teacher model, while its inference speed is faster and more efficient than a BERT-small model with a similar parameter size.

Keywords

text classification; text analysis; Transformer; distillation and quantization; feature alignment; lightweight

References

  1. [1] Okazaki N，Hattori K，Shota H，et al. Building a Large Japanese Web Corpus for Large Language Models [J/OL].arXiv:2404.17733 [cs.CL].https://arxiv.org/abs/2404.17733.
  2. [2] Peters M E，Neumann M，Iyyer M，et al. Deep Contextualized Word Representations [C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018)，2018，1：2227-2237.
  3. [3] Vaswani A，Shazeer N，Parmar N，et al. Attention is All You Need [J/OL].arXiv:1706.03762 [cs.CL].https://arxiv.org/abs/1706.03762.
  4. [4] Zhou C，Li Q，Li C，et al. A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT [J/OL].arXiv:2302.09419 [cs.AI].https://arxiv.org/abs/2302.09419.
  5. [5] Cañete J，Donoso S，Bravo-Marquez F，et al. ALBETO and DistilBETO: Lightweight Spanish Language Models [J/OL].arXiv:2204.09145 [cs.CL].https://arxiv.org/abs/2204.09145.
  6. [6] Chia Y K，Witteveen S，Andrews M. Transformer to CNN: Label-scarce Distillation for Efficient Text Classification [J/OL].arXiv:1909.03508 [cs.LG].https://arxiv.org/abs/1909.03508.
  7. [7] Zhai F Y，Wang R Z，Li P J. Knowledge distillation of Heterogeneous Teacher-student Models with Intermediate Layer Loss [C]//Proceedings of the 23rd Chinese National Conference on Computational Linguistics，2024，1：910-928.
  8. [8] Liu Y F，Cao J J，Li B，et al. Cross-architecture Knowledge Distillation [J/OL].arXiv:2207.05273 [cs.CV].https://doi.org/10.48550/arXiv.2207.05273.
  9. [9] Hao Z W，Guo J Y，Han K，et al. One-for-all: Bridge the Gap between Heterogeneous Architectures in Knowledge Distillation [J/OL].arXiv:2310.19444 [cs.CV].https://arxiv.org/abs/2310.19444.
  10. [10] Hinton G，Vinyals O，Dean J. Distilling the Knowledge in a Neural Network [J/OL].arXiv:1503.02531 [stat.ML].https://arxiv.org/abs/1503.02531.
  11. [11] Jiao X Q，Yin Y C，Shang L F，et al. TinyBERT: Distilling BERT for Natural Language Understanding [C]//Findings of the Association for Computational Linguistics: EMNLP 2020，2020：4163-4174.
  12. [12] Sanh V，Debut L，Chaumond J，et al. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter [J/OL].arXiv:1910.01108 [cs.CL].https://arxiv.org/abs/1910.01108.
  13. [13] Agarwal R，Vieillard N，Zhou Y C，et al. On-policy Distillation of Language Models: Learning from Self-Generated [J/OL].arXiv:2306.13649 [cs.LG].https://arxiv.org/abs/2306.13649.
  14. [14] Kullback S，Leibler R A. On Information and Sufficiency [J].The Annals of Mathematical Statistics，1951，22（1）：79-86.
  15. [15] Minaee S，Mikolov T，Nikzad N，et al. Large language Models: A Survey [J/OL].arXiv: 2402.06196 [cs.CL].https://arxiv.org/abs/2402.06196.
  16. [16] Liang C，Jiang H M，Li Z，et al. HomoDistil: Homotopic Task-agnostic Distillation of Pre-trained Transformers [J/OL].arXiv:2302.09632 [cs.CL].https://arxiv.org/abs/2302.09632.
  17. [17] Huang W，Peng Z L，Dong L，et al. Generic-to-specific Distillation of Masked Autoencoders [J/OL].arXiv:2302.14771 [cs.CV].https://arxiv.org/abs/2302.14771.
  18. [18] Touvron H，Cord M，Douze M，et al. Training Data-efficient Image Transformers and Distillation through Attention [J/OL].arXiv:2012.12877 [cs.CV].https://arxiv.org/abs/2012.12877.
  19. [19] Zhang J，Tao Z，Guo K H，et al. Hybrid Mix-up Contrastive Knowledge Distillation [J/OL].Information Sciences，2024，660：120107.https://doi.org/10.1016/j.ins.2024.120107.
  20. [20] Kim J，Lee H，Woo S S. IMF: Integrating Matched Features Using Attentive Logit in Knowledge Distillation [C]//Proceedings of the 32nd International Joint Conference on Artificial Intelligence，2023：974-982.https://doi.org/10.24963/ijcai.2023/108.
  21. [21] Mugisha S，Kisitu R，Tushabe F. Hybrid Knowledge Transfer through Attention and Logit Distillation for On-device Vision Systems in Agricultural IoT [J/OL].arXiv:2504.16128 [cs.CV].https://arxiv.org/abs/2504.16128.
  22. [22] Wang A，Singh A，Michael J，et al. GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding [C]//Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP，2018：353-355.
  23. [23] Martes D O，Gunderson E，Neuman C，et al. Transformer Models for Paraphrase Detection：A Comprehensive Semantic Similarity Study [J/OL].Computers，2025，14（9）：385.https://doi.org/10.3390/computers14090385.
  24. [24] Muhetaer P，Guo W Q，Lu C. A Language Model-based Approach to Sentiment Classification of Languages in Central Asia [J/OL].Scientific Reports，2025，15（1）：38392.https://link.springer.com/article/10.1038/s41598-025-22198-6.
  25. [25] Verma D，Lal Y K，Sinha S，et al. Evaluating Paraphrastic Robustness in Textual Entailment Models [C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics，2023，2：880-892.
  26. [26] Ratan S，Knight F，Jerfel G，et al. Optimizing Language Models for Grammatical Acceptability: A Comparative Study of Fine-tuning Techniques [J/OL].arXiv:2501.07853 [cs.CL].https://arxiv.org/abs/2501.07853.
  27. [27] Gou J P，Yu B S，Maybank S J，et al. Knowledge Distillation: A Survey [J].International Journal of Computer Vision，2021，129：1789-1819.

Funding

Open Project of the National Key Laboratory of Computer Aided Design and Graphic Systems, Zhejiang University (A2423); General Project of Zhejiang Provincial Department of Education (Y202351320); Natural Science Foundation of Ningbo City (2024J204); 2024 College Student Innovation and Entrepreneurship Training Program (S202410876069)

About the authors

Pan Jiacong (October 2003-), male, Han ethnicity, from Jiaxing, Zhejiang, is currently pursuing a bachelor's degree with a research focus on computer science and technology; Chai Bencheng (1978.11-), male, Han ethnicity, from Zhangqiu, Shandong, associate professor, master's degree, research direction: image processing and pattern recognition; Ye Jiahui (March 2006-), male, Han ethnicity, from Wenzhou, Zhejiang Province, is currently pursuing a bachelor's degree with a research focus on computer science and technology. Du Chenjie (April 1990-), male, Han ethnicity, from Ningbo, Zhejiang, is a senior engineer with a PhD. His research interests include object detection and multimodal analysis; Chen Yi (June 2004-), female, Han ethnicity, from Hangzhou, Zhejiang, is currently pursuing a bachelor's degree with a research focus on big data technology.