Article Details

Abstract

In the information age, automatic text classification is important for improving information retrieval efficiency. However, traditional classification models such as Naive Bayes, Support Vector Machines (SVM), and Logistic Regression, are deficiencies in handling long text and deep semantic understanding. Therefore, this study presents a text classification model based on Transformer. By utilizing Multi-Head Attention mechanism, this model effectively extracts key information and semantic relationships from the text, thus boosting text classification performance. Experiments demonstrate that this model consistently outperforms traditional CNN and LSTM across various datasets, with the highest accuracy of 94.62%, which verifies its effectiveness in text classification tasks.

Keywords

Machine Learning; NLP; Multi-Head Attention mechanism; text classification; Transformer

References

1. [1] Du Sijia, Yu Haining, Zhang Hongli Research progress on text classification based on deep learning [J]. Journal of Network and Information Security, 2020, 6 (4): 1-13
2. [2] Zhang Wenfeng, Xi Xuefeng, Cui Zhiming, etc Review and Prospect of Multi label Text Classification Research [J]. Computer Engineering and Applications, 2023, 59 (18): 28-48
3. [3] Wu Gao, Li Ming, Zhou Daoxiang, etc Text classification based on deep integrated naive Bayes model [J]. Journal of Jinan University: Natural Science Edition, 2020, 34 (5): 436-442
4. [4] Xin Ziming, Wang Fang Research on Text Classification Based on Improved Naive Bayes Algorithm [J]. Journal of Yanshan University, 2023, 47 (1): 82-88
5. [5] Yang Yipu, Zhu Yonghua, Gao Haiyan, etc A method for sentiment analysis of news comments based on article information [J]. Journal of Shanghai University: Natural Science Edition, 2022, 28 (1): 170-178
6. [6] Cheng Jie A SVM based text classification method for English/American English [J]. Electronic Design Engineering, 2021, 29 (14): 70-74+79
7. [7] Li Xiaoying, Yang Ming, Quan Rui, etc Unbalanced Text Classification Method Based on Deep Learning [J]. Journal of Jilin University: Engineering Edition, 2022, 52 (8): 1889-1895
8. [8] Jia Pengtao, Sun Wei A Review of Text Classification Based on Deep Learning [J]. Computer and Modernization, 2021 (7): 29-37
9. [9] Wang Haitao, Song Wen, Wang Hui A Text Classification Method Based on a Hybrid Model of LSTM and CNN [J]. Small Microcomputer System, 2020, 41 (6): 1163-1168
10. [10] Liu Shuang, Ding Zhe, Lv Chao, etc Analysis of Safety Issues of Offshore Fishing Vessels Based on Text Classification and Knowledge Mining [J]. Journal of Agricultural Engineering, 2023, 39 (24): 215-223
11. [11] Tang Zhuang, Wang Zhishu, Zhou Ai, etc Transformer capsule ensemble model for text classification [J]. Computer Engineering and Applications, 2020, 56 (24): 151-156
12. [12] Zhang Yangfan, Ding Manganese Improved Transformer based Bidirectional Encoder for Dialogue Text Recognition [J]. Science, Technology and Engineering, 2022, 22 (29): 12945-12953
13. [13] LEE J-S，HSIANG J. PatentBERT: Patent Classification with Fine-tuning a Pre-trained BERT Model [J/OL].arXiv:1906.02124 [cs.CL].（2019-05-14）.https://arxiv.org/abs/1906.02124.
14. [14] GREENE D，CUNNINGHAM P. Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering [C]//23rd International Conference on Machine Learning.Pittsburgh：ACM，2006：377-384.
15. [15] ZHANG X，ZHAO J B，LECUN Y. Character-level Convolutional Networks for Text Classification [J/OL].arXiv:1509.01626 [cs.LG].（2015-09-04）.https://arxiv.org/abs/1509.01626?file=1509.01626.
16. [16] MAAS A L，DALY R E，PHAM P T，et al. Learning Word Vectors for Sentiment Analysis [C]//49th Annual Meeting of the Association for Computational Linguistics：Human Language Technologies.Portland：ACM，2011：142-150.

About the authors

Lin Yun (October 1990-), male, Han nationality, from Fuqing, Fujian, lecturer, master's degree, research direction: machine learning, natural language processing, and mobile energy optimization.