Article Details

Title · Authors · Abstract · Keywords · DOI · References

Prompt Engineering Evaluation of Large Language Model for the Telecommunications Domain

FAN Wenbin1, WANG Yanyan1, WANG Yingying1, XU Yin1, SONG Qi2

(1.Knowledge Computing Intelligence Laboratory, GuoChuang Cloud Technology Co., Ltd., Hefei 230088, China; 2.School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China)

Section:Computer Technology · 2025, Vol. 9, No. 12 · Pages:123-128 · 2096-4706(2025)12-0123-06 · CLC number:TP182 · Document code:A · DOI:10.19850/j.cnki.2096-4706.2025.12.024

Abstract

A prompt evaluation system of Large Language Model (LLM) for the telecommunications domain is proposed to address the issues of incomplete evaluation of prompt parameters in prompt engineering research and the lack of consideration for the complexity in real production scenarios of evaluation method. To this end, five datasets in the telecommunications domain are constructed, covering three major tasks of sentiment text classification, customer service intent recognition, and knowledge-based question answering. Subsequently, prompt parameters are categorized into four dimensions of role, length, tone, and order, and the impact of these different dimensions on the performance of six LLMs is systematically evaluated. The research results indicate that a well-designed prompt can significantly improve model performance on the three major tasks in the telecommunications domain.

Keywords

Large Language Model; prompt engineering; model performance optimization; telecommunication domain; Natural Language Processing

References

  1. [1] BROWN T,MANN B,RYDER N,et al. Language Models Are Few-Shot Learners [J]. Advances in Neural Information Processing Systems,2020,33:1877-1901.
  2. [2] RADFORD A,WU J,CHILD R,et al. Language Models Are Unsupervised Multitask Learners [EB/OL].[2024-09-16].https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf9.
  3. [3] LIU P,YUAN W,FU J,et al. Pre-Train,Prompt,and Predict:A Systematic Survey of Prompting Methods in Natural Language Processing [J].ACM Computing Surveys,2023,55(9):1-35.
  4. [4] Xu Zhiwei, Li Hailong, Li Bo, et al. Overview of AIGC Large Model Evaluation: Enabling Technologies, Security Hazards, and Responses [J]. Computer Science and Exploration, 2024, 18 (9): 2293-2325
  5. [5] Li Yi, Li Hao, Xu Xiaozhe, et al. CFB: Evaluation Method for Large Models in the Financial Sector [J]. Computer Science and Exploration, 2024, 18 (12): 3272-3287
  6. [6] Sun Bailin Review of Large Models [J]. Computer Simulation, 2024, 41 (1): 1-7+24
  7. [7] SCHICK T,SCHÜTZE H. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference [J/OL]. arXiv:2001.07676 [cs.CL].(2020-01-21)[2024-09-16].https://arxiv.org/abs/2001.07676.
  8. [8] GAO T,FISCH A,CHEN D. Making Pre-trained Language Models Better Few-Shot Learners [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).ACL,2021:3816-3830.
  9. [9] LI X L,LIANG P. Prefix-Tuning:Optimizing Continuous Prompts for Generation [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).ACL,2021:4582-4597.
  10. [10] SHENG E,CHANG K W,NATARAJAN P,et al. The Woman Worked as a Babysitter:On Biases in Language Generation [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Hong Kong:ACL,2019:3407-3412.
  11. [11] HOLTZMAN A,BUYS J,DU L,et al. The Curious Case of Neural Text Degeneration [C]//International Conference on Learning Representations.New Orleans:ICAL,2019:1-16.
  12. [12] HU Z,YANG Z,LIANG X,et al. Toward Controlled Generation of Text [C]//International Conference on Machine Learning.Sydney:PMLR,2017:1587-1596.
  13. [13] RADFORD A,WU J,CHILD R,et al. Language Models Are Unsupervised Multitask Learners[EB/OL].[2024-09-16].https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

Funding

Zhejiang Provincial Department of Science and Technology's "Vanguard" R&D Key Project (2023C01029)

About the authors

Fan Wenbin (1990-), male, Han nationality, from Mount Huangshan, Anhui Province, Executive Director of the laboratory, bachelor's degree, research direction: large model application technology, intelligent software; Corresponding author: Wang Yanyan (1992-), female, Han ethnicity, born in Huainan, Anhui Province, with intermediate professional title and doctoral degree. Her research interests include knowledge computing, natural language processing, and large-scale modeling; Wang Yingying (1999-), female, Han ethnicity, from Anqing, Anhui Province, pursuing a master's degree in natural language processing and large-scale modeling; Xu Yin (1998-), female, Han ethnicity, from Hefei, Anhui Province, holds a master's degree in knowledge graph and natural language processing; Song Qi (1990-), male, Han nationality, from Hefei, Anhui Province, is a special professor and doctor of University of Technology, Chinese Academy of Sciences. His research direction is graph database and graph data mining.