文章详情

题名 · 作者 · 摘要 · 关键词 · DOI · 参考文献

面向电信领域的大模型提示词工程测评

范文斌1,王颜颜1,汪盈盈1,许银1,宋骐2

(1.科大国创云网科技有限公司 知识计算智能实验室,安徽 合肥 230088; 2.中国科学技术大学 计算机科学与技术学院,安徽 合肥 230027)

栏目:计算机技术 · 第 2025 年第 12 期 · 页码:123-128 · 2096-4706(2025)12-0123-06 · 中图分类号:TP182 · 文献标识码:A · DOI:10.19850/j.cnki.2096-4706.2025.12.024

摘要

针对提示词工程研究中提示词参数评估不全面,且测评方法未充分考虑实际生产场景复杂性的问题,提出一种面向电信领域的大模型提示词测评体系。为此,构建5个电信领域数据集,涵盖情感文本分类、客服意图识别以及知识问答3大任务。随后,将提示词参数细分为角色、篇幅、语气和顺序4个维度,并系统地评估不同维度对6个大语言模型性能的影响。研究结果表明,合理的提示词设计可以显著提高模型在电信领域这3大任务的表现。

关键词

大语言模型;提示词工程;模型性能优化;电信领域;自然语言处理

参考文献

  1. [1] BROWN T,MANN B,RYDER N,et al. Language Models Are Few-Shot Learners [J]. Advances in Neural Information Processing Systems,2020,33:1877-1901.
  2. [2] RADFORD A,WU J,CHILD R,et al. Language Models Are Unsupervised Multitask Learners [EB/OL].[2024-09-16].https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf9.
  3. [3] LIU P,YUAN W,FU J,et al. Pre-Train,Prompt,and Predict:A Systematic Survey of Prompting Methods in Natural Language Processing [J].ACM Computing Surveys,2023,55(9):1-35.
  4. [4] 许志伟,李海龙,李博,等. AIGC大模型测评综述:使能技术、安全隐患和应对 [J].计算机科学与探索,2024,18(9):2293-2325.
  5. [5] 李毅,李浩,许骁哲,等. CFB:金融领域大模型评估方法 [J].计算机科学与探索,2024,18(12):3272-3287.
  6. [6] 孙柏林. 大模型评述 [J].计算机仿真,2024,41(1):1-7+24.
  7. [7] SCHICK T,SCHÜTZE H. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference [J/OL]. arXiv:2001.07676 [cs.CL].(2020-01-21)[2024-09-16].https://arxiv.org/abs/2001.07676.
  8. [8] GAO T,FISCH A,CHEN D. Making Pre-trained Language Models Better Few-Shot Learners [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).ACL,2021:3816-3830.
  9. [9] LI X L,LIANG P. Prefix-Tuning:Optimizing Continuous Prompts for Generation [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).ACL,2021:4582-4597.
  10. [10] SHENG E,CHANG K W,NATARAJAN P,et al. The Woman Worked as a Babysitter:On Biases in Language Generation [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Hong Kong:ACL,2019:3407-3412.
  11. [11] HOLTZMAN A,BUYS J,DU L,et al. The Curious Case of Neural Text Degeneration [C]//International Conference on Learning Representations.New Orleans:ICAL,2019:1-16.
  12. [12] HU Z,YANG Z,LIANG X,et al. Toward Controlled Generation of Text [C]//International Conference on Machine Learning.Sydney:PMLR,2017:1587-1596.
  13. [13] RADFORD A,WU J,CHILD R,et al. Language Models Are Unsupervised Multitask Learners[EB/OL].[2024-09-16].https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

基金项目

浙江省科技厅“尖兵”研发攻关计划项目(2023C01029)

作者简介

范文斌(1990—),男,汉族,安徽黄山人,实验室执行主任,本科,研究方向:大模型应用技术、智能软件;通信作者:王颜颜(1992—),女,汉族,安徽淮南人,中级职称,博士,研究方向:知识计算、自然语言处理、大模型;汪盈盈(1999—),女,汉族,安徽安庆人,硕士在读,研究方向:自然语言处理、大模型;许银(1998—),女,汉族,安徽合肥人,硕士,研究方向:知识图谱、自然语言处理;宋骐(1990—),男,汉族,安徽合肥人,中国科学院技术大学特任教授,博士,研究方向:图数据库及图数据挖掘。