Article Details

Title · Authors · Abstract · Keywords · DOI · References

Research and Practice on Integrated Evaluation of Large Language Models

HE Qi, HAN Xiao, MAO Haotian, QIU Jianmin

(China Telecom Corporation Limited Jiangsu Branch, Nanjing 210037, China)

Section:Artificial Intelligence · 2025, Vol. 9, No. 11 · Pages:59-63,69 · 2096-4706(2025)11-0059-06 · CLC number:TP391.1 · Document code:A · DOI:10.19850/j.cnki.2096-4706.2025.11.012

Abstract

With the increasing application of LLMs, how to accurately, objectively and comprehensively evaluate the ability of large models has become an important topic of common concern in academia and industry. In recent years, Jiangsu Telecom has actively carried out the exploration and practice of LLMs, and reconstructed multiple applications in the BMO domains through large models. This paper introduces the integrated evaluation scheme and system practice of Jiangsu Telecom based on the current open source big model ecology. This scheme can agilely access the latest released open source large models, and realize the blind test selection of large models based on practical applications, providing a useful reference for building a more scientific and perfect Large Language Model evaluation system.

Keywords

LLMs; evaluation; framework

References

  1. [1] Jin Yuan, Li Chengzhi Financial Capability Assessment and Implications of AI Large Models: ChatGPT Based on CPA Exam and Domestic Large Model Testing [J]. Finance and Accounting Monthly, 2024, 45 (18): 44-51
  2. [2] Yan Jing, Chen Yan, Liu Jing Training method and text understanding method and device for text understanding model: CN116050465A [P]. 2022-05-02
  3. [3] Yu Chuanming, Guo Yajing, Zhu Xingyu, etc Research on Extractive Text Abstraction Model Based on Maximum Boundary Relevance [J]. Information Science, 2021, 39 (2): 34-43
  4. [4] Shen Liping, He Chaofan, Cao Dongxu, etc Evaluation and Analysis of the Application of Large Language Models in Middle School History [J]. Modern Educational Technology, 2024, 34 (2): 62-71
  5. [5] Zhang Ran, Zhao Hui, Dong Haonan A Brief Analysis of the Application of Large Model Code Generation Technology in Digital Banking [J]. China Financial Computer, 2024 (6): 53-55
  6. [6] Li Zhiyun, Feng Luoluo, Li Xiaoyu, etc A Dialogue Intent Recognition Method Combining Large Models: CN116775870A [P]. 2022-09-19
  7. [7] GitHub. evalplus/evalplus [EB/OL].[2024-12-10].https://github.com/evalplus/evalplus.
  8. [8] GitHub. THUDM/AgentBench [EB/OL].[2024-12-16].https://github.com/THUDM/AgentBench.
  9. [9] NI S W,KONG X G,LI C M,et al. Training on the Benchmark Is Not All You Need [J/OL].arXiv:2409.01790 [cs.CL].[2024-12-23].https://doi.org/10.48550/arXiv.2409.01790.
  10. [10] Xu Zhiwei, Li Hailong, Li Bo, etc Overview of AIGC Large Model Evaluation: Enabling Technologies, Security Hazards, and Responses [J]. Computer Science and Exploration, 2024, 18 (9): 2293-2325

About the authors

He Qi (1983.06-), male, Han nationality, from Yancheng, Jiangsu, engineer, master's student, research direction: IT system planning and construction, big data AI。