Article Details

Title · Authors · Abstract · Keywords · DOI · References

Comparative Study of Large Language Models and Student Performance in Exams —Taking Qwen as an Example

LING Dalian, FENG Shiying, CHEN Sinan, PAN Weiquan

(School of Mathematics and Statistics, Yulin Normal University, Yulin 537000, China)

Section:Artificial Intelligence · 2025, Vol. 9, No. 12 · Pages:50-57,62 · 2096-4706(2025)12-0050-09 · CLC number:TP39;G434 · Document code:A · DOI:10.19850/j.cnki.2096-4706.2025.12.011

Abstract

The research focuses on the application potential of Qwen, an AI chatbot driven by LLM, in educational assessment. Based on 2 190 final examination questions of “Probability and Mathematical Statistics” in a university from 2019 to 2023, eight teachers double-blind score the Qwen Model, the optimized model and the students' answers. The results show that the performance of Qwen is stable in multiple choice questions, but there is much room for improvement in the answer questions. Especially after Prompt Engineering optimization, the performance of the answer questions is significantly improved. Teachers' scores on AI-generated content are more stringent, and the scores are significantly affected by the question type and the answer subject. This study provides empirical evidence for AI-assisted educational assessment, emphasizing the importance of updating standards and exploring new models.

Keywords

LLM; Qwen; educational assessment; AI-assisted learning

References

  1. [1] JURAFSKY D,MARTIN H J. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models [M/OL].[2025-01-02].https://web.stanford.edu/~jurafsky/slp3.
  2. [2] RUMJAUN A,NAROD F. Social Learning Theory—Albert Bandura [M].Science Education in Theory and Practice: An Introductory Guide to Learning Theory,Cham:Springer,2025:85-99.
  3. [3] NABAVI R T. Bandura's Social Learning Theory and Social Cognitive Learning Theory [J].Theory of Developmental Psychology,2012,1(1):1-24.
  4. [4] MORGAN H. Howard Gardner's Multiple Intelligences Theory and his Ideas on Promoting Creativity [M]//REISMAN F. Celebrating Giants and Trailblazers: A-Z of Who's Who in Creativity Research and Related Fields.London:KIE Publications,2021:124-141.
  5. [5] MORGAN H. Maximizing Student Success with Differentiated Learning [J].the Clearing House: A Journal of Educational Strategies, Issues and Ideas, 2014,87(1):34-38.
  6. [6] HWANG G J,CHANG C Y. A Review of Opportunities and Challenges of Chatbots in Education [J].Interactive Learning Environments,2023,31(7):4099-4112.
  7. [7] CHIU T K F,MOORHOUSE B L,CHAI C S,et al. Teacher Support and Student Motivation to Learn with Artificial Intelligence (AI) Based Chatbot [J].Interactive Learning Environments,2024,32(7):3240-3256.
  8. [8] IDC.AI Large Model Technical Capability Assessment Report [R/OL]. [2022-6-11] https://www.idc.com/getdoc.jsp?containerId=CHC49698923.
  9. [9] GIRAY L. Prompt Engineering with ChatGPT: A Guide for Academic Writers [J].Ann Biomed Engineering,2023,51(12):2629-2633.
  10. [10] GARTNER.5 Impactful Technologies from the Gartner Emerging Technologies and Trends Impact Radar for 2022 [EB/OL].(2021-12-08)[2025-06-11].https://www.gartner.com/en/articles/5-impactful-technologies-from-the-gartner-emerging-technologies-and-trends-impact-radar-for-2022.
  11. [11] Yu Guoming, Su Jianwei The Communication Revolution and Media Ecology under the Wave of Generative Artificial Intelligence: From ChatGPT to the Future of Comprehensive Intelligence Era [J]. Journal of Xinjiang Normal University: Philosophy and Social Sciences Edition, 2023, 44 (5): 81-90
  12. [12] FARROKHNIA M,BANIHASHEM S K,NOROOZI O,et al. A SWOT Analysis of ChatGPT: Implications for Educational Practice and Research [J].Innovations in Education and Teaching International,2024,61(3):460-474.
  13. [13] PRADANA M,ELISA H P,SYARIFUDDIN S. Discussing ChatGPT in Education: A Literature Review and Bibliometric Analysis [J].Cogent Education,2023,10(2):2243134.
  14. [14] Wu Di, Wu Hejiang Potential risks and avoidance of the application of general large-scale models in education: from the perspective of technological ethics [J]. Journal of East China Normal University: Education Science Edition, 2024, 42 (8): 64-75
  15. [15] Shi Zhoulong, Zhao Feiyan Thoughts on Empowering Education Transformation and Development with Generative Artificial Intelligence [J]. Education Progress, 2024, 14 (5): 950-956
  16. [16] HUANG R,SHI L Y. Content Framework for Artificial Intelligence Literacy Instruction in China from a Global Perspective [J].Documentation,Information and Knowledge,2024,41(3):27-37.
  17. [17] Wang Chen, Liu Yonggui Exploration of Teacher Autonomous Learning Mode Based on Generative Artificial Intelligence: A Case Study of ChatGPT [J]. Software Guide, 2023, 22 (11): 219-225
  18. [18] FARAZOULI A,CERRATTO-PARGMAN T,BOLANDER-LAKSOV K,et al. Hello GPT! Goodbye Home Examination? An Exploratory Study of AI Chatbots Impact on University Teachers' Assessment Practices [J].Assessment and Evaluation in Higher Education,2024,49(3):363-375.
  19. [19] ADESHOLA I,ADEPOJU A P. The Opportunities and Challenges of ChatGPT in Education [J].Interactive Learning Environments,2024,32(10):6159-6172.
  20. [20] BAI J,BAI S,YANG S,et al. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond [J/OL].arXiv:2308.12966 [cs.CV].[2025-06-11].https://arxiv.org/abs/2308.12966.

Funding

2023 Guangxi Higher Education Undergraduate Teaching Reform Project (2023JGB329); College Student Innovation and Entrepreneurship Training Program Project (202110606020)

About the authors

Ling Dalian (May 2004-), female, Han ethnicity, from Beiliu, Guangxi, undergraduate studies, research direction: statistics; Feng Shiying (May 2004-), female, Han ethnicity, from Huai'an, Jiangsu Province, is currently pursuing a bachelor's degree with a research focus on statistics; Chen Sinan (October 2001-), female, Han ethnicity, native of Pubei, Guangxi, undergraduate student, research direction: statistics; Corresponding author: Pan Weiquan (December 1980-), male, Han ethnicity, from Pingnan, Guangxi, associate professor, PhD, research direction: teaching and research of statistics, probability theory, and mathematical statistics courses.