Article Details

Abstract

To address the issues of general large language models in audit applications—such as deviations in understanding professional terminology, insufficient dynamic adaptation to regulatory standards, and high fine-tuning costs—this paper proposes a construction scheme for an audit professional question-answering large model integrating Retrieval-Augmented Language Model Pre-training (REALM) and Low-Rank Adaptation (LoRA) fine-tuning. By integrating industry data including audit reports, laws and regulations, and professional standards, a dedicated dataset containing 10,000 high-quality audit question-answering dialogues is constructed. The REALM retrieval-augmented pre-training mechanism enables modular storage and accurate retrieval of audit knowledge during the pre-training phase. Qwen-7B-Chat serves as the base model, with LoRA technology employed for parameter fine-tuning; performance differences among full LoRA, QLoRA-8bit, and QLoRA-4bit quantization schemes are also evaluated. Experimental results show that after REALM pre-training and LoRA fine-tuning, Qwen-7B-Chat achieves 26.34, 48.79, 29.87, and 42.51 on the BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L metrics, respectively, improving by 9.5%, 6.9%, 8.9%, and 8.3% over the single LoRA fine-tuned model. The audit qualitative accuracy reaches 94.2%, the precision of regulatory citation increases to 93.7%, and the training computing power demand is reduced by more than 78%. This model demonstrates good accuracy and professionalism in audit professional knowledge question-answering, providing a lightweight and cost-effective technical approach for assisting auditors in decision-making, professional training, intelligent compliance review, and other scenarios.

Keywords

intelligent auditing; Large Language Model; REALM pre-training; LoRA fine-tuning; dataset construction

References

1. [1] Shan Wentao, Wang Yongqing Research on Automatic Generation of Audit Reports Based on Generative Artificial Intelligence [J]. Finance and Accounting Monthly, 2024, 45 (24): 16-21
2. [2] Jiangsu Provincial Department of Science and Technology Research on Enterprise Risk Control and Digital Audit Model Construction [Z]. Nanjing Audit University, 2023
3. [3] Hu E J，Shen Y L，Wallis P，et al. LoRA: Low-Rank Adaptation of Large Language Models [PP/OL].arXiv:2106.09685 [cs.CL].（2021-10-16）.[2026-01-03]https://doi.org/10.48550/arXiv.2106.09685.
4. [4] Zheng Lifeng An analysis of the influencing factors and countermeasures of audit quality in accounting firms [J]. Accounting for Township Enterprises in China, 2025 (4): 17-19
5. [5] Zhang Chaoran, Liang Suping, Yang Jie, etc Research on Patent Retrieval Agent Based on Large Model RAG Architecture [J]. Chinese Invention and Patent, 2025, 22 (2): 14-25
6. [6] Guu K，Lee K，Tung Z，et al. REALM: Retrieval-Augmented Language Model Pre-Training [PP/OL].arXiv:2002.08909 [cs.CL].（2020-02-10）[2026-01-09].https://doi.org/10.48550/arXiv.2002.08909.
7. [7] Li Y X，Li Z H，Zhang K，et al. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge [PP/OL].Cureus，2023，15（6）：e40895.（2023-01-24）[2026-01-17].https://doi.org/10.7759/cureus.40895.
8. [8] Jie Wenquan, Ye Qiongyu, Wang Yingni, etc Application of Artificial Intelligence in Breast Ultrasound Report Generation [J]. Beijing Biomedical Engineering, 2023, 42 (5): 483-487495
9. [9] Chai Jingxian, Lang Xufeng, Li Hongyan, etc Research on the Lightweight Language Model of Traditional Chinese Medicine Ancient Books Based on LoRA Fine tuning [J]. World Science and Technology - Modernization of Traditional Chinese Medicine, 2025, 27 (3): 823-831
10. [10] Ren Wenxiong, Jia Xin, Wu Yang, etc Research on Large Language Model in Non destructive Testing Field Based on LoRA Fine tuning [J]. Computer Measurement and Control, 2025, 33 (10): 90-96
11. [11] Administrative Audit Department and Regulatory Department of the National Audit Office Guidelines for Qualitative Expressions and Applicable Laws in Central Department Budget Execution Audit (2023 Edition) [M]. Beijing: China Times Economic Publishing House, 2024
12. [12] Ministry of Finance of the People's Republic of China, China Securities Regulatory Commission, National Audit Office of the People's Republic of China Notice on Issuing the Basic Norms for Enterprise Internal Control [EB/OL]. (08-07-04) [225-12-10] https://kjs.mof.gov.cn/zhengcefabu/200807/t20080704_55982.htm.
13. [13] The Ministry of Finance of the People's Republic of China Notice on Issuing Supporting Guidelines for Enterprise Internal Control [EB/OL]. (2010-04-05) [225-12-10] https://kjs.mof.gov.cn/zhengcefabu/201005/t20100505_290459.htm.
14. [14] Lin Ruifan, Zhou Hongwei, Liu Liangliang, etc Constructing a knowledge graph of the "Treatise on Cold Damage" version based on ontology method [J]. Chinese Journal of Basic Traditional Chinese Medicine, 2023, 29 (10): 1630-1633
15. [15] Papineni K，Roukos S，Ward T，et al. Bleu: a Method for Automatic Evaluation of Machine Translation [C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics，2002：311-318.
16. [16] Lin C Y. ROUGE: A Package for Automatic Evaluation of Summaries [J].Association for Computational Linguistics，2004：74-81.
17. [17] The Ministry of Finance of the People's Republic of China Notice on Issuing Standards such as "Chinese Certified Public Accountants Auditing Standards No. 1211- Identification and Assessment of Significant misstatement Risks" [EB/OL]. (2022-12-22) [2026-01-02] https://kjs.mof.gov.cn/zhengcefabu/202301/t20230103_3861825.htm.
18. [18] The Ministry of Finance of the People's Republic of China Notice on the Revision and Issuance of Enterprise Accounting Standards No. 14- Revenue [EB/OL]. (July 15, 2017) [December 21, 2025] https://www.mof.gov.cn/zcsjtsgb/gfxwj/201707/t20170705_3583546.htm.
19. [19] The Ministry of Finance of the People's Republic of China Notice on Issuing Standards such as "Chinese Certified Public Accountants Auditing Standards No. 1211- Identification and Assessment of Significant misstatement Risks" [EB/OL]. (2022-12-22) [2026-01-13] https://kjs.mof.gov.cn/zhengcefabu/202301/t20230103_3861825.htm.
20. [20] The Ministry of Finance of the People's Republic of China https://kjs.mof.gov.cn/zhengcefabu/202301/t20230103_3861825.htm [EB/OL].（2022-12-22）[2025-12-26]. https://kjs.mof.gov.cn/zhengcefabu/202301/t20230103_3861825.htm.

Funding

National College Student Innovation and Entrepreneurship Program of Jiangsu Province (X2025112870021)

About the authors

Su Xingyu (November 2004-), male, Han ethnicity, from Jingjiang, Jiangsu, undergraduate student, research interests include applied computer technology and intelligent large models for auditing; Corresponding author: Liu Min (October 2001-), female, Han ethnicity, from Zhangjiakou, Hebei, master's student, research interests include retrieval-augmented generation for large models.