Article Details

Abstract

Log anomaly detection is an important foundational technology for intelligent operations and maintenance, fault diagnosis, and system security analysis. As the scale of distributed systems continues to grow, traditional methods that rely on manual analysis and static rules have become increasingly inadequate for complex scenarios. This study introduces the basic workflow of log anomaly detection, including log collection and preprocessing, log parsing, feature construction, anomaly identification, and evaluation metrics. It then reviews the major methodological categories, including traditional statistical and rule-based approaches, sequence modeling, semantic and pre-trained methods, graph-based approaches, and large language model-based methods, and analyzes their characteristics, applicable scenarios, and current challenges. Overall, log anomaly detection is continuing to develop toward greater semantic understanding, intelligence, and engineering practicality.

Keywords

log anomaly detection; log parsing; pre-trained model; Graph Neural Network; Large Language Model

References

1. [1] Liao Xiangke, Li Shanshan, Dong Wei, etc A review of research on large-scale software system logs [J]. Journal of Software, 2016, 27 (8): 1934-1947
2. [2] Oliner A J，Ganapathi A，Xu W. Advances and Challenges in Log Analysis [J].Communications of the ACM，2012，55（2）：55-61.
3. [3] Xu W，Huang L，Fox A，et al. Detecting Large-Scale System Problems by Mining Console Logs [C]//Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles，2009：117-132.
4. [4] Lou J G，Fu Q，Yang S，et al. Mining Invariants from Console Logs for System Problem Detection [C]//USENIX Annual Technical Conference，2010：1-14.
5. [5] He S，Zhu J，He P，et al. Experience Report: System Log Analysis for Anomaly Detection [C]//2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)，2016：207-218.
6. [6] Le V H，Zhang H. Log-Based Anomaly Detection without Log Parsing [C]//2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)， 2021：492-504.
7. [7] Zhu J，He S，He P，et al. Loghub: A Large Collection of System Log Datasets for AI-Driven Log Analytics [C]//2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)，2023：355-366.
8. [8] Yu B，Yao J，Fu Q，et al. Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection [C]//Proceedings of the 46th IEEE/ACM International Conference on Software Engineering，2024：1-13.
9. [9] Ma X，Zou H，He P，et al. On the Influence of Data Resampling for Deep Learning-Based Log Anomaly Detection: Insights and Recommendations [J].IEEE Transactions on Software Engineering，2024，51（1）：243-261.
10. [10] Makanju A，Zincir-Heywood A N，Milios E E. A Lightweight Algorithm for Message Type Extraction in System Application Logs [J].IEEE Transactions on Knowledge and Data Engineering，2011，24（11）：1921-1936.
11. [11] Du M，Li F. Spell: Streaming Parsing of System Event Logs [C]//2016 IEEE 16th International Conference on Data Mining (ICDM)，2016：859-864.
12. [12] He P，Zhu J，Zheng Z，et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree [C]//2017 IEEE International Conference on Web Services (ICWS)， 2017：33-40.
13. [13] Dai H，Li H，Chen C S，et al. Logram: Efficient Log Parsing Using n-Gram Dictionaries [J].IEEE Transactions on Software Engineering，2020，48（3）：879-892.
14. [14] Zhu J，He S，Liu J，et al. Tools and Benchmarks for Automated Log Parsing [C]//2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)，2019：121-130.
15. [15] Meng W，Liu Y，Zhu Y，et al. Loganomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs [C]// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence Main track，2019，19（7）：4739-4745.
16. [16] Du M，Li F，Zheng G，et al. Deeplog: Anomaly Detection and Diagnosis from System Logs through Deep Learning [C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security，2017：1285-1298.
17. [17] Guo H，Yuan S，Wu X. Logbert: Log Anomaly Detection via BERT [C]//2021 International Joint Conference on Neural Networks (IJCNN)，2021：1-8.
18. [18] Huang S，Liu Y，Fung C，et al. Improving Log-Based Anomaly Detection by Pre-Training Hierarchical Transformers [J].IEEE Transactions on Computers，2023，72（9）：2656-2667.
19. [19] Xie Y，Zhang H，Babar M A. Loggd: Detecting Anomalies from System Logs with Graph Neural Networks [C]//2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)，2022：299-310.
20. [20] Zhang C，Peng X，Sha C，et al. Deeptralog: Trace-Log Combined Microservice Anomaly Detection through Graph-Based Deep Learning [C]//Proceedings of the 44th International Conference on Software Engineering，2022：623-634.
21. [21] Zhang X，Xu Y，Lin Q，et al. Robust Log-Based Anomaly Detection on Unstable Log Data [C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering，2019：807-817.
22. [22] Yang L，Chen J，Wang Z，et al. Plelog: Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation [C]//2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)，2021：230-231.
23. [23] Wang X，Song J，Zhang X，et al. Logonline: A Semi-Supervised Log-Based Anomaly Detector Aided with Online Learning Mechanism [C]//2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)，2023：141-152.
24. [24] Nguyen H T，Nguyen L V，Le V H，et al. Efficient Log-Based Anomaly Detection with Knowledge Distillation [C]//2024 IEEE International Conference on Web Services (ICWS)，2024：578-589.
25. [25] Xie X，Jian S，Huang C，et al. LogTraceAD: Anomaly Detection from Both Logs and Traces with Graph Representation Learning [C]//CNCIT '23: Proceedings of the 2023 2nd International Conference on Networks, Communications and Information Technology，2023：116-121.
26. [26] Han X，Yuan S， Trabelsi M. Loggpt: Log Anomaly Detection via GPT [C]//2023 IEEE International Conference on Big Data (BigData)，2023：1117-1122.
27. [27] Cui T，Ma S，Chen Z，et al. LogEval: A Comprehensive Benchmark Suite for LLMs in Log Analysis [J].Empirical Software Engineering，2025，30（6）：1-32.
28. [28] He M，Jia T，Duan C，et al. Llmelog: An Approach for Anomaly Detection Based on LLM-Enriched Log Events [C]//2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)，2024：132-143.
29. [29] Zhang W，Zhang Q，Yu E，et al. Leveraging RAG-Enhanced Large Language Model for Semi-Supervised Log Anomaly Detection [C]//2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)，2024：168-179.
30. [30] He M，Jia T，Duan C，et al. Weakly-Supervised Log-Based Anomaly Detection with Inexact Labels via Multi-Instance Learning [C]//2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)，2025：2918-2930.

Funding

National Natural Science Foundation of China (62261045)

About the authors

Li Meichen (2002-), female, Han ethnicity, from Hengshui, Hebei, master's student, research interests include log anomaly detection; Corresponding author: Geng Taijia (1981-), male, Tibetan ethnicity, from Xining, Qinghai, professor, master's supervisor, doctoral student, research interests include speech recognition, speech synthesis and natural language processing; Wei Jianguo (1971-), male, Han ethnicity, from Tianjin, professor, doctoral supervisor, doctoral student, research interests include speech recognition, voiceprint recognition and medical image analysis.