Article Details

Title · Authors · Abstract · Keywords · DOI · References

Streaming Log Compression Algorithm for Embedded Systems Based on Pattern Detection

LI Bin, LU Jun, ZHANG Cheng

(AVIC Xi'an Aeronautics Computing Technique Research Institute, Xi'an 710068, China)

Section：Electronic Engineering · Vol. 2026, Issue 8 · Pages：44-48 · 2096-4706（2026）08-0044-05 · CLC number：TP311.5 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2026.08.008

Abstract

With the widespread deployment of embedded systems in IoT, industrial control, and intelligent devices, log recording as a critical means for system runtime and fault diagnosis, faces dual challenges of limited storage capacity and stringent real-time requirements. This paper proposes a Streaming Log Compression algorithm based on Pattern Detection (SLC-PD), which employs an adaptive sliding window mechanism to analyze repetitive pattern characteristics in log streams in real time, and achieves efficient online compression through finite state machine control. Theoretical analysis demonstrates that the algorithm achieves O(1) time complexity and O(W) space complexity, where W denotes the window size. Experimental results indicate that SLC-PD maintains low processing latency while achieving compression ratio of up to 80%, with memory overhead less than 18 KB, making it suitable for diverse embedded application scenarios.

Keywords

embedded system; pattern detection; compression; log recording

References

  1. [1] ZHU Z，XU S，LI J，et al. LogHub: A Large-scale Log Dataset for Automated Log Analysis [J].IEEE Transactions on Dependable and Secure Computing，2021，19（6）：3853-3868.
  2. [2] ZIV J，LEMPEL A. A Universal Algorithm for Sequential Data Compression [J].IEEE Transactions on Information Theory，1977，23（3）：337-343.
  3. [3] LI X Y，ZHANG H Y，LE V-H，et al. LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data [C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE).Lisbon：IEEE，2024：254-265.
  4. [4] LIU J，ZHU J，HE S，et al. LogZip: Extracting Hidden Structures Via Iterative Clustering for Log Compression [C]//2019 34th IEEE/ACM International Conference on Automated Software Engineering.San Diego：IEEE，2019：863-873.
  5. [5] RODRIGUES K，LUO Y，YUAN D，et al. CLP: Efficient and Scalable Search on Compressed Text Logs [C]//Proceedings of USENIX Symposium on Operating Systems Design and Implementations.Boston：USENIX，2021：183-198.
  6. [6] LIU Q，XU Y，LI Z，et al. Fast and Effective Compression for IoT Systems [D].Chicago：University of Chicago，2022.
  7. [7] ZHANG Y，CHEN W，LEI L，et al. CompressStreamDB: Fine-grained Adaptive Stream Processing without Decompression [J].IEEE Transactions on Knowledge and Data Engineering，2024，36 （5）：1123-1138.
  8. [8] XIANG S，LIU C，XIAO X，et al. Task-oriented Compression Framework for Remote Sensing Satellite Data Transmission [J].IEEE Transactions on Industrial Informatics，2024，20（3）：3456-3468.
  9. [9] IDREES S K，AZAR J，COUTURIER R，et al. SZ4IoT: An Adaptive Lightweight Lossy Compression Algorithm for Diverse IoT Devices and Data Types [J/OL].The Journal of Supercomputing，2025，81（1）：1-25（2025-01-14）[2026-01-04].https://doi.org/10.1007/s11227-024-06667-9.
  10. [10] GEETHAKUMARI P R，SOURDIS I. Stream Aggregation with Compressed Sliding Windows [J/OL].ACM Transactions on Reconfigurable Technology and Systems，2023，16（3）：1-28（2023-06-20）[2026-01-04].https://doi.org/10.1145/3590774.
  11. [11] TANG B Z，YANG S Y，SHEN Z T，et al. LogLite: Lightweight Plug-and-Play Streaming Log Compression [J/OL].arXiv:2507.10337 [cs.DB].（2025-07-14）.https://arxiv.org/abs/2507.10337.
  12. [12] COLLET Y. LZ4 Fast Compression Algorithm [EB/OL].（2011-04-24）[2026-01-03].https://lz4.github.io/lz4/.
  13. [13] GOOGLE. Snappy: A Fast Compressor/Decompressor [EB/OL].（2011-03-18）[2026-01-03].https://google.github.io/snappy/.

About the authors

Li Bin (1988-), male, Han ethnicity, born in Zhuanglang, Gansu, is a deputy senior engineer with a master's degree. His research direction is computer application technology.