Article Details

Abstract

Aiming at the current situation that the customer data of traditional small and medium enterprises is disorderly and lacks standardization, an innovative data governance technology is proposed. This technology integrates multi-source heterogeneous data, fuses Optical Character Recognition (OCR) and other methods, and constructs a standardized basic information data lake of small and medium enterprises, to improve data quality from the source. By introducing the concept of “entropy decrease” and using intelligent algorithms to quantitatively evaluate data quality, data quality problems can be located and solved in time. At the same time, a time series database is built and a Markov Chain model based on entropy decrease is constructed to predict future data quality trends and accurately govern potential problem areas. This technology not only maximizes the value of data, but also significantly reduces the cost of governance. It improves the efficiency and accuracy of data governance and provides strong support for enterprises to decrease costs and increase efficiency.

Keywords

entropy decrease; data governance; Markov Chain; data lake of small and medium enterprises; time series database

References

1. [1] Li Weigang, Zhong Zheng, Wang Yongqiang, etc Synchronous positioning and map construction algorithm based on time distance entropy reduction strategy [J]. Information and Control, 2023, 52 (5): 660-668+688
2. [2] Yan Jiahe, Li Honghui, Ma Ying, etc Key technologies for multi-source heterogeneous data fusion and governance system of government big data [J]. Computer Science, 2024, 51 (2): 1-14
3. [3] Huang Junfeng, Ye Pangjun, Wang Min Practice of Data Governance Based on Big Data Infrastructure Platform [J]. Information Technology and Standardization, 2022 (6): 19-23
4. [4] Chen Lu, Guo Yuxiang, Ge Congcong, etc Cross source data error detection method based on federated learning [J]. Journal of Software, 2023, 34 (3): 1126-1147
5. [5] Yu Qichao, Han Xu, Ma Danxuan, etc Design and Implementation of Streaming Big Data Data Cleaning System [J]. Computer Age, 2021 (9): 1-5
6. [6] Liu Luwen, Chen Xingrong, He Tao Teaching Effect Evaluation Method Based on Markov Chain [J]. Statistics and Decision, 2014 (3): 93-94
7. [7] Liao Puming Commodity Market State Prediction Based on Markov Chain State Transition Probability Matrix [J]. Statistics and Decision, 2015 (2): 97-99
8. [8] Yang Haimin, Pan Zhisong, Bai Wei A Review of Time Series Prediction Methods [J]. Computer Science, 2019, 46 (1): 21-28
9. [9] Zhang Jianjin, Wang Yunbo, Long Mingsheng, etc Predictive Recurrent Network for Seasonal Spatiotemporal Data and Its Application in Urban Computing [J]. Journal of Computer Science, 2020, 43 (2): 286-302
10. [10] Zheng Yuebin, Zhu Guohun The application of time series model based on Twitter data in the prediction of influenza [J]. Chinese Journal of Preventive Medicine, 2019, 20 (9): 793-798

About the authors

Liu Min (1975-), female, Han nationality, from Taizhou, Zhejiang, engineer, master's degree, research direction: big data, artificial intelligence; Huang Yixiao (1978-), male, Han ethnicity, from Longchuan, Guangdong, is a senior engineer with a master's degree. His research interests include big data and artificial intelligence; Chen Zhiyang (1971-), male, Han ethnicity, from Shenzhen, Guangdong, is a senior engineer with a master's degree. His research interests include big data, artificial intelligence, and network information security; Zhang Zhanmei (1979-), female, Han ethnicity, born in Yangchun, Guangdong, is a senior engineer with a master's degree. Her research interests include big data and artificial intelligence.