Article Details

Abstract

This study addresses issues related to unstructured data service and storage in the provincial meteorological big data cloud platform (Tianqing). It aims to resolve problems such as data retrieval failures and excessive storage utilization rates caused by inconsistencies between file indexes and physical storage, thereby enhancing the standardization and efficiency of data management on the platform. Based on meteorological metadata and data lineage, this study employs database technology. Through methods such as time-series-based lightweight I/O parallel validation and differentiated time-range filtering, it systematically analyzes the current state of unstructured data storage, monthly growth characteristics, and anomalies in data service. The results indicate that Tianqing exhibits five categories of issues: inconsistent data categories, changes in storage locations, replacement of outdated data with new data, anomalous test data, and abnormalities in data directories. In terms of storage distribution, numerical model data, satellite data, and radar data account for a relatively high proportion, and some exhibit significant monthly growth. Based on data service status analysis and storage assessment, this study optimizes storage strategies according to provincial data distribution and growth patterns, promoting the standardized management and efficient service of meteorological data in the operational applications of the meteorological big data cloud platform.

Keywords

Tianqing; unstructured data; NAS; storage optimization; metadata; data lineage

References

1. [1] Xu Jungang, Shao Peiying Distributed Database System and Its Applications [M]. Beijing: Science Press, 2012
2. [2] Xu Yongjun, He Wenchun, Liu Zhen, etc Distributed storage testing and application of massive meteorological station data [J]. Guizhou Meteorology, 2016, 40 (5): 61-68
3. [3] Thunder Design and Implementation of Distributed Storage for Meteorological Big Data [J]. Computer Technology and Development, 2021, 31 (5): 193-197
4. [4] Wang Hua, Li Bo, Wang Biao, etc Application Practice of Distributed Database Technology with Fusion Architecture on Meteorological Big Data Platform [J]. Mid low Latitude Mountain Meteorology, 2020, 44 (5): 93-96
5. [5] Xu Juan, Liu Yang, Wang Xudong, etc Implementation and Application of a Compatible Meteorological Service Interface System [J]. Science and Innovation, 2022 (18): 152-153
6. [6] National Technical Committee for Standardization of Meteorological Basic Information (SAC/TC 346). Interface Specification for Meteorological Data Services: QX/T 618-2021 [S]. Beijing: Meteorological Publishing House, 2021
7. [7] Xu Juan, Liu Xin, Xi Xiaohui, etc Application of Data Lake in Meteorological Information System [J]. Modern Information Technology, 2022, 6 (12): 127-129
8. [8] Zhou Xueying, Wei Mindan, Hu Lili Research on Meteorological Algorithm Management System Technology Based on Container Cloud [J]. Electronic Design Engineering, 2025, 33 (5): 35-39
9. [9] Huang Zhi, Huang Heng, Liang Weiliang, etc Preliminary exploration of business integration design and application based on "Tianqing" DPL [J]. Meteorological Research and Application, 2022, 43 (1): 73-77
10. [10] Li Xiang, Guo Li The Development History of Meteorological Communication Systems [J]. Advances in Meteorological Technology, 2018, 8 (1): 44-45
11. [11] Zhao Fang, He Wenchun, Zhang Xiaoying, etc Construction of National Comprehensive Meteorological Information Sharing Platform [J]. Progress in Meteorological Science and Technology, 2018, 8 (1): 171-180
12. [12] Jia Chengang, Wang Wei Optimization Design of File Storage Based on Meteorological Big Data Cloud Platform [J]. Shaanxi Meteorological, 2025 (5): 82-86
13. [13] Zhu Tianfu, Yu Yongcheng, Wang Xiao Design and Application of Fujian Meteorological Geographic Information Service Platform [J]. Strait Science, 2024 (5): 110-114145

Funding

Gansu Province Joint Research Fund Project (25JRRA1116); Meteorological Science and Technology Research Project of Gansu Provincial Meteorological Bureau (Ms2025-A-3); Gansu Meteorological Information and Technology Equipment Support Center Science and Technology Innovation Fund Project (Ms202402); Gansu Meteorological Information and Technology Equipment Support Center Science and Technology Innovation Fund Project (XXZXZd202601)

About the authors

Zhang Weiwei (1995-), female, Han ethnicity, from Dingxi, Gansu, engineer, master's student, research direction: meteorological information technology and data processing; Corresponding author: Xu Zhuxia (1987-), female, Han ethnicity, from Lanzhou, Gansu Province, senior engineer, master's student, research direction: meteorological information processing.