The Adaptive Research Of Data Layout In Large-scale Meteorological Data Storage System

With the rapid development of information technology, human’s social activities are becoming more and more dependent on all kinds of digital information. No matter personal data, core operational data of the enterprises and institutions, or files which ensures the normal functioning of the state and society are inseparable from the storage, processing and transmission of mass information data. In particular, like China, affected by natural disasters is very serious. Meteorological data as a massive information data and meteorological career as China’s public welfare, provides an important guarantee for national security, disaster prevention and mitigation. This work has been highly valued by our country.However, in the existing large-scale meteorological data storage systems, the real-time of meteorological data applications is poor, the state of data and the process of storage and transmission about data process is opaque, the meteorological data management is too centralized, the meteorological data sharing is complex and self-adaptability is bad, the capabilities of data quality assurance is not good. For these problems, this paper presents a solution of large-scale meteorological data storage systems LMDSS. The storage system uses a distributed file system for object-oriented storage, combined with hierarchical CDN network and virtualization technology for storage and backup of meteorological data to ensure the real-time and safety and reliability of meteorological data. On account of the transmission and re-layout about meteorological data between the meteorological data device and meteorological CDN and Key Data Cache, this paper presents a hierarchical consistent Hash data layout algorithm at the same time. The algorithm firstly divides the large-scale meteorological data storage devices into various small sets of storage devices, then it uses the consistent Hash data layout algorithm among each set of storage devices. At last, it uses the consistent Hash data layout algorithm in each set of storage devices to re-layout the data again. Then it can distribute meteorological data fairly. The algorithm can be adaptive the dynamic changes of the meteorological data storage devices. It can re-layout the meteorological data effectively and can be able to locate data quickly to reduce the consumption of meteorological data storage device space. This it can meet the accuracy and real-time requirements of the national meteorological service system, and it can solve the I/O bottlenecks in the process of accessing and storing of meteorological data during the uploading and downloading meteorological data in meteorological applications.Therefore, in the large-scale network storage systems, managing meteorological data efficiently to improve the accuracy and real-time of meteorological services, enhance the predictability of regional climate and achieve the automation and intelligent of weather prediction systems is a basis to protect people’s lives and property safety, and promote social harmony and stability. It has an important practical significance.