Design And Implementation Of Real-time Astronomical Meta-data Archiving System

With the deployments of various large-scale high-precision observation equipment and advanced detection devices,there has been an explosive growth in astronomical observation data.Astronomical meta-data archiving system archives astronomical observation data by managing meta-data,and users can quickly locate the required astronomical observation data sets by retrieving specific metadata information.However,the current off-line or periodic archiving method adopted by the astronomy metadata archiving system has the problem of insufficient real-time performance.At the same time,the data query response is slow and it is difficult to meet the application requirements of real-time data query.Aiming at above sloving these problems,a real-time astronomical metadata archive system was designed and implemented using techniques such as distributed streaming data processing architecture and graph database.For the problem of insufficient real-time metadata archives,the metadata metadata acquisition module and metadata processing module are designed to achieve the purpose of real-time metadata processing.At the same time,a data storage module was designed to improve the problem of longer data retrieval response time.This paper first analyzes the data features of astronomical metadata,designs a method based on real-time monitoring of data sources to obtain data source data dynamic information,and combines Apache Flume and other technologies to achieve the real-time metadata acquisition function of the meta-data acquisition module.Then,in response to the demand for real-time processing of astronomical metadata,the meta-data processing module based on the distributed streaming data processing architecture was used to develop the Storm Topology data processing topology.The astronomical metadata was calculated in the form of a stream and the astronomical element was implemented.Finally,for the demand of real-time data query,combined with the data characteristics of astronomical metadata,the limitations of the existing astronomical metadata storage solutions are analyzed.At the same time,a point-edge storage technology of Neo4 j database is combined to design a space-based astronomical metadata storage model to realize the storage of multi-source,heterogeneous,and massive astronomical metadata.The experimental results of system function and performance test show that the real-time astronomical metadata archiving system designed by this paper can realize real-time data processing and can store multi-source,heterogeneous,and massive astronomical metadata to achieve real-time data query.At the same time,the system stability and concurrency meet the design goals and it is of great significance to further astronomy research.