Design And Implementation Of Astronomical Anomalies On-line Detection And Off-line Mining System

Astronomical observation and research is a national key development strategy,showing the level of natural science research of a large country.Many countries have invested in superior resources to develop famous astronomical observation systems,such as the Large-scale Synthetic Survey Telescope(LSST)system in the United States and the Ground Wide Angle Camera(GWAC)system in China.At the same time,the supporting astronomical big data management system is becoming more and more important.They provide data management and retrieval functions for the massive data generated by astronomical observation system,and provide data sharing platform for scientific discovery and astronomical fans.For example,Sloan database provides massive astronomical data management services to Sloan Digital Sky Survey(SDSS)in the United States.The GWAC system developed by the National Astronomical Observatory of China is an optical telescope array which can detect the astronomical phenomena with short-time scale.GWAC astronomical system is composed of 40 cameras matrix of small aperture Sky Survey Telescope.Compared with other sky survey projects,it covers a larger observation field and has a higher time sampling rate.The establishment of GWAC corresponding data processing system,not only in the lens sampling interval to achieve rapid astronomical anomaly detection,but also to meet the long-term storage data availability needs.Based on the background of the 13th five year national key R&D plan project and the data processing requirements of GWAC system,a set of astronomical big data processing software with the functions of real-time astronomical anomaly detection,long-term data management and data mining is developed,which supports the combination of flow detection and batch processing and coexistence of various anomaly detection algorithms.According to the gwac camera matrix data processing workflow,the system mainly provides the following functions:First,online astronomical anomaly detection,docking the observation camera array,monitoring the astronomical data generated in real time,to achieve accurate and rapid early warning for short-term astronomical anomalies;second,offline astronomical anomaly mining,establishing a historical data warehouse to achieve long-term astronomical data preservation,while in-depth mining of astronomical anomalies with a larger time span and a wider variety;third,all day area The data visualization of the celestial sphere provides the system interface applicable to astronomers,and dynamically displays the real-time abnormal stars and their brightness curves;fourthly,the astronomical phenomenon retrieval provides a high-speed search engine with multiple data sources,which supports the typical astronomical query and batch access of users;fifthly,the historical data management establishes a sharing platform for some astronomical data,which allows astronomers to enjoy Download to view published scientific data.According to the processing requirements of "fast real-time analysis+big data management" of GWAC system data,the system adopts the master-slave distributed model and deploys the online and offline parallel double-layer architecture based on 40 physical nodes.The online part is built based on the memory database,which is mainly used for monitoring the abnormal phenomena of real-time astronomical data and performing light-weight analysis tasks for a small part of data;the offline part is selected as the NoSQL database,which is mainly used for large-scale long-term data persistent storage and deep abnormal mining,and uses the low-intensity running time of the system to handle complex calculations.At present,the system has been formally applied to the GWAC system of Xinglong observation station of the National Astronomical Observatory,which has realized rapid astronomical anomaly detection and effective long-term data management.The online anomaly detection delay of single batch of data is less than 4 seconds,the total data storage is up to 10PB,and the astronomical observation data is processed more than 10TB every day.In the trial operation stage,the system found about 30 minutes of short-term object flare events from the real sky survey data,and the early warning effect was verified by astronomers.At the same time,the system provides the whole sky data visualization platform,supports the display of astronomical data features in the self-defined space-time range,and has the typical astronomical query function of composite retrieval conditions.The application shows that the system and its method are effective and provide strong support for further large-scale astronomical discovery.