Research On Massive Data Processing Methods For Mingantu Spectral Radioheliograph

Radio astronomy forms an important part of modern astronomy.With the construction of large radio telescopes,radio observation provides a great possibility for the realization of many scientific goals of mankind.The high-performance processing and analysis of the massive complex data of radio telescope is an essential part of astronomy research in the new age.MingantU SpEctral Radioheliograph(MUSER)is a new generation cm-decimeter aperture telescope in China.MUSER is a dedicated solar radio interferometer,imaging the sun in the frequency range of 400 MHz ~ 15 GHz with high time,high space and high frequency resolution almost simultaneously.The massive observational data brings great challenge to high-performance real-time and post data processing.This dissertation focus on the automation process and data processing performance of the MUSER.The work of runs through the whole process of data preprocessing to imaging,aiming at achieving breakthroughs in the quality of MUSER imaging and computing performance.The detailed work description is as follows:(1)In the data preprocessing stage,further researches are carried out on automatic abnormal data flagging based on machine learning.Through the analysis of the original observation data,this study realize the automatic abnormal data flagging in MUSER data processing system by the machine learning methods of support vector machine(SVM)and recurrent neural network(RNN).This work has achieved high precision rate,which solves the problem of the flagging process over-reliance on artificial records in the previous work and provides a guarantee for imaging quality.(2)A complete MUSER data processing pipeline is preliminary designed.The format definition of MUSER UVFITS is given,which lays a foundation for data exchange.At the same time,the phase calibration,UVW calculation and observation data integration methods are systematically studied to improve the final image quality.(3)A further study of MUSER high-performance imaging algorithms is presented and algorithms including weight,gridding and CLEAN are implemented based on GPU.A hybrid CLEAN algorithm for MUSER and a parallel cleaning algorithm based on multi-scale bandpass filtering are studied,resulting in improvement both on imaging quality and performance.(4)Correction of the system phase error in the current stage and estimation of the number of iterations for clean algorithms.Owing to the tracking accuracy the only available calibration source in the current stage,the solar disk in the original dirty map deviates from the image center.Through the analysis of the dirty map,the paper realizes the detection of the solar disk and sky brightness,and then calculates the deviation parameter to correct the Phase error.Improvement of the CLEAN algorithm by using the estimated sky brightness as the iterative threshold overcomes the problem that determination of the number of iterations depends on experience.(5)Implementation of the MUSER high-performance data processing pipeline.This paper gives the GPU implementation and the performance analysis of the algorithms.Furthermore,the whole data processing pipeline is integrated into the distributed computing framework—OpenCluster,which is designed for high performance data processing in astronomy.At the same time,the MUSER data processing command line system is developed based on Python.The currently implemented system can meet real-time and post-processing requirements.The work of this dissertation further improved and finally constructed a highperformance MUSER data processing system,which lays a foundation for the full play of MUSER's performance advantages and improving the follow-up research outputs.At the same time,in the context of China's participation in the international science and technology innovation cooperation project— the world's largest radio telescope SKA(Square kilometre Array),MUSER can be used as an effective experimental environment for SKA solar observation.Moreover,the current study of mass data processing methods lays a good foundation for the future development of SKA.