Research On Key Technology Of MUSER Massive Data Pretreatment

With the rapid development of astronomical observation techniques,more and more ground and space-based telescopes are built and put into observation,ledding to the explosive growth of astronomical observation data. Therefore, it has become an important issue to deal with the growing massive data quickly and effectively in the current astronomical community.In this paper, we focus on the data processing and analysis system of the Chinese Mingantu Ultrawide Spectral Radioheliograph. A detailed investigation and research on the integral architecture of the overall architecure of MUSER,the original data format and the data processing requirements is carried out. On this basis, key technologies of data preprocessing for MUSER are studied:1) We have Conducted a research on current massive data retrieval and query approach. The results we get from experiments on a variety of retrieval uchniques show that FastBit performs better in case of massive data index creation and retrieval2)The practical application of FastBit in observation Data real-time index and indexes rebuild is proved to perform extremely well, and we propose an optimized double buffering technology to further improve the efficiency. Meanwhile, a verification method is used to insure index consistency3)To achieve compliance with the actual demand of MUSER,we integrate the high-performance data retrieval to retrieve related data extraction, calibration, compensation, antenna coordinate transformation,and integrating computing4)To meet the actual needs of MUSER,we designe the suitable FITS file for the project,the final FITS nles generated by the system can be impoiied into CASA for further processing Overall, the implementation of this study solves the massive observation data of MUSER in real time index,store and retrieval. The use of FastBit in high-performance mass data Yast retrieval in astronomical data processing is creative. The research results have been applied to the data processing system of MUSER and have obtained a good application effect.