Design And Implementation Of Compression Algorithms For Large Scale Astronomical Data

As the era of big data comes,the scale of astronomical observation data has increased significantly.The radio astronomical data collected from the Five-hundred-meter Aperture Spherical radio Telescope(FAST)is in large scale with complex structure and features,which has brought severe challenges to the data storage,analysis and transmission procedure in astronomical observation systems.The customized compression algorithm design and system implementation for FAST astronomical observation data is particularly valuable and meaningful in scientific researches,which can significantly improve the efficiency of data processing in astronomical observation systems.This research takes the FAST radio observation data as the research object,and completes the entire process to design and implement the compression algorithm.This thesis first demonstrates the classic theory and design principles of compression algorithms,and specifically introduces the LZ series of lossless compression encoding algorithms and the SZ lossy compression encoding algorithm.Based on the feature analysis results of the targeted astronomical data,this thesis respectively proposes and optimizes the bit-shift based preprocessing algorithm for lossless compression standard and the integer wavelet transform based preprocessing algorithm for lossy compression standard.Through integrating the preprocessing algorithms with the corresponding compression encoding algorithms,the detailed technical flows of the complete lossless and lossy compression algorithm are designed.Compared with the existing compression algorithms without data preprocessing method,the lossless compression algorithm designed in this thesis can achieve a compression ratio improvement up to30%,and the lossy compression algorithm designed in this thesis can achieve an average compression ratio improvement of more than 30%.Next,this thesis applies the OpenCL heterogeneous computing framework to design the system for the lossless compression preprocessing algorithms,and performs the algorithm implementations respectively in CPU architecture and GPU architecture.Compared with the execution results of the software algorithm implemented on traditional architecture,the implementations based on different heterogeneous platforms can achieve performance improvement in different level.