Study Of Rapid Lossless Compression Technology Based On Spectral Data And Cloud Platform

In recent years, with the surge improvement of computer performance, the development of near-infrared instrument and the research of the processing method of chemometric data, near-infrared spectroscopy technique has been widely used in industry, agriculture, medicine and other fields. Because the near-infrared spectroscopy equipment had a series of characteristics such as fast, non-destructive detection, low operating cost, without complicated pretreatment process and so on, it could combined with stoichiometry methods to analyzes the ingredients of tobacco quantitative and qualitative. Spectrometer will produce vast amounts of data, and How to carry on these data storage, signal processing, data mining has restricted the application of spectroscopy. Aiming at these problems, this article study the technology of fast compression and storage of the spectroscopy data under the cloud storage environment from the following aspects.First, the principle, concept and architecture of cloud storage were introduced in this paper. The cloud storage technologies including virtualization technology, parallel programming model distributed file management were also analyzed. The MapReduce, HDFS and other technical principles were used in the platform of Hadoop, and analyzed its process of realization.Second, through the experimental, LZO and LZW, LZSS were compared each other in the use of memory, compression time, compression ratio and the compression efficiency on different data. Under the same conditions, the compression and decompression speed of LZO were 20 times more than the other two compression algorithms, while its compression rate was lower about 10% than LZW and LZ77 algorithm. The amount of compressed data is greater, the more obvious advantage of LZO algorithm. Under the cloud platform for distributed processing and parallel transmission environment,it is worth to loss part of the compression rate in exchange for compression time.LZO algorithm is more suitable for near-infrared spectroscopy files to realize fast and real-time compression at the cloud platform.Third, in order to improve the compression efficiency and satisfy the requirements of the spectral data integrity, the goal can be achieved by compressing the critical band set. Therefore, the mathematical models of the critical band set were established by using the continuous wavelet transform(CWT). It could quickly detected the critical band set from the mass spectroscopy data. The algorithm could control the size of compressed file through personalized setting the detection range and step. Finally, The critical band set were compressed through LZO algorithm, and the document processing efficiency was improved at the same time.Last, the near-infrared spectral data information management system was built on the Hadoop framework according to user needs and system needs. The system enable the users to transfer, delete, query, analysis, and manage the spectral data. The system was also show that the compression methods and the critical band set detection method were feasible.