The Automatic Recognition And Detection Of Sky-Subtraction Residual Component Based On Hadoop Platform

The skylines,superimposing on the target spectrum as a main noise,will reduce the signal-to-noise ratio of the spectrum.LAMOST,as the largest fiber spectroscopic telescope in our country,has a complete set of observation and operation system and data processing flows.Sky-subtraction as one of the most important steps,its purpose is to remove the skylight noises.The effectiveness of the sky-subtraction processes will affect the signal-to-noise ratio of the spectrum directly.If the spectrum still contains a large number of high strength skylight residuals after sky-subtraction processing,it will not be conducive to the follow-up analysis of the target spectrum.At present,the study on the automatic recognition of the abnormal sky-subtraction stellar spectra is not too much.We can only find the abnormal sky-subtraction spectra by manual inspection,and this will reduce the speed of detection.In addition,LAMOST project can observe tens of thousands of spectra overnight,so it needs a reliable and efficient processing platform to enhance the handling ability to the large amount of spectral data.Hadoop,as one kind of distributed data processing platform,can provide efficient recognition and detection of the abnormal sky-subtraction spectra.So the main research work includes:(1)Firstly,introduces the process of the LAMOST spectrum and analyzes the influence factors of sky-subtraction results to finds the characteristics of the abnormal sky-subtraction spectra.Then,A simple and effective method is presented to automatic recognize the abnormal sky-subtraction stellar spectra which have been processed by the LAMOST Pipeline processing procedure and find the positions of the abnormal skylines.(2)To do the data preprocessing based on Hadoop platform,and using the median filter algorithm to process the continuum normalization to remove the continuous spectrum,after that,the spectral data only leave the spectral lines and noise information.Experimental results show that the algorithm can reserve the spectral lines effectively and improve the processing efficiency by using Hadoop.(3)The abnormal sky-subtraction skyline is determined by detecting whether there exits any high strength skyline residuals which are similar to the emission line or absorption line on the Hadoop platform.Finally,all the abnormal skyline positions in the spectra are obtained in this method.The experimental results with the LAMOST spectroscopic dataset show that this method can recognize the abnormal sky-subtraction spectra and find the abnormal skyline positions of different residual strength effectively.In addition,the method is simple and has a high recognition efficiency,and can be applied to the automatic detection of abnormal sky-subtraction of large number of spectra.