Research On Feature Extraction And Identification From Fluorescence Spectra Of Organic Pollutants In Drinking Water Based On Autoencoder

Urban water supply has a vital impact on the national economy,people’s life and social stability.In recent years,the sudden pollution accidents of water supply source and pipe network in China are frequent,which seriously threaten the safety of urban water supply.It is urgent to speed up the construction of water quality pollution detection and early warning system to improve the level of urban water supply security.Thanks to its features as high sensitivity,good selectivity,fast and real-time response,the three-dimensional fluorescence spectrum detection technology has gained increasing attention and application in water quality safety monitoring,pollutant quantitative analysis,pollutant identification and other fields.However,the detection effect of three-dimensional fluorescence spectrum analysis would be affected due to the factors as low concentration of organic pollutants in water,the background fluctuation of water quality,weak characteristic signal,and approximate material structure.In addition,the linear feature extraction method applied in the traditional detection algorithm has weak generalization ability for new samples and difficult application in engineering.In view of the above problems,the autoencoder is used as a self-monitoring feature learning method to automatically extract the low-dimensional essential features of the three-dimensional fluorescence spectrum of organic pollutants in drinking water,and the research on the detection and classification of pollutants is carried out.The main contents and innovations are as follows:(1)An autoencoder-based method of feature extraction and anomaly detection of organic pollutants three-dimensional fluorescence spectrum is proposed viewing the fact that the spectral characteristic signal of organic pollutants is weak and the detection rate is low at low concentration.To begin with,interpolation resampling is carried out for the collected drinking water spectral data samples so as to increase the drinking water samples under the background fluctuation;Then,the model of three-dimensional fluorescence spectrum of drinking water is built by stacked autoencoder.The model parameters are effectively trained by pre-training and fine-tuning,and the nonlinear characteristics of three-dimensional fluorescence spectrum are extracted;Finally,the reconstruction error of the test sample after the model reconstruction and the original sample spectrum calculation is combined with the threshold method for anomaly detection.The experimental results show that the method can automatically extract the non-linear characteristics of the three-dimensional fluorescence spectrum,and the detection effect of low concentration organic pollutants is significantly improved.(2)A method of feature extraction and classification recognition of 3D fluorescence spectrum of organic pollutants based on convolutional autoencoder is proposed so as to solve the problem of insufficient generalization ability of traditional feature extraction methods of 3D fluorescence spectrum.Firstly,the convolution neural network is introduced into the algorithm,which can effectively extract the neighborhood features of the three-dimensional fluorescence spectrum from the local field of view and pool layer,guaranteeing the feature invariance of the organic pollutant spectrum under the background change and the automatic learning of the nonlinear features of the organic pollutant spectrum with generalization;Then,the classification and recognition model of organic pollutants is established based on the lifting tree algorithm XGBoost.According to the experiment,this method has better generalization in the feature level and obvious advantages in the result statistics level compared with such traditional three-dimensional fluorescence spectrum feature extraction methods as principal components analysis(PCA)and parallel factor analysis(PARAFAC),which has proved that the convolutional autoencoder can automatically learn the essential features of the three-dimensional fluorescence spectrum of organic pollutants,so as to improve the detection effect of organic pollutants in drinking water.(3)A method of feature extraction and recognition of three-dimensional fluorescence spectrum based on multi-scale convolutional autoencoder is proposed to solve the difficulty to distinguish the high similarity of spectra due to the similar structure of organic pollutants in drinking water.Firstly,the multi-scale convolutional features are fused by using the deconvolution up-sampling and skip structure.The fused feature spectrum takes into account local and global information,and effectively extracts the texture features of the structure similar to the organic pollutants spectrum;Then,the model of organic pollutants identification is established by using the lifting tree algorithm XGBoost.The experiment results show that,compared with the convolutional autoencoder,the multi-scale convolutional autoencoder effectively complements the deficiency of the feature information,and the improvement of the discrimination effect proves the effectiveness of the method in the case of highly similar structure of organic three-dimensional fluorescence spectrum.(4)A detection system of organic pollutants in drinking water based on three-dimensional fluorescence spectrum is designed and developed.The detection system is divided into two parts:algorithm module and detection platform,the later of which is based on the Spring MVC framework,Bootstrap and MySQL technology,responsible for the business logic of the system;Based on the Scikit-learn of Python,Keras and Tensorflow modules,algorithm modules are built to provide computing services and algorithm interfaces,and finally reduce the system coupling through Docker deployment.In conclusion,this paper mainly studies the feature extraction and recognition method of organic pollutants fluorescence spectrum in drinking water based on autoencoder.Stacked autoencoder is applied to exert organic pollutants anomaly detection in drinking water so as to improve the detection rate of low concentration organic pollutants;Convolutional autoencoder is applied to classify and identify organic pollutants in drinking water so as to improve the generalization ability of spectral characteristics of organic pollutants;The multi-scale convolutional autoencoder is used to distinguish the substance of the structure similar organic pollutants,so as to improve the discrimination effect of the structure similar organic pollutants.Based on the research of this paper,the detection system of organic pollutants based on three-dimensional fluorescence spectrum is designed and developed,which provides technical support for the safety of urban drinking water quality.