Optimization Design And Application Of Sparse Echo State Network

Recently,with the rapidly development of artificial intelligence technology,artificial neural network（ANN）has been concerned by scholars because of its good memory ability especially.But it also has some problems,such as gradient explosion,gradient disappearance and high complexity of training.As a novel RNN,the hidden layer of echo state network（ESN）consists of a large number of sparsely connected neurons.The input and internal weights remain unchanged,and the output weights is the only one which need to be updated in the training process.Therefore,compared with other RNNs,ESN has a simpler training method and good learning ability.The core structure of ESN is its sparsely connected reservoir.During network initialization,the reservoir size is generally large to better capture complex sample features,and the redundant network structure will reduce its generalization ability.Therefore,how to sparse network structure and improve its generalization ability is an urgent problem to be solved.To solve this problem,this thesis proposes the corresponding sparse ESN optimization design methods from the perspective of single objective and multi-objective respectively,and improves the network performance by pruning the output weights.The main research of this thesis is as follows:（1）Single objective optimization design of sparse ESN based on l₁regularization.A sparse ESN based on coordinate descent method and l₁ regularization（CD-ESN）is proposed to improve the generalization ability.Firstly,the regularization term is added to the objective function to punish the output weights,and the l₁regularization method is used to shrink the output weights with low contribution to zero,so as to indirectly trim the reservoir neurons and obtain the sparse network.Secondly,to accelerate the convergence of the algorithm,a weight learning algorithm based on coordinate descent method is designed to train and update the output weights.Finally,the model is applied to Lorzen and Mackey-Glass time series prediction to verify its performance.Results show that CD-ESN can sparse network structure,improve the network generalization ability and predict accuracy effectively.（2）Multi-objective optimization design of sparse ESN based on MOEA/D.The regularization parameter in CD-ESN is difficult to determine,so a sparse ESN based on MOEA/D（MDO-ESN）is proposed.Firstly,the structure and training error of ESN are modeled as a multi-objective optimization problem,and the network structure and error are minimized simultaneously by MOEA/D.Secondly,a local search algorithm based on coordinate descent method is proposed to accelerate the convergence of the algorithm.Finally,an adaptive weight updating algorithm is proposed,which adaptively generates weight vectors according to the preferences of decision-makers and guides the algorithm to converge to the preferred region.Results show that compared with other ESNs,MDO-ESN can effectively sparse the network structure,accelerate the convergence of the algorithm to the preferred region,improve the network generalization performance,and has better prediction ability.（3）Soft measurement model design of effluent ammonia nitrogen based on sparse ESN.Aiming at the difficulty of on-line measurement of effluent ammonia nitrogen concentration in municipal wastewater treatment processes（WWTP）,soft measurement models of effluent ammonia nitrogen based on sparse ESN are established.Firstly,the input data are cleaned and normalized by means of detection interpolation.Secondly,an auxiliary variable selecting method based on random forest is proposed.Characteristic variables closely related to effluent ammonia nitrogen are selected by calculating the importance of each characteristic variable to effluent ammonia nitrogen.Soft measurement models of effluent ammonia nitrogen are established by CD-ESN and MDO-ESN.Finally,the soft measurement models are tested on the data from a wastewater treatment plant in Chaoyang District,Beijing.Results show that established models can sparse network structure,have stronger generalization ability than other models,and can predict the ammonia nitrogen of wastewater treatment effluent accurately.（4）Development of effluent ammonia nitrogen prediction system based on sparse ESN.This thesis develops a set of online prediction system of effluent ammonia nitrogen based on sparse ESN.Firstly,according to the user demands,analysis the functions system needs to have;Secondly,the system is designed,which is mainly divided into two parts:system development and analysis,function design.It includes user management functions,data pretreatment functions such as data cleaning and normalization,and effluent ammonia nitrogen prediction function.Finally,the system is applied to WWTP.The system combines WWTP and simulation technology closely.The effluent ammonia nitrogen is predicted through advanced simulation technology,which is the basis of online detection and real-time control of wastewater quality.