Research On Optimization Algorithms Of Deep Echo State Network

In this thesis,the research subject is deep echo state network(DESN).Compared with echo state network(ESN),DESN has richer dynamic characteristics,stronger shortterm memory storage capacity and more accurate timing prediction ability.However,DESN still has problems such as colinearity,unfitness and poor numerical stability.Therefore,it is of great significance to study the optimization scheme of DESN for the further development of DESN and its application in the industry with higher requirements on accuracy and stability.The performances of DESN from the variable memory length,the change of activation function and the change of dimension of reservoir state matrix are studied.Firstly,a DESN with variable memory length is proposed,and the reservoir status update is directly related to multiple historical reservoir states,which can remember information for longer periods of time.Secondly,a new activation function is proposed as the activation function of DESN,which is composed of five Sigmoid functions.Each Sigmoid function has a fixed unsaturated region,and the effective region of the new activation function can be changed by changing the coefficient of each Sigmoid function.As is known to all,the DESN must satisfy the echo state property(ESP).In this thesis,the constraint condition that the new model satisfies the ESP is calculated,and it is found that the constraint condition is not changed when the new activation function is still Sigmoid function.To replace the activation function of DESN is a modification from the design of artificial neurons of DESN,and the reduction of the dimension of DESN reservoir state matrix is a modification from the training of DESN.Considering that the DESN training relies heavily on the reservoir state matrix,when the reservoir state matrix is morbid matrix,the DESN is sensitive to the input,and the numerical stability is poor.The Laplace feature mapping dimension reduction algorithm is adopted to reduce reservoir state matrix dimension,and it to a certain extent,reduces the ill-posedness of reservoir state matrix.Due to the DESN with multiple reservoirs,and every reservoir corresponds to an optimal target subspace,the genetic algorithm is applied when looking for a best target subspace dimension combination of DESN.The echo state property of aforementioned novel DESN models are also caculated and four types of simulation experiments were also conducted and compared with the original DESN,verifying that novel models have stronger anti-disturbance ability,more stable transition stability,higher timing prediction accuracy and larger memory capacity.