Multivariate Time Series Prediction Based On Echo State Networks

Complex systems are worldwide pervasive in many aspects, such as meteorology, hydrology and so on. It presents characters of chaos, multivariate dynamic evolution, and multiple level organization, so that it is difficult to construct the analytical models exactly. In this context, the multivariate time series modeling technique came into being, which provided a new way for studying the dynamics of the complex systems. In the past decade, a new kind of neural networks, called echo state networks, were proposed and then attracted domestic and foreign scholars’ attention. However, the research on training algorithm and structural optimization has not been completed yet. In this paper, training algorithms and structures are modified, in order to heighten the prediction accuracy of multivariate time series and realize the multiple-step-ahead prediction and online prediction. The main contents are as follows:1. For the complicated computation of the large-scale reservoir state matrix in the existed dimensionality-reduction-based echo state networks, some novel modified echo state networks are proposed. Due to the fact that hundreds or thousands of nodes are in the echo state networks, the reservoir state matrix is extremely large and prone to be ill-posed. Dimensionality reduction methods can be used to extract a low-dimension subspace out of the reservoir state matrix. Then the subspace replaces the original state matrix to calculate the output weights. But the existed work is computation consuming, such as PCAESN which involves the eigenvalue decomposition of a large-scale dense matrix. For this reason, first, we build the fast subspace echo state network model, in which a small size of tridiagonal matrix is established by Lanzos iteration to approximate the original large-scale matrix. By this way, the computational complexity of the model will be reduced and the prediction accuracy can be increased. Second, a large number of reservoir state variables are attributed to a small amount of integrated factors based on factor analysis method, which aims to eliminate the impact of colinearity produced by large correlation coefficients between reservoir state variables. And next, the output weights would be calculated by integrated factors, instead of the original reservoir state matrix. Finally, considering the influence of the noise in the actual observed data, Laplacian eigenmap is introduced to restore its nonlinear relationship, which is in order to extract a low-dimension manifold of the state matrix. Furthermore, the models’ transient stability, transient controllability and transient observability are analyzed herein.2. To solve the problem that there are some redundant components in the large number of reservoir state variables, sparse regularized echo state networks are proposed. The adaptive L1 norm and L2 norm penalties are imposed on the output weights and construct adaptive elastic echo state network model,in order to remove the redundant components as well as reduce the correlation coefficients between reservoir states and the amplitude of non-zero output weights simultaneously. L1 norm penalty could shrink the amplitude of the output weights to zero which relate to redundant components, while L2 norm penalty could continuously compress the correlation between the reservoir states, and reduce the amplitude of non-zero output weights. Meanwhile, the adaptive term could reduce the deviation, which results from large output weights, and improve the accuracy of the model. As the computation of the adaptive parameters enlarges the computation complexity of the model, considering that the L1/2 norm regularization does better on sparsity, a hybrid regularization algorithm is proposed based on L1/2 norm and L2 norm. Compared with other regularization methods, the hybrid regularization method could draw sparser and smaller amplitude of output weights with a small regularization parameter. Since a small parameter introduces a small deviation, there is no need to impose an adaptive penalty, which simplify the computation. The model has a good generalization performance. To achieve multiple-step-ahead prediction, the horizon of predictability of a chaotic time series is calculated based on Lyapunov exponent. Experimental results show that the proposed model achieves a high accuracy within the horizon of predictability.3. Combination models based on echo state network and other methods are proposed when considering the problem that simply modified echo state network could not map the complex statistical properties of multivariate time series sufficiently. To avoid the ill-posed problem and reduce the redundant components, the number of reservoir neurons need to be declined, which demands each neuron with a strong mapping ability. Herein, wavelet analysis is combined with echo state network to construct wavelet diagonal echo state network, in which the composite function based on sigmoid function and wavelet basis function is used as the activation function to enrich the diversity of the neurons. Considering that there are both linear features and non-linear features in time series, and the residual series still contain certain rules, a new prediction model based on error compensation is presented. The ARMA model is adopted to fit the linear feature first, and then the obtained residual sequence which obeys weak Gaussian distribution is input to the ridge-regression-based echo state network. The predicted value is compensated to ARMA model prediction. Besides, as the dynamics of complex systems vary over time, there are non-stationary characteristics in the observed multivariate time series. For this reason, an online echo state network model based on square root cubature Kalman filter is proposed to adjust the network parameters real-timely. Experimental results demonstrate the proposed model with good adaptability and with ability to achieve a high online prediction accuracy.