Research On Singularities In The Parameter Identification Of Dynamic Systems

The parameter space of hierarchical models such as neural networks includes singularities ubiquitously. The research of the singular problem has become a hot topic. Many strange phenomena arise because of the singularities existing in the hierarchical models: Cramer-Rao theorem does not hold, the maximum likelihood estimator is no longer subject to the Gassian distribution even asymptotically, the learning speed becomes extremely slow, the plateau phenomenon arises and so on. How to depress the influence of the singularities and speed up the learning speed in the learning dynamics are meaningful research topics. Surround the singular problems existing in the Radial Basis Function Networks (RBFs) and the identification of the nonlinear systems, this paper proposes an adaptive natural gradient method for the RBFs and the identification of the nonlinear systems based on the methods of the information geometry. The content of this paper is concluded as follows:(1)In this paper, we study the singularities in the identification of linear systems and nonlinear systems. We divide the nonlinear systems into two kinds and study the existence of the singularity in the identification of the two kinds of nonlinear systems respectively. As far as the linear systems and the nonlinear systems which are linear related to the parameter are concerned, we can obtain the two corresponding Fisher information matrices, provided that the noise is Gaussian. We prove that the Fisher information matrices are positive if the signals are persistently exciting. As far as the nonlinear systems which are nonlinear related to the parameter, the variety of the kind of system is large, so we can only select one kind of the classical system to analyse. Provided that the noise signal is subject to one kind of distribution, we obtain the Fisher information matrix of the kind of the system and the singular regions where the matrix degenerates. We give the error tendency curves, observe the existence of the plateau in the curve and the regions where the plateau phenomenon arises to demonstrate the theoretical analysis.(2) In this paper, we give the natural gradient method for the RBFs. We get the explicit forms of the Fisher information matrix, provided that the input is Gaussian. We obtain the inverse of the Fisher information matrix by using the Sherman-Morrison formula. However, when the number of the hidden units is large, it is very difficult to calculate the Fisher information matrix and its inverse. Combining the Kalman fiter technique with an equation, we propose the adaptive natural gradient method for the RBFs. At last, we apply the proposed method to fit the nonlinear functions and to identify the nonlinear time-variant system. Comparing the results with the results existing in the literature, we find that the performance of the proposed method is very well.(3) Corresponding to the singularities in the nonlinear system identification, we study the natural gradient method for the nonlinear system identification. We get the Fisher information matrix, provided that the input signal is uniform. We give the inverse of the Fisher information matrix by using the Sherman-Morrison formula. However, when the number of the system terms is large, it is also difficult to calculate the Fisher information matrix and its inverse. Additionally, we can hardly get the prior knowledge of the input signal in the practical engineering. We propose the adaptive natural gradient method for the identification of the nonlinear system by using the technique of the Kalman filter and an equation. In the simulations, we can find that the proposed method can depress the influence of the singularities, speed up the learning dynamics and give a well performance.(4) As far as the singularities in the Gaussian mixture models are concerned, we study the natural gradient method for the Gaussian mixture models. Assuming that x is Gaussian, we get the explicit expressions of the Fisher information matrix and its inverse. When the number of the Gaussian compents is large, it is difficult to calculate the Fisher information matrix and its inverse. Additionally, the prior knowdge of the input signal is hardly given in the practical engeering, so we propose the adaptive natural gradient method for the Gaussian mixture models.