Research On Several Learning Problems Of Linear State-Space Model

Sequential data are orderly observations collected from a real physical system,re-flecting the statistical characteristics possessed by the actual physical system.Different from the researches on general statistic feature vectors,the study on sequential data mainly focuses on analyzing the underlying physical system,rather than observations.Therefore,modeling sequential data is actually finding ways to describe the physical system in a possible mathematical form.The fitted mathematical model may serve as a foundation for the following data analysis and/or model optimization.Linear Time-Invariant State-Space Model combines the merits of both generation model and Bayesian filtering theory.Its validity has been widely recognized and its ap-plications can be discovered in many fields.This model provides a tool for approaching the physical model behind observations.It can not only reflect the correlations between different dimensions of sequential data,but reflect the general trend of data changes.In addition,state-space model inherits the merits of being a generation model.Statis-tical inference is applicable to this model and may discover latent information out of noise-corrupted observations.The simple mathematical form of state-space model also makes the subsequent analysis and interpretation easier to proceed.This dissertation focuses on two key problems in linear time-invariant state-space model:multi-model identification and parameter optimization.The former studies how to build multiple models from sequential data or improving single model identification using ambient conditions,while the latter studies how to speed up the model param-eter learning through an optimization algorithm and/or by mortifying model topology.These two problems have attracted extensive attention since they were proposed.Mean-while,joint multi-model optimization also poses new challenges for the conventional identification algorithm and the parameter optimization.This dissertation discusses and studies some relevant aspects of linear time-invariant state-space model,including multi-model identification,model stability,and hidden state dimension estimation etc.The main work and innovation of this dissertation are as follows:.proposes a supervised multi-model identification method.This method uses pairs of sequences in two different classes as training data.The difference on data classes implies that there should be identifiable "diversity" between underlying physical systems.Therefore,this information is added to the training objective as a priori.The training target balances the-goodness-of-fit and the model diver-sity so that the generated models become discriminable.As a result,it is con-ducive for the following classification and other discriminative learning tasks on the yielded model set.Different from the traditional methods,which treat model parameters equally and ignore their specific physical meanings,this newly pro-posed method establishes the relationship between model parameters and the cor-responding model behavior in the process of model comparison.Moreover,this method can be easily extended to many other likelihood-based training methods since it does not depend on a special treatment of class labels.·improves the existing Bayesian optimization algorithm for higher efficiency in time-limited applications.In this work,a directional constraint is proposed to provide local "gradient-like" information for the optimizer.It can help to im-prove the randomness and jiggles in the local search.The proposed algorithm can balance global search and local search at different stages of the ambient time budget,thus gaining an overall advantage in a given time window.The numerical simulation experiments verify the correctness of the algorithm,and the parame-ter inference experiments on the state-space model confirm the superiority of the newly proposed algorithm over other parameter optimization algorithms..replaces the trainable hidden state with a randomly connected nonlinear "reserve pool",which aims at solving the problem of high complexity of state-space model training methods.In this work,every single model is used to represent one se-quence in the data set.The proposed method improves internal hidden nodes on their "memory capability" and stimulates computing nodes in the pool with input signals and singles from connected nodes.Hidden state is constructed by con-catenating readings of all internal nodes in the reserve pool.Finally,collected hidden states along time line are employed to reconstruct input signals through a linear operation with adjustable parameters.The trainable part of this model is only the adjustable parameters in the readout operation.The linear operation incurs low computational complexity and has no local optimum.This work also makes a preliminary discussion on the measurement between models,and ver-ifies the effectiveness of the proposed measurements in different tasks through experiments..discusses the complexity of the linear time-invariant state-space model.Based to the theory of information transmission,the concept of model complexity is estab-lished and estimated in terms of "describing length".The "appropriateness" of a state-space model class for given data is also discussed.Based on the analysis,a model order elimination algorithm is proposed,which can integrate model selec-tion with model identification,eliminate unnecessary hidden state dimension of the model,and thus reduce the probability of over-fitting.The experimental re-sults on different synthetic data sets verify the good performance of the proposed algorithm on the model selection.