Research On Curvature Method Of Model Selection

Machine learning is an important research field of artificial intelligence, and model selection is an important research field of machine learning. In many real world machine learning application, the problem is to infer the true model from the given data. However, there may be many possible models, and model selection studies how to choose the model which most matches the real world application. Two core problems of machine learning, generalization and representation, both are related to model selection. To sum up, model selection is the key of solving machine learning problems.Learning theories investigate the problem of model selection from various views. Classic statistic learning theories choose the model through the balance of bias and variance. Regularization methods which slove inverse problems penalize complex models through regularization. Statistic machine learning methods choose the model which can generalize well on unseen data through a compromise between the accuracy on training data and the complexity of the model. With the development of differential geometry in mathematical, researchers began to use geometric method to solve the problems in machine learning of statistical models. This theory views a learning machine as a sub-manifold embedded in the space which comprises all possible distribution, and evaluate accuracy and complexity of the models through the information of its position and shape in the space. All these learning theories reflect the Occam’s Razor-’Entities should not be multiplied unnecessarily’, that the generalization ability of a model is related to its goodness of fit to the training data, as also related to its complexity. Different learning theories have different measure of complexity.This dissertation using the curvature method studies the geometric properties of statistical models to address several key issues in model selection, like generalization ability and reparameterization invariance complexity of model, global properties of statistical model and a unified framework of model selection. First, we analyse and prove the local curvature of model is a reparameterization invariant in the manifold space of models and has a geometric intuition, present a new model selection criterion GKCIC to measure local property of a statistical model. Then curvature can be used to calculate topology information and global geometric information of the model manifold further, we present a new method EPTIC to measure global property of statistical models. Finally we propose a unified framework for model selection and build a hierarchal "memory-prediction" perceptual learning model incorporated the characteristic of perceptual learning. Based on differential geometry, the problem of model selection and its application in perceptual learning are in-depth studied. The methods and technologies proposed in this dissertation are verified through experiments. So it paves the way for further studies. The main contributions are summarized as the following:1. A model selection algorithm GKCIC (Gauss-Kronecker Curvature Information Criterion) based on curvature is proposed. The relationship of generalization ability of a model with its complexity and its accuracy is analyzed. A Gauss-Kroneker curvature method to measure the complexity of learning machine is proposed. A method to calculate the curvature of model through the solution locus in the neighborhood of the expectation of parameters is given, and the normalization criterion is given. Prove that the future residual that is qualified to measure the generalization ability can be expressed by using the intrinsic curvature of model. The proposed algorithm has solid theoretical foundation, intrinsic geometric properties and reparameterization invariant, it can illustrate the geometrical nature of model selection methods intuitively. Experimental results show that the performance of the proposed algorithm is better than the parameter related methods.2. A method EPTIC (Euler-Poincare Topology Information Criterion) using topology information to measure the global property of the model manifold is proposed. Based on the interdependence relationship of curvature and metric, curvature is used as geometrical element of local information. Euler-Poincare characteristic which is a topology invariant of statistical manifold is gotten through the integration of curvature, and it is viewed as a global topology invariant. Some geometrical measure such as the volume of manifold is gotten through the Gauss-Bonnet theorem and Minkowski integration formulation. A model selection method which uses global properties of model manifold is proposed to select the model with global generalization ability. The significance of topology property and global geometry property to the model manifold are analyzed, the computational method to get the topology property from the local property is given. Experimental results show that it can perform better than some other geometry based methods.3. An unified framework of curvature based geometry method of model selection is proposed. Based on the work of previous two chapters, an unified framework of curvature based model selection is proposed which considering both the local and global property of statistical model. The comparision between our curvature based method and the statistical learning theory is given.Under this unified framework, a hierarchical computation model of perceptual learning is proposed which incorporate the research achievement of perceptual learning and cognitive psychology. The global topology and geometry property, which is gotten through the bottom-up abstraction of local information, is used as priori knowledge. The model predicts the input and contract the prediction with the real input through the top-down process. If the prediction is wrong, then the priori knowledge is revised to guide the next prediction. The proposed computation model is local specifically and global generalized by combing the bottom-up abstraction process and the top-down guidance process. The unified framework considers the local and global generalization ability synthetically, it realizes hierarchal "abstract-prediction" mechanism and reflects the specificity and integrity of perceptual learning.