Research On Model Selection For Support Vector Regression And Application Of It

Statistical Learning Theory (SLT) focuses on the learning theory for small samples. The core of the theory is to control the generalization of learning machine by controlling the complexity of models. Support Vector Machine (SVM) is a general learning algorithm developed from SLT. It has been successfully used in pattern recognition, regression and time series prediction. Support Vector Regression (SVR) is the expansion of SVM to regression problems. To meet the requirements of combat effectiveness analysis based on combat simulation, SVR is used to construct response surface model in this thesis. Model selection for SVR is studied to improve the generalization performance.SVR uses kernel function to extend to nonlinear problems. Kernel function corresponds to the special nonlinear mapping and feature space. The choice or construction of kernel fit to a given problem is important to improve the performance of SVR. In this paper, the robustness of SVR with two typical common Mercer kernels, polynomial kernel and Radial Basis Function (RBF) kernel, is investigated by experiments. Then the possible ways to mix the two kernels are discussed. The mixture kernels are used to improve the robustness of SVR. The objective information on attributes is incorporated into SVR by local kernels to improve the performance of it. To get one kernel value, the different ways to integrate local kernels are discussed. The experiments show that with introduction of objective information, SVR performs better than that with the usual kernels in some cases.For fixed functional form of the kernel, model selection amounts to tuning hyperparameters. This is usually done by minimizing an estimate of the generalization error. The existing methods for estimating generalization error are more expensive for tuning SVR hyperparameters. A leave-one-out error bound for SVR is derived in this paper. After the SVR model is obtained for a given set of hyperparameters, the bound can be computed with very little additional work. Considering the properties of the bound, an optimization method based on difference quotient is proposed. The method replaces the grads in gradient method with difference quotient and can be used to many cases. According to the empirical methods for tuning hyperparameters, the trend of the cross validation error changing with hyperparameters is analyzed for SVR with RBF kernel. Then a heuristic method for tuning hyperparameters is designed. Experiments on benchmark datasets show that the two methods are effective and efficient.Methods for constructing response surface use response surface to approximate the real relationship between the input parameters and the response of simulation system. Based on the response surface the optimization and analysis of complex system become direct and efficient. Response Surface Methodology (RSM) uses low order polynomials to construct response surface. This is not fit to the cases of nonlinear response function and independent parameters with wide value range. Then SVR is used to construct the response surface. The feature selecting and multiple response modeling using SVR are studied to meet the need of response surface construction. A linear programming method for feature selection is proposed. A multiple response SVR model is developed and it is easy solved with the current training algorithms. Then the Design of Experiments (DOE) methods for SVR are discussed. Considering the incremental experiment design, an incremental training method for SVR is designed. Further more, the SVR method is compared with some other methods for constructing response surface.Combat simulation is an important method for combat effectiveness analysis. The framework of combat effectiveness analysis based on simulation is designed. Response surface model is introduced to reduce the simulation expense and meet the need of complex analysis. The combat effectiveness analysis of main tank is studied. The background of this study is the project of general simulation and combat effectiveness analysis of armored vehicles. SVR is used to identify the important measures of performance and construct the response surface model. Based on the model, the measures of performance are optimized under some restrictions. The results show that it is effective to use SVR to combat effectiveness analysis based on simulation.