Research On Optimization Model And Algorithm Of Twin Support Vector Machine

In recent years,support vector machine(SVM)has become a powerful paradigm for pattern classification and regression.The most popular“maximum margin”SVM attempts to reduce generalization error by maximizing the margin between two disjoint half planes.Twin support vector machine is a new machine learning method based on support vector machine.Different from SVM,twin support vector machine(TSVM)aims to generate two nonparallel hyperplanes,making each plane closer to one of the two classes and as far away from the other as possible.In TSVM,a pair of small-scale quadratic programming problems are used to replace the solution of a single large problem in support vector machine,so that the computing speed of twin support vector machine is about four times faster than that of standard support vector machine.At present,twin support vector machine has become one of the popular methods in the fields of pattern recognition,data classification and function fitting due to its good learning performance.From the perspectives of privileged information learning,domain adaptation and security sample screening,this paper proposes a series of methods to deal with different tasks.The main works can be described as follows(1)A novel robust capped L1-norm twin support vector machine with privileged information(R-CTSVM+)is proposed in this paper.In the new paradigm,learning using privileged information(LUPI)creates a more informative strategy for tasks to achieve better prediction.Support vector machines using privileged information(SVM+)has achieved great success in LUPI on the clean data.However,these works typically suffer from the noise and outlier contained in the data,which leads to larger fluctuations in performance.To handle this problem,in this paper,based on rigorous theoretical analysis,we propose a novel Robust Capped L1-norm Twin Support Vector Machine with Privileged Information(R-CTSVM+).The proposed pair of regularization functions(up and down-bound)can definitely help to increase the learning model’s tolerance to disturbance,as the up-bound function aims to maximize the lower bound of the perturbation of both main feature and privilege feature,meanwhile,the down-bound function aims to minimize the upper bound of the perturbation of both main feature and privilege feature.Moreover,as the widely employed squared L2-norm distance typically exaggerates the impact of outliers,we adopt the capped L1 regularized distance to further guarantee the robustness of the model.We present that the resulted optimization problem is theoretical converged and can be solved by an effective alternating optimization procedure.Experimental resutls on benchmark datasets indicate that the proposed robust model can produce superior performance in the case where data samples contain much noise and outliers.(2)A new domain adaptive twin support vector machine learning using privileged information(A-TSVMPI)is proposed in this paper.Domain adaptation has been extensively studied and the main challenge in the case is how to transform the existing classifier(s)into an effective adaptive classifier to exploit the latent information in the new data source which typically have a different distribution compared with the original data source.Currently,the Adaptive Support Vector Machines(A-SVM)has been proposed to deal with the domain adaptation problem,which is an effective strategy.However,the resulting optimization task by minimizing a convex quadratic function in A-SVM can not effectively minimize the distance between a source and a target domain as much as possible and typically has high computational complexity.In order to handle these problems,in this paper,we extend the A-SVM by determining a pair of nonparallel up-and down-bound functions solved by two smaller sized quadratic programming problems(QPPs)to achieve a faster learning speed.Notably,our method yields two nonparallel separating hyperplanes to exploit the latent discriminant information based on SVM classification mechanism,which can naturally enhance the classification performance.This method is named as Adaptive Twin Support Vector Machine Learning(A-TSVM).Moreover,we consider a high-level learning paradigm with privilege information to learn a induced model that further constrains the solution in the target space.The learned model is named as domain Adaptive Twin Support Vector Machine Learning Using Privileged Information(A-TSVMPI).Finally,a series of comparative experiments with many other methods are performed on three data sets.The experiment results effectively indicate that the proposed Adaptive Twin Support Vector Machine Learning Using Privileged Information can not only greatly improve the accuracy of classification,but also save computing time.(3)A new sample screening for robust twin support vector machine(SSS-RTSVM)is proposed in this paper.TSVM definitely improves the computational speed compared with the classical SVM,and has been widely used in classification and regression problems.However,two problems should be aroused.First,since the convex hinge loss function of TSVM is unbounded,the generalization performance of TSVM declines under the noisy environment.Second,TSVM is challenging to deal with large-scale data.To handle these problem,in this paper,we propose a new method named Safe Sample Screening for Robust TSVM(SSS-RTSVM).Specifically,as the ramp loss is bounded,we clip the hinge loss in the traditional soft margin twin support vector machine to the ramp loss,and introduce a pair of nonparallel proximal hyperplanes to achieve good anti-noise ability to noisy data and outlier data.However,the non-convex problem of Robust TSVM can be considered as a DC programming problem which is computationally inefficient.Then we integrate safe sample screening rules for RTSVM based on the framework of concave-convex procedure(CCCP)to delete the most training samples,i.e.,a subset of the samples called support vectors(SVs)is selected to reduce the computational cost without sacrificing the optimal accuracy.Notably,for the proposed SSS-RTSVM,the security guarantee is provided to the sample screening rule.Extensive experiments are conducted on several benchmark datasets to fully demonstrate the robustness and acceleration of the proposed method.(4)A novel multi-output parameter-insensitive twin support vector regression(MPITSVR)is proposed in this paper.Multi-output regression aims at mapping a multivariate input feature space to a multivariate output space.Currently,it is effective to extend the traditional support vector regression(SVR)mechanism to solve the multi-output case.However,some methods adopting a combination of single-output SVR models exhibit the severe drawback of not considering the possible correlations between outputs,and other multi-output SVRs show high computational complexity and are typically sensitive to parameters due to the influence of noise.To handle these problems,in this study,we determine the multi-output regression function through a pair of nonparallel up-and down-bound functions solved by two smallersized quadratic programming problems,which results in a fast learning speed.This method is named multi-output twin support vector regression(M-TSVR).Moreover,when the noise is heteroscedastic,based on our MTSVR,we introduce a pair of multi-input/output nonparallel parameter insensitive upand down-bound functions to evaluate a regression model named multi-output parameter-insensitive twin support vector regression(M-PITSVR).To handle the nonlinear case,we derive the kernelized extensions of M-TSVR and M-PITSVR.Finally,a series of comparative experiments with several other multi-output-based methods are performed on twelve multi-output datasets.The experimental results indicate that the proposed multi-output regressors yield fast learning speed as well as a better and more stable prediction performance.