Research On Sparse Online Kernel Learning Algorithms

Online learning is an incremental machine learning method that can update models and make predictions on unknown data in a real-time manner.Compared with traditional offline learning,online learning algorithms do not require all training data to be prepared in advance,but can update and optimize models in real-time according to the real-time data streams.With the introduction of kernel functions,online kernel learning algorithms can handle linearly non-separable data which greatly expanded their application scope.In actual engineering scenarios with limited computing and memory resources,sparse online kernel learning algorithms can not only reduce the complexity of the models and improve system response time,but also reduce the use of computing resources.Therefore,this thesis conducts research on the sparsity of online kernel learning algorithms,while trying to explore more properties that make online kernel learning algorithms adaptable to different applications.Contributions of this thesis are described in the following three aspects:(1)This thesis proposes a sparse online kernel learning algorithm based on random Fourier features(RFF-SOKL).In the high-dimensional feature space induced by a explicit mapping,the online kernel learning algorithm based on random Fourier features(RFFOKL)lacks sparsity.To address this issue,this thesis replaces the random optimization strategy with a sparse optimization strategy based on the original algorithm and proposes RFF-SOKL.The sparse optimization strategy truncates the gradient in a more delicate way,which can effectively reduce redundant features.The proposed algorithm can effectively generate sparse models and reduce space complexity.The experimental results show that the proposed algorithm not only has better sparsity and classification performance,but also has shorter response time for unknown samples.(2)This thesis proposes an online kernel classification algorithm based on local Fisher risk(LFR-SOKL).To address the problem that common loss functions do not establish a connection with historical data and ignore data distribution information,this article designs a local Fisher discriminant criterion which simultaneously minimizes the local Fisher loss function and local Fisher regularizer,thereby proposing a LFR-SOKL algorithm.Both the local Fisher loss function and local Fisher regularizer describe local scatter information through the nearest neighbor relationships between samples.In the process of generating sparsity during the budget maintenance,we design a new outlier scheme for the algorithm to further optimize its time efficiency while improving its robustness.Experimental results show that the proposed algorithm performs well in both regular data stream and noise data stream classification tasks.(3)This thesis proposes two parallel sparse online kernel learning(PSOKL)algorithms by combining the above two algorithms with a parallel computing framework.The parallel implementation of online kernel learning algorithms can significantly improve the time efficiency of the original algorithms and avoid the problem of data accumulation that may be encountered in serial training.Experimental results show that the proposed PSOKL algorithms can produce online kernel learning models that are relatively sparse and have good classification performance in extremely short time without sacrificing too much sparsity.