Research On Sparse Broad Learning System Based On Binary Mask

Broad learning system is a shallow network,which is composed of input layer,feature layer,enhancement layer and output layer.Compared with the deep network,it has the advantages of simple structure,less parameters to be optimized and high computational efficiency.In addition,broad learning system can quickly update the model through the incremental learning algorithm,which shows its good model scalability.Although the broad learning system has achieved good results in many fields,there are still some shortcomings and challenges that limit its further application in practical scenarios.Firstly,the input weights of the broad learning system are randomly generated,which is easy to cause the redundancy of parameters and model structure.In addition,the existing broad learning system has a large amount of training computation when training sparse models,resulting in a waste of computing resources.Secondly,in the process of sparse broad learning system,it is easy to ignore the importance changes of different weights,which leads to incorrect pruning during model sparse.Finally,in the case of sparse broad learning system,the model cannot be dynamically sparse with a fixed number of parameters,which makes the broad learning system unable to be effectively applied in the application scenarios with strict computational resource constraints.This thesis studies the above problems and proposes corresponding solutions.The specific research content is as follows:(1)In order to solve the problem of node redundancy and large amount of model training computation in broad learning system,a sparse gating broad learning system based on conditional computation is proposed.This method introduces the idea of conditional calculation,extracts features from the input data through the gating network to generate the binary gating mask,and uses the binary mask to guide the selective generation of nodes of the broad learning system model to avoid unnecessary calculation and resources.By sacrificing an additional gating computation,the computation amount and memory resources of broad learning system are reduced.Specifically,first of all,this method extracts the features of the data by establishing a gating network that exists in parallel with the broad learning system,and obtains the binary gating mask through mapping.Secondly,the broad learning system generates nodes selectively under the guidance of gating mask and trains the model.This method can sparse the model and reduce the amount of model performance.Finally,the proposed method is applied to the regression dataset and the classification dataset to verify its validity.(2)In the process of model sparseness,the broad learning system is prone to the problem that it is difficult to recover by wrong pruning.To solve this problem,a broad learning system based on dynamic sparse training is proposed.The regularization term is introduced into the objective function of the standard broad learning system to constrain the output weight threshold,and the optimal network parameters and sparse network structure are found through the joint training of the output weight and the output weight threshold.The output weight threshold is introduced for each output weight,and the output weight mask of the control model structure is generated according to the change of the importance of the output weight.This method can indirectly sparse the model through the output weight mask on the basis of preserving the output weight.Through this dynamic training,where the weight sparsity has been changing,the optimal balance between network structure and network accuracy is found to improve the overall performance of the model.Finally,the proposed method is applied to multiple UCI public datasets to verify its effectiveness.(3)Aiming at the problem that the broad learning system can not control the model sparsity when it uses dynamic sparsity for model sparse and can not be applied in the scenario with strict resource budget,a sparse broad learning system based on adaptive connectivity training is proposed.Given the number of parameters,this method can delete and connect the weights by generating a mask that can control the weight sparsity through the weight importance,and finally find the optimal network structure and network parameters.In essence,it is to use dynamic training to search the optimal model parameters and network structure with a fixed number of parameters.Specifically,first,determine the expected sparsity of the model,and then cut out the unimportant weights through the analysis of the importance of the weights.Secondly,some weights are restored randomly.Then,under the expected sparsity,an optimal model structure is searched through multiple iterations.Finally,the proposed method is applied to multiple UCI datasets to verify its effectiveness.