Research On An MCP-based Ensemble Pruning Technique Of Adaboost

Ada Boost is powerful ensemble learning method,combining a bunch of weak learners to produce a strong committee with higher accuracy.However,similar to other ensemble methods,Ada Boost uses numerous base learners to produce the final results and thus poses a challenge to memory space when dealing with high-dimensional data.Feature selection methods are able to significantly reduce dimensionality in regression and have been proved to be applicable in ensemble pruning.By pruning the ensemble,it is possible to generate a simpler ensemble with less base learners yet higher accuracy.In this article,we propose the minimax concave penalty(MCP)function to prune an Ada Boost ensemble in order to simplify the model and improve the accuracy simultaneously.We first screen the features of the high-dimensional data with MCPregularized regression,and then generate an Ada Boost model with the screened data.Experiment results show that the screening process does not deteriorate the performance of Ada Boost but improves it.Then we regard the predictions of the base learners as the design matrix in regression task,which allows us to apply the MCPregularized logistic regression to ensemble pruning.The MCP function is compared with LASSO and SCAD in terms of the performance in pruning the ensemble.Experiments performed on real datasets show that MCP pruning outperforms other two methods and it can not only reduce the ensemble size effectively but also generate slightly more accurate predictions than the unpruned Ada Boost model.