The Research On Probabilistic Prediction Based On Natural Gradient Boosting(NGBoost)

In the real world,many of the supervised machine learning problems we have encountered have tabular features and real-valued targets.But on some of these issues,the uncertainty estimation of forecasts is often crucial,such as when dealing with tasks such as health care and weather forecasting.Probabilistic prediction,which is the approach where the model outputs a full probability distribution over the entire outcome space,is a natural way to quantify those uncertainties.Gradient Boosting Machines(GBM)have been widely successful in prediction tasks on structured input data,but a simple boosting solution for probabilistic prediction of real valued outputs is yet to be made.In this article,we try to use the natural gradient boost(NGBoost)method which uses the Natural Gradient to address technical challenges.In the theoretical knowledge part of this article,we introduced that the natural gradient boosting method is modular,and the base learner,probability distribution and scoring rules in the structure can be freely selected.Based on the logarithmic scoring rule(MLE),we use the KL divergence in information theory to measure the difference between the distributions,and lead to the concept of natural gradient,which defines the search direction in the iterative algorithm.In the simulation experiment part of this article,we use the content data of SO₂in the air observed at the Beijing Olympic Sports Center site to predict the probability of air quality.We improved the choice of the base learner in the algorithm,and solved the probability prediction problem in regression tasks by using ridge regression,support vector regression,and echo state network models.At the same time,we use the original gradient booster to perform regression prediction.By comparing the effects of the simulation experiments under the two gradient algorithms,we can find that for the probability prediction problem,using natural gradients for model enhancement will have good performance in practical applications.