Quantile Regression For Massive High-dimensional Data

The quantile regression model has strong robustness and can more comprehensively reflect the relationship between the distribution of independent variables and response variables,so it has been widely used in research by various scholars,and its application in massive highdimensional data also has significance.The application of quantile regression in massive high-dimensional data has the following problems:(1)The storage and calculation of massive data exceeds the computer memory.(2)Too high dimension leads to the increase of unimportant variables selected by the model.In this paper,we adopt the divide-and-conquer method to divide the entire data set into several small data sets for parameter calculation,then we can reduce the memory usage.We also add the L1 regular term to the objective function to achieve the purpose of variable selection..At the same time,in the solution of quantile regression,this paper transforms the minimization of the objective function of the quantile regression model into the maximization of the likelihood function of the nonlinear regression model whose error term obeys the asymmetric Laplace distribution(ALD),Thus,the non-smooth loss function is transformed into a smooth convex quadratic function,and then solved by the idea of maximum likelihood.Then,the model solution is transformed into the objective function solution problem with missing data,and the EM algorithm is used to solve the quantile regression model.Finally,the PQREM method for solving the quantile regression model of massive high-level data is obtained,and the effectiveness of the method is tested by data simulation and empirical analysis.