Likelihood Adaptive Punishment Variable Selection Method Research

The penalized likelihood approach is a widely used statistical method in variable selection. Generally, with a convex penalty which often has an algorithm quick and stable enough, the parsimony in the estimates can’t be easily gained, and the oracle property can’t be achieved:only (suppose not coordinate adaptive) with a nonconvex penalty can we achieve the oracle property, but when the concavity is too large a converging algorithm needs to be constructed and there is often a lack of stability in the solution. It is thus necessary to control the concavity for the non-concave penalties. Using penalties with too large a concavity the aim function will not have a single (global) optimum, not have sparse convexity in high dimensional case, not have a converging algorithm, converge to a stable point but not local optimum, or have a set of local optimums that diverges, which lead to relative unstableness:with too small a concavity (such as LASSO), the solution to the choice may not be sparse enough, or the bias will be relatively too large. So we aim to find a possible penalizing approach, which will weaken the constraint for the largest concavity, while with the oracle property simultaneously, to have the best prediction which uses variables as less as we can.Here we use the stability defined by [7], that is, if small changes in the dataset will cause large changes in the prediction. In practice for penalized likelihood stability is often influenced by the property of the optimizer in some algorithm, including the existence of global convexity, convexity in iterations, or sparse convexity in high dimensional case, continuity of the solution, the asymptotic property of the set of local optimizers, or the strictly local convexity in path. We can put some constraints on the concavity of the penalty to make the optimizer in the algorithm more stable.Inspired by a Bayesian interpretation, we develop the likelihood adaptively mod-ified penalty (LAMP), where the smoothness and shape effect reduction property can balance the parsimony and stability better in some case, and also has an oracle property. This paper mainly studies the theoretical properties and applications of the LAMP in both low dimensional and ultra-high dimensional cases based on either parametric or semiparametric models. In parametric models, we construct the explicit form of LAMP, focusing on gener-alized linear regression, and give the theoretical properties in both the low and ultra-high dimensional cases. That is, consistency, sparsity, oracle property, and the high-dimensional weak oracle property;we introduce the concept of asymptotic stability for low dimensional case and show the conditions for it:we discover the shape effect re-duction property of LAMP from the conditions for strictly local convexity; we combine CD algorithm. LLA algorithm, IRLS, and the design of a "Violate" function to a new algorithm, which is proved to converge under certain conditions; we testify the oracle property and shape effect reduction through low dimensional simulations and compare with other penalties in different schemes such as selection consistency, fitting error, sen-sitivity on perturbation and division of the cross validation;we deal with real data using LAMP and the result turns out satisfying.In semiparametric models, we mainly focus on categorical models with a retrospec-tive sampling in low dimensional case. The form of LAMP we consider the same as sampling prospectively. We display the construction of the penalized likelihood and give the conditions on both the profile likelihood and penalty to obtain an estimate with good properties such as oracle property:we perform simulations on the difference of different sampling schemes and point out that the variable selection of a categorial model with retrospective sampling can use the algorithm in the prospective sampling case only when it is a Logistic-type model; we also refer to another approach to construct the penalized likelihood, which is based on the Taylor expansion of the profile likelihood.LAMP in the paper is a family of penalties mainly based on parametric models, which is nonconvex for most GLM models. They have some Bayesian interpretation and more good properties than other penalties:the efficiency of algorithm, consistency, sparsity, oracle property, low-dimensional asymptotic stability, high-dimensional weak oracle property, shape effect reduction, and infinitely differentiable. Both the results of the simulations and real data turn out good, in low and ultra-high dimensional case respectively. Both the theoretical properties and applications deserve to be explored in depth.