An investigation of Type I error rate control for independent variable subset tests with a binary dependent variable using ordinary least squares, logistic regression analysis, and nonparametric regressio

This study examines and compares, through simulation, the Type I error rate for testing subsets of variables with a binary dependent variable using the likelihood ratio test (LRT) in logistic regression analysis (LRA), logistic regression analysis evaluated with an F (LRF), ordinary least squares regression (OLS), and the Serlin-Harwell Aligned-Rank Procedure (SHARP). Individual and family-wise Type I error rates were evaluated for four types of designs: regression analysis with subsets chosen from four quantitative predictors, analysis of covariance with one categorical variable and one quantitative covariate, analysis of covariance with one categorical variable and three quantitative covariates, and analysis of covariance with two categorical variables and their interaction and one quantitative covariate. Subset tests for OLS were evaluated with extra sum of squares using an F test. Subset tests for SHARP were evaluated with extra sum of squares using a Chi-square test. Subset tests for LRF were evaluated with an F. This is compared to subset tests for LRA evaluated with the LRT. Simulation sample sizes consisted of 40, 60 and 80 observations for the designs. Overall, OLS performed the best in controlling Type I error rate near the nominal level. SHARP also did well but tended to be conservative. LRA evaluated with the LRT had in general overly inflated Type I error rate inflation which was the highest of the four methods evaluated here for these conditions. LRF performed better than LRT but still was not as good as OLS.