Bootstrap strategies for variance component estimation: Theoretical and empirical results

This study examined strategies for carrying out bootstrap estimation of variance components generated by random effects models. Prior studies (Brennan et al., 1987; Othman, 1995; Luecht & Smith, 1989) have tested a variety of bootstrap resampling strategies (such as resampling persons, resampling items, or simultaneously resampling persons and items) for variance component estimation. Each of these studies reported widely divergent estimates across the different bootstrap strategies.; This study takes an analytic approach toward (1) describing the bias in variance component estimates resulting from different strategies of bootstrap resampling, and (2) demonstrating the inappropriateness of specific strategies in certain contexts based the mechanism of bootstrap sampling. A set of principles is provided for guidance in selecting a specific bootstrap strategy for bootstrap variance component estimation. Results reported in previous studies are re-analyzed using bias adjustments based on the analytic approach. These widely divergent estimates actually converged upon adjustment.; A series of empirical simulation studies further examined the efficacy of the bias adjustments. Adjusted point estimates approached exact values more closely than their unadjusted counterparts in all cases regardless of design. Standard error and interval estimates followed patterns expected based on the mechanisms of bootstrap resampling. When applied to empirical performance assessment data from a 600 x 5 design, the adjustments produced a similar pattern of equivalence of estimates across bootstrap strategies.; The case is made that choice of bootstrap strategy should be carried out on a component-by-component basis. In general, resampling along the dimension(s) represented by the component of interest will give the most accurate standard error and interval estimates, except in the case of the residual component, which requires the additional resampling of estimated residuals. Recommendations are extended to complex designs (e.g. nested designs). Principles underlying the selection of a bootstrap strategy are recommended for applying the bootstrap to additional analytical contexts.