Statistical methods in a high school transcript survey

In complex surveys that involve stratification and clustering structures, given the budget, time and resource restrictions, the surveys are usually designed to produce specific accuracy of direct estimation at high levels of aggregation. Sample sizes for small geographical areas or subpopulations are typically small such that direct estimates in these areas are very unreliable.;Particularly in designs where a single primary sampling unit (PSU) is sampled in some strata, direct variance estimates for these strata are not possible. Alternative variance estimation procedures for strata with one PSU sampled are studied. The first option is a collapsed strata variance estimator followed by synthetic variance redistribution to the stratum level. The second alternative is the use of generalized variance functions (GVFs) estimated by direct variance estimates in strata with more than one PSU sampled for predicting variances in strata with only one PSU in the sample. The GVF methodology shows advantages in simulation studies and in an application of a stratified multi-stage sample survey conducted by Iowa's State Board of Education (ISBE).;In the context of small area estimation, hierarchical Bayesian (HB) analysis is proposed to produce more reliable estimates of small area quantities than direct estimation. A method that benchmarks the HB estimates to the higher level direct estimates and measures the relative inflation of posterior mean squared error in the posterior predictions is developed to evaluate the performance of hierarchical models. Both numerical and graphical summaries of the posterior predictive discrepancy measures are available. The benchmarked HB posterior predictive model comparison method is shown to be able to select proper models effectively in an illustrative example. The method is then applied to fitting models to the ISBE survey data. In this study a small sample of school districts was selected from a two-way stratification of school districts. The survey strata serve as small areas for which hierarchical Bayesian estimators are suggested. The proposed method is used to select a generalized linear mixed model for analyzing the data. Potential applications extend beyond the survey and education contexts.