Reliability, validity and tolerance for error: A multivariate generalizability study of the Chinese College Entrance Comprehensive Examination

The Chinese College Entrance Comprehensive Examination (CCECE), which was installed in 1999, is a comprehensive achievement test to measure an examinee's academic readiness for tertiary education. Over one million high school graduates take the CCECE each year. However, there has been little psychometric evaluation of the test.; The purpose of this study was to evaluate reliability and validity of the CCECE. The study used multivariate generalizability theory to estimate measurement errors and generalizability (reliability) coefficients. It also employed Kane's approach to estimate the ratio of the measurement error to the tolerance for error in making pass-fail decisions on the examination. In addition, traditional measurement considerations based on classical test theory, e.g., content validity, relations with other measures, concurrent validity, and consequences validity, were also examined.; The G-study and D-study results suggest that a composite universe score approach was not ideal both because variance distributions among subtests were inconsistent with the planned score weighting and because correlations among some subtests were low. For example, chemistry and history accounted for more variance than other subtests combined. Among the problematic subtests, geography contributed negatively to reliability of the whole test. The generalizability coefficient of the whole test was .78, which was considered moderate but acceptable. Doubling the items for all the subtests would increase generalizability coefficient to .87. Except for geography and political studies, reliability estimates for the subtests were acceptable.; Several conclusions were made and some suggestions were provided for the National Educational Examination Authority and future study.