An evaluative argument-based investigation of validity evidence for the Utah pre-algebra criterion-referenced test

This study collected evidence to address the assumptions underlying the use of the Utah Core Assessment to Pre-Algebra (UCAP) to (a) measure student achievement in pre-algebra, and (b) assist teachers in making adjustments to instruction. An evaluative argument was defined to guide the collection of evidence. Each of the assumptions in the evaluative argument was addressed using data from a suburban northern Utah school district. To collect the evidence, test content was examined including item match to course objectives, reliability, and subtest intercorrelations. Analyses of correlations of the UCAP with convergent and discriminant measures were completed using student test data (N = 1,461), including an examination of both the pattern of correlations and tests of statistical significance. Pre-algebra teachers (N = 12) were interviewed to ascertain the degree to which UCAP results were used to make necessary adjustments to instruction.; It was found that the UCAP was technically sound, but measured only 65% of course objectives. Correlation coefficients were analyzed using pattern comparisons and tests of statistical significance. It was found that the pattern of correlation coefficients and the distinction of convergent and discriminant measures supported the UCAP as a measure of mathematics. Teacher interview data revealed that teachers did not make substantive adjustments to the instruction of pre-algebra based on test scores.; Based on these results it was concluded that the underlying assumptions concerning the use of the UCAP were not fully supported. The lack of complete coverage of the pre-algebra course objectives calls into question the ability of the UCAP scores to be used as measures of student achievement, in spite of the technical quality of the test. There was support for the assumption that the UCAP measures mathematics. There was little evidence that teachers use the UCAP score reports to make meaningful and appropriate adjustments to instruction. More evidence is needed to understand the factors that may have led to this lack of use.; The evaluative argument framework defined in this study provides guidance for future research to collect evidence of the validity of decisions based on UCAP scores.