How much is good enough? Setting and validating performance standards and cut scores for the Web-based English as a Second Language Placement Exam at UCLA (California)

This dissertation investigated standard setting procedures for the new web-based ESLPE at UCLA and validated the resulting standards. This research posed the following research questions: (1) How do program policy issues (both within and beyond the program) influence standard setting procedures, such as the definition of performance standards, the selection of standard setting methods, and the validation of standards? (2) How well do different standard setting methods agree with each other in assigning students to various levels of placement? (3) What type of evidence needs to be collected to support the claim that the cut scores do represent the intended test standards?; These questions were answered using both qualitative and quantitative analyses. For Research Question 1, two qualitative research methods, interviews and participant observation, were used to examine how the ESL program defined the performance standards of the new ESLPE. The analyses reveal that the program did not have explicit performance standards for the listening and reading sections and that the content of the ESLPE was not directly related to what was taught in the ESL classes. These issues made it difficult for the program to make criterion-referenced placement decisions.; For Research Question 2, three standard setting methods were used to derive cut scores on the new ESLPE, and the cut scores derived from the three methods did not entirely agree with each other in assigning students into different levels.; For Research Question 3, the analyses of procedural integrity show that the standard setting process employed a sound procedure in a thorough and systematic way. The results of Generalizability studies indicate that raters agreed with each other in predicting minimally acceptable student performance on five test tasks. Finally, two dependability indices (Ф(lambda) and pˆ0) suggest that the consistency of the placement decisions was very high.; These findings were interpreted as having implications for language testing theory and practice.