A validity argument for score meaning of a computer-based ESL academic collocational ability test based on a corpus-driven approach to test design

Among the language capacities that L2 learners of English develop as they increase their proficiency is the ability to use appropriate collocations in the relevant registers of language use. Assessment of this capacity may therefore provide an efficient means of distinguishing among examinees with various levels of language proficiency within a particular register, although research has not yet attempted to operationalize such a measure. The purpose of this research is to explore the use of a measure of developmental, register-specific collocational knowledge as a means of making relevant distinctions among examinees in a minimal amount of time. Ultimately, such a test could be used in conjunction with other information for norm-referenced decisions such as placement, with a convincing argument for such uses. The first step, however, is to develop the interpretive argument and validity argument for score meaning of such a measure.;This validation study followed a mixed-methods embedded and sequential explanatory design consisting of quantitative and qualitative data collection (Creswell & Plano Clark, 2007). Quantitative data were collected from performance on the test of collocational ability, a productive vocabulary size test, and a reading test by 206 Chinese-speaking learners of English at three levels of English proficiency. Qualitative data were collected for a sample of the participants during and after the quantitative data collection and analyzed in order to identify evidence to explain initial quantitative results from the collocational ability test including test-taking processes. Qualitative data included screen capturing during test administration (n = 10), post-test interviews (n = 6) and a test reflection survey (n = 206). The test development, piloting, data collection, and analysis provide backing for the assumptions underlying inferences in the interpretive argument.;The study presents the development of a validity argument for the meaning of the score on the computer-based ESL collocational ability test, which could be used to contribute to the decision to allow a test-taker to participate in English-medium instruction or place the test-taker in an appropriate English language course. The first stage in the validity argument begins with the interpretive argument, which defines the grounds, inferences, warrants, and claims that provide the foundation for score interpretation (Kane, 2006). The claim that the interpretive argument needs to support is that the test score reflects the ability of the test-taker to use and understand lexical collocations as they are written in college and university settings. Backing for the inferences in the interpretive argument includes a theoretical analysis and empirical data. Theoretical evidence provides the backing for the domain definition, evaluation, and part of the explanation inferences. These empirical results provide evidentiary support for assumptions backing part of generalization, explanation, and extrapolation inference in the interpretive argument. Backing for the utilization and impact intention inferences is beyond the scope of this study. The backing for the assumptions that underlie the interpretive argument provide the basis for the validity argument.