Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking

The purpose of the study was to investigate the extent to which scores from the speaking test designed for this study could be interpreted as indicators of test-takers' grammatical and pragmatic ability. In other words, this study investigated the extent to which test-takers' performance on the speaking test at different ability levels was affected by the components of language ability operationalized in this test (i.e., grammatical ability, defined in terms of accuracy and meaningfulness, and pragmatic ability, defined in terms of sociolinguistic, sociocultural, and psychological appropriateness) as opposed to method factors. The study also aimed to examine how grammatical and pragmatic knowledge could be assessed through a range of contextually rich, communicative tasks.;This study used a mixed methods design in two parts. The first part of the study used an exploratory-quantitative-statistical paradigm, employing both multivariate generalizability theory and many-facet Rasch measurement, in order to show evidence of validity of the underlying test construct, both within and across levels of ability. The second part of the study used an exploratory-interpretive paradigm, involving a discourse analysis of the pragmatic meanings present in native speaker responses and nonnative speaker responses at different ability levels. This analysis, informed by interactional sociolinguistics, was used to lend support to the statistical evidence presented, specifically in relation to the efficacy of the test tasks in eliciting selected dimensions of pragmatic knowledge, and to demonstrate how a range of pragmatic meanings could be scored by the raters.;The results of the study presented empirical evidence in support of the validity of the underlying construct operationalized in the speaking test. The findings suggested that pragmatic knowledge was highly context-dependent, reflecting an interactionalist approach to construct definition. The results also suggested that pragmatic knowledge should be explicitly tested at all levels of ability, and at the expert level, in particular. Furthermore, the results suggested that knowledge of grammatical form and meaning should be tested separately at all levels Finally, findings from this study showed that highly contextualized tasks can be used to elicit a range of pragmatic meanings that are scalable across different levels of language ability.