| The purpose of the study was to explore the use of a response-process-oriented approach and the Rasch partial credit model for designing a multiple-choice ESL grammatical proficiency test. The options of the items were based on cognitive and linguistic analyses of the processes underlying examinee responses to an item and scored on a partial credit scale.;A theoretical/substantive basis for item construction was established by providing a contextualized definition of the construct, defining the content domains and content elements, and hypothesizing the processes underlying examinee responses for each domain. Partial credit scoring schemes were developed using a combination approach based on: (1) a priori specification of option weights, (2) experts' rating of response option difficulty, and (3) a iterative empirical weighting procedure. A variety of methods including fit statistics, residual analysis, experts' judgment, and substantive analysis were used in item fit analysis which provided the basis for item selection.;Test validation focused on (1) content relevance, (2) psychometric properties (response consistency, item discrimination, minimal guessing, and test difficulty), (3) trait validity, and (4) perceived test utility. The results indicate content relevance was reasonably well attained in terms of the match between the items, their intended assessment objectives, and their perceived curriculum relevance. The desired psychometric properties were also well attained as indicated by IRT-based and correlational analyses. The evidence for trait validity was inconclusive. And the test was shown to be potentially useful not only as a proficiency test, but also as a diagnostic test for both classroom teachers and second language acquisition researchers.;The major implication of this study is that, with a sound theoretical basis and refined psychometric techniques, multiple-choice tests can be made more rational and explanatory in terms of the construct measured than they are commonly thought of. |
