Study Of Sources Of Score Variability In Performance Testing Using Many-facet Rasch Model

As a comparatively direct measure of learners' communicative language ability, performance test (typically writing and speaking test) claims construct validity and a strong power for predictive utility of test scores and is gaining increasing interest both in practice and research in language testing. Despite the seemingly direct relationship between the underlying constructs and scores, a routine but more complicated procedure should still be followed in test validation. The present thesis aims to give a general overview of justifications for and problems involved in test validation in performance testing context and illustrate how Many-facet Rasch Model (MFRM) could be adopted as an effective tool in addressing problems in test validation.As an illustration, the present study investigates and models possible sources of score variability within the framework of MFRM, based upon raw scores of CET-SET (College English Test-Spoken English Test) administered on 19^th and 20^th, Nov. 2005 in Hangzhou. The results demonstrate that CET-SET could successfully separate candidates into statistically distinct levels of proficiency and facets in the model such as rater, task, rating domains and rating scales generally function well within certain limits. However, tendency to give central ratings along the scale, less variability across three rating domains and certain patterns of bias in raters' rating pattern are detected. Fair scores for each candidate are also provided, minimizing the variability due to facets other than candidates' ability.MFRM manifests itself as effective in detecting whether each test method facet functions as intended in such performance test setting and providing useful feedback for test improvement.