An Empirical Study Of Rater Effects In Chinese Pragmatic Test

This research has studied raters’ rater effects, which includes overall severitydifference, rater-criterion interactions, rater-item interactions and central tendencyeffect, by controlling raters’ variables, to make comparisons of raters’ rating behavior.A total of6raters and30examinees have participated in this study. Among the6raters,3comes from the north part of China and3from the south part;3of them aremale raters and3are female.The research results indicate that all6raters come within the acceptable rangeof self-consistency, however, they have demonstrated great inter-rater severitydifference:18levels of severity. Moreover, their bias interaction patterns with testitems are diverse and widespread, mainly because of the social parameter of thispragmatic test. There also exist some group differences: the north raters aresignificantly harsher than south raters, as two of male north raters have includedcandidates’ cross-cultural competence into criteria. Moreover, north raters are moreopened-up and warm-hearted towards strangers. Besides, male raters have moresense of status, power than female raters.The research hence yields the following conclusions: because of raters’ differingvariables and the social parameter of this test, raters have shown great ratervariability. Hence, for validity and reliability concerns, it is of great significance toconduct through and targeted training to raters rating Chinese pragmatic test.Moreover, this research has demonstrated the feasibility of combing quantitative andqualitative research in studying rater effects. To base qualitative analysis on thequantitative data generated by MFRM, is of great utility to enhance the effectivenessof rater training.