Rater Effects In Engilish Writing Test

Rating scale types such as holistic and analytic scales, and rater effects such asseverity or leniency, consistency are the source of variance in observed ratings andthus affect the appropriation of score interpretations. This thesis investigated theeffects of different rating types and main rater effects on the rating procedures andultimately on the scores by a comparative study of holistic and analytic scorings. Thestudy does have important implications for the choice of rating scales, rater training,and implementations of rating operations to improve the way of English writingassessment.Four raters took part in this experiment. They were trained and instructed toscore the scripts using a holistic scale and an analytic scale respectively. Then the datacollected from the two scorings were separately analyzed with the softwares of SPSSand FACETS. The SPSS analysis showed that, in the holistic scoring, raters differedsignificantly in the ordering of examinees but agreed on the mean scores; in theanalytic scoring, raters agreed on the ordering, but one rater significantly differedfrom the others in the means; scores between the two ratings were correlated butdifferent in the means.The FACETS analyses (a) provide support for the notion that analytic scales aremore likely to result in the consistency of ratings than holistic scales and thus arefavorable on the ground of reliability, in particular when raters are less experienced ordifferently oriented;(b) imply that holistic scales can be more appropriately used inthe large-scale exams on the condition that all raters are experienced with holisticscorings;(c) suggest that, in any of the ratings, test developers should monitor ratersconstantly on the basis of the rater effects using FACETS program or other methodsto get the scores as fair as possible.