Using Multi-facet Rasch Model Analyzing Rater Effects In Writing Scoring

The Multi-facet Rasch measurement approach was used to investigate four types of rater effects: severity/leniency, rater instability, halo effects and extremism/central tendency. 9 raters participating in this study rated 30 essays written by Meiya ESL students of Hunan University on two writing tasks. We adopted a 7-point analytic rating scale which consisted of five separate domains: content, organization, language, vocabulary, and. mechanics. We analyzed the ratings using a Multi-facet Rasch model implemented by the software package FACETS. Multi-facet Rasch model, built upon the modern Item Response Theory (IRT), models each examinee's performance based on his latent trait. The performance of individual examinee on a certain item can be predicted by his item response function. The IRT approach is different from the Classical True Score Theory's way of analyzing through observing raw score.This paper first investigated the severity levels and the potential halo effects of the raters using the primary analysis of the Multi-facet Rasch model and then examined whether raters remained stable across the two writing tasks and the five domains through the bias analysis. We got the following results:1) Raters showed significant differences in their severity level, with rater 9 being the most severe rater and rater 5 being the most lenient rater.2) Raters behaved consistently in rating the two different writing tasks but 7 raters showed bias towards certain domains.3) Only rater 1 and rater 2 exhibited significant halo effects, i.e., they tended to give similar scores to different domains. The other 7 raters were able to differentiate the five domains.4) Rater 4 and 6 didn't use the full range to score; rater 1, rater 4, and rater 6 were exhibiting central tendency effects.This study provided implications for enhancing the rater reliability in a writing performance test. According to the model, those ambiguous or rough descriptions could be amended; it provided reference for the selection of the raters because it could trace down the individual performance of the raters; those unreliable raters should be trained again or get replaced; finally, those aberrant ratings could be adjusted according to the model measurement. The Multi-facet Rasch model proves itself to be a powerful weapon in quality monitoring of the writing scoring and is of great significance to guarantee the reliability and objectivity of the writing assessment.