A Study On Rater Bias Patterns In Rating CEPT Writing

In order to have a further study on writing scoring, including analytic scoring andholistic scoring, the study, based on IRT theory and multi-facet Rasch model, adoptedstatistic software package Facets to detect the reliability and bias interactions inanalytic scoring and holistic scoring as well as attempt to summarize systematicpatterns in rater bias. The holistic scoring was involved as a reference for observinganalytic scoring results.The procedures of the research are summarized as follows: first, select writingscripts based on stratified sampling method; second, fifteen raters with sufficientrating experience rate all the writing scripts in two rating sessions; and third, after aprimary examination of the data, conduct multi-facet Rasch analysis to get thereliability and observe rating bias patterns and systematic bias.Seen from the data analysis: first, analytic scoring had similar severity level withholistic scoring and had the least bias interactions with15raters who all exhibitedgreat intra-rater reliability, though raters9,12and10showed relatively more centraltendency; second, the final analytic scoring had only one lenient-bias interaction, withrater15, while rater10had only one lenient-bias interaction, with category “grammar”.Raters showed reverse bias direction towards category “task” and “mechanism”, andsame bias direction with “task” and holistic scoring. Third, raters displayed much moresignificantly lenient bias than severe ones, tended more bias towards high level writers,and exhibited more severe bias interactions with lower ability writers and more lenientbias towards higher ability writers. Rater1showed no significant rater-writer biasinteraction, and rater2reported more significant severe bias interactions than lenientones，which was reverse to other raters. Fourth, the analysis by Facets can facilitateresearch in systematic rater bias patterns and thus help improve rater training.The findings in the study suggest that special rater training may be conducted forthose categories and writer groups that attracted more rater bias. In addition, thedescription of the categories in the rating scale may also be studied and adjusted forconsistent interpretation by raters, thus reducing rater bias, which also has empiricaland practical implications for rating scale improvement. At the end of the study, somelimitations are pointed out and further research suggestions are proposed.