Effects Of Rater-scale Interaction On EFL Essay Rating Outcomes And Processes

This research is an attempt at investigating the effects of rater-scale interaction on essay rating outcomes and processes in the context of CET6.A group of9experienced CET6raters rated the same batch of60CET6essays produced in an operational CET6administration twice, using both the CET6holistic essay rating scale and an analytic rating scale designed specifically for the study. In order to collect data on raters’ rating processes, all the nine raters provided think-aloud protocols while rating10of these essays. In addition, the raters also completed two questionnaires as well as two semi-structured interviews about their rating process as well as their perception of the rating scales. The think-aloud protocols were coded in terms of the rating strategies adopted as well as the aspects of writing attended to. The results were then compared quantitatively across rating scales. Meanwhile, interpretative analysis was also carried out on both think-aloud protocols and raters’responses to questionnaires and semi-structured interviews. Essay scores were analyzed using G-theory and MFRM to estimate both facet-and item-level reliability indices across rating scales.With regard to essay rating outcomes, it is found that the use of the analytic scale led to finer distinctions among examinees in terms of their English writing ability and higher proportion of examinees with acceptable fit. Meanwhile, though there was considerable variability in terms of rater severity with the use of both scales, the impact of this variability on examinee ability estimates was smaller with the use of the analytic scale. What’s more, while there was a lack of clear distinction between most adjacent holistic scores, this problem was not detected for the analytic scale categories. All in all, though the use of the analytic scale led to higher proportions of rater-examinee and rater-scale interactions, the use of this scale still led to more favorable impact on essay rating outcome. As to essay rating processes, it is found that the degree of conformity between raters’ understanding and application of the rating criteria and that stipulated in the scales differed across the scales. Meanwhile, such interaction affected both the types and frequencies of rating strategies adopted and the specific aspects of essays attended to. On the whole, when applying the analytic scale, raters tended to focus more on scale-based criteria and there also seemed to be more similarity in their understanding and application of the scale.