Research Into Raters' Reading Styles In NMET English Writing Evaluation

Writing tests are known for its high validity but relatively low reliability, because scoring is a crucial factor in which raters, or markers show great differences. Raters, even when they are experienced and trained to use and interpret rating scales in similar ways, differ widely in their rating behavior. More importantly, raters seem to exhibit distinct reading styles, each style characterized by rater-specific ways to focus on, and to process essay-relevant information (Eckes 2008).The purpose of the present study is to provide some implications for further research into rater variability, rater monitoring, and rater training through the classification of raters'reading styles.The thesis proposes two hypotheses. One is that experienced raters fall into different styles or clusters that are clearly distinguishable from one another in respect of the importance they attach to scoring criteria. The other is that raters can be classified by the way they read the essay.To examine the first hypothesis, we asked 45 raters actively involved in the NMET writing scoring to indicate how much importance they would attach to each of the seven routinely used criteria based on a four-point scale (less important, important, very important, extremely important).The criteria covered various performance aspects, such as grammar, vocabulary, syntax, content, fluency, legibility and organization. In a preliminary step, two-facet Rasch analysis revealed that the criterion importance scale functioned as intended and raters differed significantly in their views on the importance of the various criteria. A Quick cluster technique yielded a joint classification of raters and criteria, with six rater types emerging from the analysis.Each of these types was characterized by a distinct scoring profile and they are Content Type, Grammar Type, Legibility Type, Organization Type, Two-category Type (Content and Grammar), Non-legibility Type. Finally, their written reports on an NMET essay revealed the operational behavior of different rater types.However, ANOVA analysis found out that the scores derived from different rater types had no significant difference.As for verifying the second hypothesis, the 45 raters were asked to provide retrospective written reports on how they read the essay (read or scan, once or twice) and their thinking process in the course of reading or scanning. After quantitative and qualitative analysis of the written reports, four styles were found, such as "the read through","read twice","read through then scan","scan then read through",and the distinct thinking process of different types was also obtained, of which the "read through then scan"style proved to be more efficient and effective. However, the scores awarded by raters who adopted different reading styles showed no discrepancy.The classification approach to rater variability hasn't been tried out so far in the context of China. Therefore, the findings of this research have significant implications for writing assessment and rater characteristic research in Chinese NMET and other large-scale high-stakes examination contexts.