A Study On The Validation Of An EFL Writing Rubric

This study aims at validating a new EFL writing rubric-the Comprehensibility Model, which was originated by He Xiangming, a professor in Capital Normal University, in 2005. This new model is different from traditional writing rubrics in that its three subscales- comprehensibility, maturity and creativity-are designed to fit the unique EFL writing environment and needs.385 English Majors (from freshmen to senior students) from the English Education Department of Capital Normal University participated in this writing assessment. Through the Pearson correlation and other functions of SPSS 10.0 and Excel, the results of the new rating scale and the comparison rubric with that of the objective measures were compared, and correlation between both methods were calculated. Then comparisons between the inter-rater reliability using both the comprehensibility rubric and the comparison rubric, and further comparisons across rubrics were done. Finally raters' reflections and students' questionnaires were collected for a composite analysis of the utility of the comprehensibility rubric.The results of this study showed that: 1) Although both scores of the comprehensibility rubric and the comparison rubric were highly correlated with the objective measures, compared with the comparison rubric, the score of the comprehensibility rubric had a higher correlation with the objective measures, the score variance of the comprehensibility model is closer to the results of the objective scoring. 2) Raters reached consistent agreement using both rubric, but compared with the comparison rubric, the comprehensibility model had a higher reliability coefficient between raters. 3) Though the overall score obtained from the comprehensibility rubric was lower than that from the comparison rubric, compared with the comparison rubric, the comprehensibility model tended to predict the writing development of different grades more precisely. 4) The scores for the comprehensibility rubric and the comparison rubric are significantly different. Except the creativity subscale, the other subscales of both rubrics are highly correlated with each other i.e. most scales for both rubrics are not empirically distinct. 5) Raters' reflections showed that compared with the comparison rubric, the comprehensibility model was more practical and more popular among the raters. The comprehensibility model is regarded to have more potential positive washback effect. However, raters also put forward some problems in the application of the comprehensibility rubric. 6) Students' questionnaires showed that the comprehensibility model is in line with how subjects perceive the rating scales, but a large number of students tended to neglect the creativity of writing.Though this tentative study here has provided satisfactory evidence for the validity and reliability of the comprehensibility rubric in large scale writing assessment, it is in no way conclusive. The author of the present study believed that further study is needed to explore the validity and reliability of the comprehensibility model based on a larger size of raters. Besides more emphasis and training should be given to the creativity subscale in the future study to make it more specific and understandable.