The Development And Validation Of An Analytic Rating Scale For The Writing Assessment Of College English Test Band 4

In recent decades,with the ever-growing demand of score meaningfulness on behalf of test users,test developers around the world are deemed responsible for providing defensible and explicit interpretations of test scores so as to encourage appropriate test use(Chapelle,Enright & Jamieson,2008).Within such a context,the writing assessments adopted in large-scale EFL tests have increasingly aroused validity concerns.The focus of these concerns is that the construct that the writing assessment generally measures remains implicit,which,as a result,greatly hinders the interpretation of the score meanings.In view of these concerns,the rating scales used in EFL writing assessments have also attracted research attention due to the widespread consensus that the rating scale represents the de-facto test construct of writing assessment(McNamara,1996;McNamara,2002;Turner,2000;Weigle,2002).However,previous research has revealed that the status quo of the commonly used rating scales in writing assessments is not as inspiring(Brindley,1998;Knoch,2009;Todd,Thienpermpool &Keyuravong,2004;Upshur & Turner,1995),which apparently does not correspond with the significant role that rating scales are assumed to play in the writing assessment.What made the situation even awkward is that there are only a few empirical studies devoted to the development of rating scales for EFL writing assessments in real test contexts.Such a paucity deserves immediate research attention considering the fact that writing assessments have been widely adopted by large-scale EFL tests both nationally and internationally(e.g.,TOEFL,IELTS,CET,TEM)as an indispensable test component.The current study,therefore,set its research scope on the writing assessment adopted in the CET4(hereafter the CET4 writing),one of the largest-scale EFL tests in China.Specifically,the study was conducted to develop and validate an analytic rating scale for the CET4 writing(hereafter CET4 W-ARS)with a view to addressingthe validity concerns over the CET4 writing on one hand,and on the other,to providing theoretical and methodological implications for rating scale inquiries in other contexts.To fulfill this purpose,the study addressed two general research questions with each further breaking down into three sub-questions:1.How should an analytic rating scale be developed for the CET4 writing?1)How well do CET4 raters perceive the existing rating scale used in the CET4writing?2)How can the construct of the CET4 writing be defined and how can the construct be operationalized in an analytic rating scale ?3)How can the levels of the analytic rating scale be described?2.To what extent is the scale valid when used to rate the CET4 writing performance?4)How effective is the new scale in discriminating the CET4 writing performance?5)To what extent can each of the rating categories on the new scale function as intended in the rating of CET4 writing performance?6)How well do the CET4 raters perceive the effectiveness of the new scale?Different from traditional initiatives to develop rating scales solely on the basis of experts' intuitions,this study featured the adoption of a multi-method approach which integrated various research methods in different research phases.The first research phase adopted a mixed-methods approach with a view to a)investigating the raters' perceptions of the existing rating scale used in the CET4 writing,and b)pinpointing the most useful rating criteria for the CET4 writing.During the first phase,179 raters of the CET4 writing participated in a questionnaire survey,and the collected data were analyzed using SPSS 19.0.Based on the quantitative results,12 raters were invited to follow-up interviews during which their comments were recorded and then analyzed qualitatively,thus further interpreting and complementing the quantitative findings.The second research phase adopted a multi-method approach which took advantage of document analysis,intuitive judgment and Rasch model analysis inproviding adequate descriptors for the CET4 writing,and calibrating the descriptors.During this phase,a descriptor pool was initially set up by referring to such documents as the existing rating scales,curriculum requirements,teaching syllabi,and textbooks.Then the collected descriptors were edited based on a set of principles and categorized by referring to the descriptive scheme.Afterwards,workshops were organized in which 6 experienced raters of the CET4 writing helped examine the descriptor quality.Based on their expertise,revisions were made to the descriptors to enhance their clarity,and relevance with the CET4 writing performances.The best descriptors surviving the workshops were selected to constitute the bulk of a questionnaire to be used in a survey eliciting the CET4 raters' perceptions of descriptor difficulty.The results of the questionnaire survey finally yielded a draft version of the scale.In contrast to the first two phases which were devoted to the development of the scale,the third research phase focused on the a posteriori validation of the scale.This phase mainly adopted a mixed-methods approach to a)collecting both quantitative and qualitative evidence for the post hoc validity of the scale,and b)further enhancing the scale quality.To fulfill such purposes,a rating experiment taking counterbalanced design was carried out in which 21 raters were asked to rate 30 pieces of CET4 writing scripts using the existing rating scale and the CET4 W-ARS consecutively.Following the rating experiment,in-depth interviews were conducted with 10 raters so as to gain more useful insights into the scale validity.In the end,based on the results yielded through both quantitative and qualitative analysis,further modifications were made to the draft version of the CET4 W-ARS.The efforts of the study eventually yielded an analytic rating scale for the CET4 writing which was comprised of 4 rating categories(i.e.Linguistic Range and Control,Content and Idea,Discourse Organization,Linguistic Appropriacy)substantiated by detailed calibrated descriptions spanning across five band levels.When used in rating CET4 writing performances,the new scale could generally function effectively as a whole,whereas the 4 rating categories could also function effectively as individuals.In addition,the raters' perceptions of the scale validity were by and large positive.Onthe whole,the validity of the new scale was satisfactory despite that minor revisions need to be made to the draft version of the scale.In conclusion,the current study focused on the development and validation of an analytic rating scale for the CET4 writing,which is among the few attempts to develop rating scales for the writing assessment of large-scale EFL tests on an empirical basis.The outcomes yielded through this study can not only help to address the validity concerns over the CET4 writing,but have significant implications for the development and validation of rating scales in other contexts.