Rating Scales For Summary Writing Assessment:construction And Validation

Reading-to-write tasks,which aim to assess students' ability in accomplishing writing tasks based on their comprehension of source texts,are attracting more attention and given more emphasis to due to its high degree of authenticity in testing contexts.As an important member of this task category,summary writing has been employed in many large-scale and high-stakes English tests home and abroad,for instance,TOEFL iBT,TEM4/8(since 2016),NMET Shanghai version(since 2017).However,most of the rating scales used in actual ratings of summary writing have been constructed only on the basis of theoretical analysis and expert judgement,and rating scales developed on the basis of authentic test performances and students' test data are quite rare.In addition,classroom-based formative assessments are short of rating scales of summary writing for peer or self-assessment.Therefore,the present research,by adopting a data-based approach,aims to construct rating scales for both teacher raters and peer raters,which are expected to well represent the construct of summary writing,and to be reasonable,normative,and convenient in real use.Validation of the two rating scales was conducted from multiple perspectives.In the phase of constructing rating scales,the present research initially collected and extracted textual attributes of summary writing in the following ways:(1)exploration of the construct of summary writing based on literature review;(2)analysis of rating scales of summary writing in current English tests;(3)questionnaire-based data collection;(4)8 teachers' free commentary feedback on 2samples of summary writing scripts;(5)collection of 325 college English teachers' feedback on the textual attributes using a five-point Likert scale questionnaire;(6)determination,on the basis of questionnaire results in(5)and expert judgement,of the20 final textual attributes to be used to write descriptors for rating scales.Furthermore,SPSS and AMOS were employed to conduct respectively exploratory and confirmatory factor analysis,which helped to determine that the rating scale for teacher raters should consist of such 5 dimensions as Linguistic Accuracy,Coherence and Cohesion,Fidelity to Source Text,Linguistic Complexity and Normalization.With multiple linear regression analysis and expert judgement,the present research determined that the weightings for the 5 dimensions should respectively be 25%,25%,20%,20%,10%.With the above results,the present research developed the rating scale for teacher raters,and the rating scale for peer raters was constructed on the basis of the preceding research.Considering students' limited linguistic proficiency and cognitive ability,the peer-version rating scale,compared with the teacher version,was made easier to comprehend by removing complicated jargons and using plain language.Besides,the peer version was presented in a 5-point format rather than the format of score range in the teacher version so as to facilitate students' making decisions in their rating.In the phase of validation,the present research employed MFRM,correlation analysis,independent samples t test,TAPs and semi-structured interviews to validate rating scales for both teachers and peer raters.On the whole,the two rating scales were clear in diction,convenient in use and suitable in actual application.The rating scales were satisfactory in their discriminative power and could well guarantee intra-rater consistency.The deficiency of the teacher-version rating scale was that raters found it hard to make objective judgements and were prone to random use of scores because each level on the 5 dimensions of the rating scale consisted of a certain range of scores.Such deficiency,however,was not detected in the peer-version rating scale,which had been constructed with a 5-point format.In addition,all teacher raters unanimously agreed that the weighting of the dimension of Fidelity to Source Text be lifted and the criteria for judgement of plagiarism be lowered.Correlation analysis and independent samples t test revealed that English proficiency did not have significant effects on peer rating and the results of rating using the peer-version rating scale were quite reliable,indicating that the rating scale had practical significance and promotion value.Finally,the present research made improvements in the two rating scales based upon the validation results.Thus,both rating scales have demonstrated satisfactory validity in actual use.The present research has,based on its findings,proposed a working model for construction and validation of rating scales,which can be used for reference in construction and validation of rating scales of summary writing on other levels or even rating scales for other English writing assessment tasks.