Validation Of TEM-8 Writing Test Rating Rubric

As one of the skills necessary to learn English,writing has always been an important component in TEM-8,which also shows the importance of English writing ability.TEM-8 writing test is a comprehensive writing test in which the rater judges the writing performance of the candidates according to the rating rubric.Therefore,as an important tool to measure and evaluate students’ writing ability,the validation of the rating rubric is very important.In previous study,relevant scholars have verified the validity of some large scale examination rating rubric from the perspective of text features,providing text-based evidence support for scale validity.However,there are few studies on the validity of the rating scale of the TEM-8 writing test from the perspective of text features,and the indicators,especially on the content,are not comprehensive enough.Therefore,based on the validity verification framework of Knoch(2017),120 texts are selected to verify the validity of the TEM-8 writing test rating rubric from the perspective of text feature analysis.According to the descriptors of the rating rubric,the current professional software for analyzing the characteristics of English writing text,and the indexes designed by the author on the basis of the literature,a total of 93 analysis indexes are determined.Among these indicators,41 indicators correspond to the sub-item of"language use",34 of them correspond to the sub-item of "textual organization",and 18 indexes correspond to the sub-item of "content" in the rating rubric.On this basis,the research questions of this paper are as follows:(1)What is the difference of each indicator corresponding to the "language use"in different ranks,and what is the correlation with its writing score?(2)What is the difference of each indicator corresponding to the "textual organization" in different ranks,and what is the correlation with its writing score?(3)What is the difference of each indicator corresponding to the "content" in different ranks,and what is the correlation with its writing score?(4)How can these indicators predict writing scores?The major findings are:among the 93 analysis indicators,there are 49 effective indicators,20 indicators corresponding to the "language use" sub-item,17 indicators corresponding to the "textual organization" sub-item,12 indicators corresponding to the "content" sub-item in the scoring table.As for correlation analysis,parallel relationship is negatively correlated with writing performance,while other indicators,including lexical density,lexical variation,mean length and fluency are positively correlated with writing performance in language use.As for "textual organization",five indicators(CNCAll,CNCCause,CNCAdd,RDFRE,and SMCAUSr)are positively correlated with writing performance,while other indicators are negatively correlated with writing performance.In terms of content,three indicators,including citation density,direct reference,and citation as evidence,are negatively correlated with writing performance,while other indicators are positively correlated with writing performance.By regression analysis,eight variables,including causal connectives incidence(CNCCause),corrected type-token ratio(CTTR),number of different words in the first 50 words(NDW-50),squared verb variation(SVV1),adverb variation(ADVV),summary,discussion,and the number of evidence can explain 79.4%variance of writing score.The results show that the relevant indexes corresponding to the three sub-items of TEM-8 writing scoring scale can significantly distinguish texts of different ranks.As for the TEM-8 rating rubric,the descriptors corresponding to different ranks are consistent with the text features,such as lexical richness,syntactic complexity,cohesion and coherence,and content organization.Besides,the descriptors in the TEM-8 rating rubric are accurate and unambiguous.Therefore,the TEM-8 writing test rating rubric is of good validation in general.Besides,further differentiation and refinement are also needed in a few descriptors.For example,it is necessary to further distinguish cohesive devices in different aspects and their reflected coherence characteristics in the rating scale.Finally,this study provides some suggestions for the rating rubric of the TEM-8 writing test according to the research.