Validation Of Diagnostic Rating Scales For An English Speaking Test

In the field of EFL testing,assessing speaking ability is a topic that has been receiving much attention for a long time.Since the last two decades,researchers have been making modifications to both the theoretical basis and the implementation methods to assess EFL speaking ability,which in turn contributed to the application of several large-scale,standardized EFL speaking tests.With the development of testing theories,researchers start to realize the deficiencies of conventional EFL speaking tests.Traditional speaking tests tend to reflect a learner's speaking ability by providing a holistic score or crude grades,yet a detailed profile of strengths and weaknesses would be more desirable to individual EFL learners.Therefore,against this backdrop,the concept of "diagnostic assessment" has been brought into the research scope of language testing experts,thus becoming a new direction of research.Currently in China,applying EFL speaking tests for diagnostic purposes is still an under-researched sub-area.Moreover,to develop a valid and feasible diagnostic spcaking test,the priority concern is the development of a diagnostic rating scale.This study sets out to examine the validity of diagnostic rating scales.Based on the Communicative Language Assessment model(CLA)(Bachman&Palmer,1996)and the defining features of diagnostic assessment raised by Alderson(2005),this paper designed two diagnostic rating scales specifically for one pre-existing EFL speaking test task while referring to several currently widely-acknowledged rating scales in proficiency tests of EFL speaking ability.To achieve comparative analysis,this research attempts to control the number of band levels,dimensions or traits for rating,and contents of the band level descriptors while leaving two factors varied in the two rating scales,including the layout of the rating scale and richness of descriptors for each scoring level.7 raters rated the same 30 audio samples using the two rating scales respectively.Following each rating practice,an open-ended questionnaire is administered to explore the raters' perceptions about the rating scale utilized in this task.By adopting a mixed-method approach for data analysis,the scoring outcomes were analyzed via multi-facet Rasch model(MFRM)to investigate and compare the validity of the two rating scales.Furthermore.thematic analysis is conducted to code the responses from questionnaires to help elicit the raters' behaviors in the rating process.Based on the findings of both quantitative and qualitative analysis,the study reconsiders the questions of interest that emerge in analysis,illustrates the differences between the two rating scales in terms of validity,and then investigates the practicality of embedding diagnostic rating scales in traditional speaking test context.