Evaluating Rater Consistency In The Critical Thinking Skill Tasks

The purpose of this study was to investigate the validity for the critical thinking skill(CTS)tasks in the development from the perspective of raters.CTS tasks are aimed to evaluate the critical thinking skills of the seekers for English teacher positions.As constructed-response tasks,CTS tasks require human raters in the rating process.However,it is revealed that raters are not always assigning scores accurately and consistently,which may threaten the test results as well as the validity of the test tasks.Therefore,the study employed Interpretation/Use Argument(IUA)(Kane,2006,2013,2016)to conduct a validation study.In line with the research questions,the scoring inference for the CTS tasks was constructed and justified by the evidence from inter-rater consistency,intra-rater consistency,rater bias,and the practicality of the scale steps.Data used in this study were the scores of 52 candidates who completed all the three CTS tasks.They were rated by four trained raters on an analytical rating scale.Many-facets Rasch Model(MFRM)and interviews were employed as instruments to analyze the data.Results indicated that 1)despite some severity difference,the raters in general demonstrated an acceptable level of inter-rater consistency(severity range = 0.47 logits);2)raters demonstrated a satisfactory intra-rater consistency(infit & outfit MnSq between 0.94 and 1.05);3)although the raters tended to elicit more interactions facing candidates with high critical thinking skills,they can overall maintain a satisfactory rater-candidate interaction.Moreover,notable rater-task and rater-category interaction were not found in the ratings;4)Scale steps were used as intended.These results can be preliminarily interpreted as evidence to support their corresponding warrants in the claim about consistency of test scores in IUA.The “scoring inference for the CTS tasks” constructed in this study can provide references for relevant validation study.Results of the study,as evidence,can strengthen the validity of the CTS tasks so as to enhance the appropriateness of the interpretation and use of the scores.Moreover,the diagnostic information from MFRM and rater interview may shed light on the rater training program for CTS tasks in the future.