An Argument-based Approach To Validating A Diagnostic Test

This study built and supported a validity argument for a diagnostic reading test,the UDig reading test.The test aims to diagnose the reading ability of English language learners at the tertiary level in China.Drawing on the interpretive/use argument approach(IUA;Kane,1992,2001,2013),the study sought to examine the most critical and relevant inferences of the score interpretations and uses concerning diagnostic assessment.Two defining features of diagnostic assessment were identified:(1)diagnostic assessment measures and reports on students‘ language subskills;(2)diagnostic assessment gives test users detailed feedback for remedial learning and teaching.To this end,the validation efforts centered on what was being diagnosed,how the diagnostic feedback was interpreted,and how the feedback was used and what the impact was.Accordingly,three validity inferences which were of primary concern in the IUA for the UDig were specified: the explanation inference,the decision inference,and the consequence inference.Corresponding to these inferences,three research questions were proposed:1.To what extent are UDig reading scores accurate indicators of test-takers‘ strengths and weaknesses of reading subskills?2.How do students and teachers perceive the usefulness of UDig diagnostic feedback for making decisions on learning and teaching?3.How is UDig diagnostic feedback used by students and teachers and what consequences does it have on learning and teaching?The IUA for the UDig provided a logical framework which guided the validation procedure and organized validity evidence through the articulation of the claims,warrants,and assumptions underlying the inferences.Employing a mixed-methods research design,the study triangulated multiple sources of evidence to support the validity inferences over two research phases.Phase 1 of the study explored the extent to which UDig scores could be interpreted as indicators of students‘ mastery of reading subskills(i.e.,the explanation inference).Students‘ think-aloud protocols while taking the UDig test and experts‘ judgments were elicited to shed light on the associations between the test items and the reading subskills being measured.Based on the empirical results,a Q-matrix denoting the item-by-subskill relationships was constructed and compared with the initial Q-matrix based on the test developer‘s intention.The Q-matrix with a better model fit(i.e.,the one based on students‘think-aloud protocols and experts‘ judgments)was further validated and modified,resulting in a final Q-matrix.Students‘ mastery profiles were then generated using the G-DINA framework(de la Torre,2011),with the input of large-scale test performance data and the final Q-matrix.Phase 2 investigated the ways in which the diagnostic feedback could serve as the catalyst of remedial learning and teaching(i.e.,the decision and the consequence inferences).At the beginning of an instructional semester,student questionnaire was distributed and the first round of student and teacher interviews was conducted,which provided insights into students‘ and teachers‘perceptions of UDig feedback and their decisions on future use of the feedback.Situated within an Academic English Reading and Writing course,a longitudinal study comprising data from students‘ and teachers‘ reflective tasks and the second round of interviews was then adopted to document how students and teachers utilized UDig feedback throughout the semester,as well as the consequences of the feedback on learning and teaching.Overall,the results provided strong backing for the validity inferences,whereas rebuttal evidence which posed threats to the validity of the UDig was also identified:while the majority of the intended reading subskills were supported by empirical data,a few modifications needed to be made regarding the item-by-subskill relationships;in general,students and teachers expressed positive views of the diagnostic feedback and showed willingness to utilize the information;most of the students and teachers integrated the diagnostic feedback into their learning and teaching process,which in return,exerted positive influences over students‘ English study;a few deficiencies of the diagnostic feedback were pointed out,and detailed recommendations on how to refine the diagnostic feedback as well as the diagnostic procedure as a whole were given by students and teachers.To sum up,there is considerable positive evidence to suggest that the UDig has the potential to serve as an accurate indicator of students‘reading abilities and an effective instrument to promote college English learning and teaching.At the same time,further steps to refine both the item-by-subskill relationships of the test items and the form and content of the diagnostic reports are necessitated in order to improve the quality of the UDig system.It is noteworthy that the specific population and setting in this study all contribute to the evaluation of the UDig.Future research can investigate the use of the UDig by different populations and across various contexts.This study is a new attempt to examine the validity of a diagnostic reading assessment.The findings have implications for the construction and validation of truly diagnostic assessment,the effectiveness of diagnostic feedback,and the application of argument-based approaches to low-stakes,learning-oriented assessment validation.More importantly,the study provides new perspectives on how learning-oriented diagnostic assessment tools can be implemented in the Chinese EFL context.It is believed that with the collaboration of language testers,measurement specialists and test users,the full benefits of diagnostic assessment can be realized.