Construct Validity Of TEM4 Reading Test Evidence Based On Response Processes

TEM4(Test for English Majors-Band 4)is a criterion-referenced test designed for English majors.Its aims are to examine whether students have reached the requirements prescribed by the syllabus,while at the same time examine whether students master the language ability to apply various basic skills.As a large-scale and high-stakes test,TME4 requires continual validation with evidence collected from different sources,especially when there is some adjustment to it.Some research has been done on validity of TEM4 administered in the past years,but the annual emergence of new test paper always calls for a new round of validation.With the release of the latest TEM4 of the year 2019,this new round of validation is greatly needed.This study focuses on the reading test in TEM4 of 2019,with an aim to verify its construct validity.Specifically,it collects validity evidence from test takers' response processes and answers the question to what extent TEM4 reading test measures what it is intended to measure.The participants were 45 English major sophomores from a comprehension university in Jiangsu Province,who finished a mock reading test of 2019 TEM4,which measures four levels of reading comprehension,namely literal comprehension,interpretation and reorganization,inference as well as evaluation.All the reading skills expected by the test specification were written in a checklist,and students were asked to choose the skills they used while they were responding to each test item.After that,ten students who took the mock test participated in the stimulated recall protocol,providing further insights into the response processes.The findings are rendered as follows:First,2019 TEM4 basically measures what it is intended to measure.All the 19 expected skills are used,but not by all the test takers.On average,each test taker uses 60.82%of the expected skills and each skill is used by 56.02%of the test takers.In addition,SAQ has a higher validity than MCQ,and expected skills are used more appropriately by higher scorers.However,under-representation of construct also exists.Second,in addition to the reading skills given by the test specification,test takers also apply other reading skills in their response processes.These skills mainly include recognizing key words,phrases or sentences;using L1(E-C translation)for better understanding and summarizing main ideas of paragraphs or the whole passage.These unexpected skills are construct-irrelevant variance which diminish the construct validity.In light of the findings of the present study,the implications can be drawn that combination of different question formats(such as MCQ and SAQ,but not exclusively)should be adopted more widely in reading tests.And grammatical knowledge may call for attention in teaching of reading.Furthermore,for test takers,attention should not only be paid to test scores,but to flexible use of various skills.