On Construct Validation In Language Testing

The Test for English Majors Band 8 (shortened as TEM 8) is a large scale nationwide criterion-referenced test for English majors in mainland China. Since its inception, Tem 8 underwent several modifications. The latest modification answering for the new requirements of The 2000 Curriculum for English teaching was made in the TEM 8 Syllabus in 2004. The 2005 version of TEM 8 is the first of its kind designed in line with the new TEM 8 test syllabus. This version substantiated the all major modifications in construct domain, testing time, test content, test formats, relative weight of subtest scores for listening, reading and writing. In short, a new subtest was added and testing time shorted and cognitive load increased. The newly added subtest is the general knowledge, which covers the fields of general linguistics, English and American literature and knowledge of major English speaking countries. Whether such modifications will result in psychometrically acceptable features and whether the test does test what it purports to measure remains unanswered. Moreover, it is generally acknowledged that the more a test is of high stakes in nature, the more demanding the validation should be for the test. TEM 8 is by nature a high stakes test. Therefore, a timely validation endeavor on TEM 8 in line with the state-of-art theoretical validity framework is of significance both theoretically and practically.This research embodies two main aims: one is to review and evaluate the literature concerning unitary validity in the perspectives of ontology, epistemology and methodology in the hope of providing relevant rationale for the validation argument. The second aim is, in line with unitary validity, to provide empirical (statistical) evidence to the score interpretation to the objectively scored items in TEM 8 (2005 version). Due to the many unclaimed testing error variances in subjective items, the study analyzed only the objective subtest data, namely, listening component in MC, reading comprehension in MC, and general knowledge in MC, taking up 40% of the total score. There are four questions in this regard. What are the psychometric properties of the objective items like? Are the tested traits (or dimensions) consistent with what is expected in the test specifications? Do the scores of objectively scored items mean the same across different population groups? Does the subtest of general knowledge demonstrate substantial test bias?...