A Multi-Perspective Validation Of The TEM4 Reading Module

The Test for English Majors---Band 4 (TEM4) is the only large scale nationwide criterion-referenced test for English majors at the foundation stage in mainland China. Its result has considerate impact on test takers, English teaching and society. Validation of this large scale and high-stakes test should be a continuous process in which different data need to be gathered from various aspects and perspectives.To satisfy the new standards required in the new Syllabus for TEM4 edited in 2004, some tasks in TEM4 have been adjusted since 2005, in which reading module has been changed from two secions (fast reading and in-depth reading) to only one section (in-depth reading). However, the new reading tasks are more demanding in speed and difficulty than the old ones. For various reasons, the new TEM4 reading tasks adopted in 2005 and after have been short of extensive and intensive validation until now. In addition, although some researchers have validated the different aspects of old TEM4 reading tasks, there are still unexplored areas which need further study and analysis.Taking the new TEM4 (2005) reading module as an example, this paper attempts to validate TEM4 reading tasks by investigating the divisibility and components of TEM4 reading ability, examining whether test designer's intentions and purposes have been achieved in the real operation and studying the effects of some factors on the TEM4 reading score.First, three hypothesized models are compared by using structural equation modeling (SEM) to investigate the divisibility and components of TEM4 reading ability. The research result shows that the multi-divisible model is unacceptable while both bi-divisible and unitary models are acceptable. However, by comparing goodness-of-fit indexes of these two acceptable models, the researcher finds that the bi-divisible model fits the data of TEM4 (2005) reading tasks better than the unitary one. In the meanwhile, the result of exploratory factor analysis indicates that most of the literal comprehension questions designed to test local reading ability cluster around one factor while most of other questions designed to test global reading comprehension ability belong to another factor. This implies that TEM4 reading ability is possibly composed of basic literal comprehension ability and complex reading comprehension ability. But these two factors can only account for about 25% variance of the total TEM4 (2005) reading score, which shows there are still other factors affecting the TEM4 reading score.Second, the paper investigates whether test designer's intentions and purposes have been achieved in operation by analyzing the results of questionnaires and test takers'think-aloud protocols.The questionnaires respectively examine the agreement percentage on text type, text difficulty, topic familiarity for test takers, question type, question difficulty and the focus level of each answer between the test designer and the teachers, and between the test designer and the test takers. The research result shows that both the agreement percentage between the test designer and the teachers and the agreenment percentage between the test designer and the test takers are all above 50%. Meanwhile, the result of Kendall's W coefficient test refuses the null hypothesis of judgment inconsistency among the test designer, teachers and test takers. All of these results prove that the test designer's hypotheses and judgments on these six aspects are basically correct.In addition, the paper also attempts to validate the reading tasks in TEM4 from test takers'think-aloud protocols while they are taking the test. The result shows that about 73% of the test takers find their answers to different types of questions from different levels of texts as expected by the test designer; the test-taking strategies adopted by three proficiency levels of test takers have both similarities and differences; most of the test takers try to find answers with valid test-taking strategies and the probability of guessing right answers with test-wiseness strategies is less than 1%.The data collected from questionnaires and test-taking processes with different research methods have proved that test designer's intentions and purposes have been basically achieved in operation although there are still some texts and items inconsistent with the test designer's expectations.Finally, the paper explores the effects of L1 reading ability, L2 language proficiency, topic familiarity, ability of using cognitive and metacognitive strategies on the TEM4 (2005) reading score with correlation analysis and regression analysis and the effects of text type and question type on the TEM4 (2005) reading score with the analysis of variance (ANOVA). The result of correlation analysis reveals that the TEM4 (2005) reading score has a significant correlation with L2 language proficiency, L1 reading ability and the ability of using metacognitive strategies. The result of regression analysis shows that L2 language proficiency is the primary predictor to the TEM4 (2005) reading score and L1 reading ability, which has an interaction with test takers'levels of L2 language proficiency, is the secondary. The result of ANOVA illustrates that different text types and question types have significant influence on the TEM4 (2005) reading score.In summary, all the evidence gathered from various aspects and perspectives with different qualitative and quantitative methods shows that TEM4 reading ability is divisible and it is theorectically and empirically sound to assess TEM4 reading ability with different question types designed to test basic literal reading ability and complex reading ability; test-designer's intentions and purposes can be represented by and achieved in the TEM4 (2005) reading tasks; the main factor affecting TEM4 (2005) reading score is the test takers'L2 language proficiency, and text type and question type also influence TEM4 (2005) reading score at the same time. Results of the present study provide substantial evidence for the design and validation of TEM4 reading tests, shed light on the research into the construct of L2 reading ability and other studies of L2 reading testing and meanwhile have some valuable implications for the teaching of L2 reading.