A Probe Into Test Method Effect In Reading Comprehension Tests

If tests are to provide an accurate measure of learners' language abilities, testers must try to minimize the influence of test method on test results. This research intends to investigate the effect of two different test formats, i.e., multiple-choice (MC) questions and information transfer (IT) tasks, on both the difficulty levels of the items and second language learners' performance in reading comprehension (RC) tests. In particular, item response theory (IRT) was applied to make item difficulty estimates. As planned, two reading comprehension tests were delivered to two groups of high school students simultaneously. In the tests the MC items and IT items used for comparison are based on the same texts and test points. Moreover, statistics indicate that there is no significant difference in the proficiency level of two groups of subjects. Finally, the results, which include all the candidates' scores and the item difficulties obtained through Rasch analysis, were analyzed. The main findings are: 1) subjects' scores in response to MC items are significant higher than those in response to IT items. 2) The difficulty levels of IT items are significantly higher than MC items. This proves that there is test method effect existing in the RC test in the present research.