The Effects Of Question Preview In Multiple-choice Listening Comprehension Tests

Compared with paper-and-pencil test, internet-based language tests have provided more options for MCQ test procedures. By examining some well-known large scale tests in China like CET, and international tests,like TOEFL and IELTS, it's not difficult to find that there is great divergence in practice on the issue of question preview. This may stimulate intense interests in the question whether there are significant differences or not in test difficulty and validity across different MCQ test procedures.In this quasi-experimental study, efforts are made to examine the effects of question preview on test difficulty and validity based on both quantitative and qualitative evidence. The four test procedures entail:Procedure A: allowing test takers to preview the full question, that is, both the question stems and answer options;Procedure B: allowing test takers to preview question stems but not answer options;Procedure C: allowing test takers to preview only the answer options, but not the question stems;Procedure D: allowing test takers to preview neither question stems nor answer options.Two independent experiments are carried out in the study. The first experiment provides quantitative evidence by analyzing test performances of 467 Chinese EFL learners from Shanghai Jiao Tong University on a listening comprehension test in which four MCQ test procedures are adopted, who are non-English major freshmen with a wide range of English proficiency levels. Besides, all the participants also complete questionnaires reporting their comments on each test procedure. Results indicate that test procedure A ( FQP) with both question stem and answer option preview produces significantly more correct answers, while procedure D (NP) with no question preview produces significantly less correct answers, with the other two procedures B (QSP) and C (AOP) falling between. Questionnaire responses suggest that subjects believe that question preview can supply extra information and thus facilitate their listening comprehension. The second experiment, an immediate retrospection study, attempts to provide qualitative data by examining other 12 Chinese EFL learners'listening strategies and tactics in completing the listening test. It is found that there are no significant differences between the four test procedures in test validity.This paper concludes that the choice of different test procedures leads to differences in test difficulty, but not in test validity. Test takers can benefit from previewing question stems since the previewed question stems provide extra useful information; whereas most test takers suffer from the absence of both question stems and answer options before listening. The researcher recommends that the procedure allowing only question stem preview should be widely adopted.