The Effects Of Review On Computerized Adaptive Tests: Psychometric And Attitudinal Results

The present study was intended to contribute some empirical evidence to the decision-making on whether to include review options on CATs in the Chinese context, by examining the effects of review on Chinese examinees under three different item review conditions, namely, non-review condition, 5-item restricted review condition, and unrestricted review condition. We considered trade-offs from two perspectives to decide whether to include review options on CATs: 1) psychometric and attitudinal effects of review, including magnitude and efficiency of CAT scores and examinees' perceived adequacy of review opportunity on CATs they just took; and 2) how examinees desire review options on CATs. 107 participants were randomly assigned to three different item review conditions: non-review condition (n=39), 5-item restricted review condition (n=37), and unrestricted review condition (n=31). English proficiency level served as a moderator. Results indicated that compared with non-review, unrestricted review resulted in significantly higher test performance, essentially the same testing time, and significantly higher perceived adequacy of review opportunity, while 5-item restricted review took significantly longer testing time, and did not result in significantly higher test performance. A new finding was that examinees showed significantly different perceptions of adequacy of review opportunity for any pair of item review conditions, with unrestricted review ranking the highest, restricted review the second, and non-review the lowest. The above findings all proved that unrestricted review were more beneficial than 5-item restricted review. Results also revealed that examinees of higher proficiency benefited more from review opportunity. As proficiency increased, ability estimate gains increased and RTW (from right to wrong) changes decreased. The varying benefits of review for examinees at different proficiency levels proved to be a strong piece of evidence in support of including review. Moreover, consistent with prior findings, review option was highly desired by examinees. To summarize, the findings above were generally in support of unrestricted review, while the benefits of restricted review still merit future research.