Computerized-adaptive versus paper-and-pencil testing environments: An experimental analysis of examinee experience

Computerized-adaptive testing (CAT) is quickly becoming the test mode of choice among many high-stakes testing agencies. Although some researchers suggest that CAT is comparable, if not better, than traditional paper-and-pencil testing, others have suggested that individual differences among examinees have not been fully explored. In particular, the additional cognitive demands imposed by CAT are not well understood. For example, in the CAT mode there is the added demand of moving back and forth from computer screen to scrap paper when one has to solve more complex problems. In traditional paper and-pencil (P&P) testing environments, there is but one mode with which to contend: paper-and-pencil. In addition, CAT may generate greater anxiety—which further increases cognitive load—among some test-takers than would be found with more traditional testing modes. For example, with CAT you cannot go back to review previous items or change previous answers. And CAT requires something that is anxiety provoking in and of itself for some individuals: use of a computer.; The purpose of this study was to explore the effects, particularly the cognitive effects, of different testing environments on individual test-taker experience. Forty undergraduate college students were randomly assigned to one of two testing conditions: CAT mode or P&P testing mode. All subjects took the analytical section of the Graduate Record Examination. Methodology for the study included both qualitative and quantitative methods. Data were collected using paper-and-pencil instruments, videotaped recording of subject behavior, observation, and interview. Verbal protocol analysis of subsamples of subjects' work-toward-solution behavior on two analytical reasoning problems was also included.; Results of multivariate and univariate analyses suggest significant differences across test mode, with respect to cognitive demands. Compared to subjects in the paper-and-pencil test mode, CAT subjects had significantly longer response times on analytical reasoning (AR) problems, relied more on scrap paper to model AR problems diagrammatically, and employed different problem-solving strategies. Differences in affective responses—including anxiety and frustration—further suggest test mode effects were present. CAT subjects expressed significantly more frustration with the test mode than did their P&P counterparts. Furthermore, CAT subjects with low computer confidence experienced significantly higher levels of test anxiety than did their more computer-confident peers. Implications for test developers and recommendations for further research are offered.