Can college students evaluate information sources? Validating a Web-based assessment of evaluation skills

Despite widespread recognition among educators and policy makers of the need for information and communication technology literacy (ICTL), and over a decade of education reform efforts aimed at critically integrating ICTL into curriculum and instruction, recent reports find that little is known about the level of ICTL among America's students. This is not surprising considering the lack of consensus about what it means to be ICT literate and how to best measure these skills. If the construct of ICTL continues to be inadequately understood and measured, the efficacy of reform efforts intended to improve students' ICTL will be open to question. This study investigates the extent to which test scores from the Educational Testing Service's (ETS) iSkills(TM) assessment for higher education can support inferences about the ability of undergraduate students to evaluate information in a digital environment. Data from in-depth interviews with undergraduate students are used to develop a set of "naturalistic" tasks that measure students' abilities to evaluate information in a digital environment. Student performance on the naturalistic and corresponding iSkills assessment tasks is observed, and concurrent cognitive interviews are conducted in order to infer the response processes elicited by the tasks. Latent proficiency scores and response processes are compared between the respective task sets, and their degree of correspondence is used as a measure of the extent to which valid inferences can be made about the ability of college students to evaluate information in a digital environment. Additionally, students' latent proficiency scores are compared to their levels of exposure to, and grades in, ICT intensive courses. This study provides empirical validity evidence regarding ETS's iSkills assessment, and valuable insight regarding how undergraduate students evaluate information in a digital environment.