Oral retelling as a measure of reading comprehension: The generalizability of ratings of college-aged second language learners reading expository text

The purpose of this study was to refine a rating procedure used to assess college-aged ESL students' ability to orally summarize ideas they had read from three expository passages. Oral retellings from 24 students were tape-recorded and rated on two different occasions by each of two trained raters. Three different dependent variables were included for each student: (a) recall of main ideas, (b) recall of details, and (c) a composite rating that was a total of the previous two ratings. A three-facet (passages, raters, and rating occasions) generalizability study was conducted using the GENOVA computer program (Crick & Brennan, 1982). The largest sources of error variance in the G study were identified as (a) passages, (b) the student by passage interaction, (c) the passage by rater interaction, and (d) the student by passage by rater interaction. A series of D studies were conducted to predict the effects of using more practical designs for collecting the ratings and the effects of varying the number of passages, raters, and rating occasions. Graphs were prepared for each design showing how the error variance decreased and the generalizability coefficients increased as a function of the number of passages, raters, and rating occasions.; No matter which dependent variable was used, and no matter what design was used, the D studies consistently showed that the most important step that could be taken to reduce measurement error and increase the generalizability of the ratings would be to use at least four, and preferably six, passages to assess ESL students' reading comprehension. Ratings for recall of details produced greater generalizability am ratings for recall of main ideas, but the composite of the two ratings produced even better results. The gain in generalizability from using additional raters varied somewhat depending upon which design was used, but using more than four raters was generally not worth the extra time and expense. The gain in generalizability from having the raters rate each passage more than once was minimal in the original design with one nested facet, but slightly greater in the design with raters nested within students, and in the design with raters nested within passages and occasions.