A comparison of summarization and free recall as reading comprehension tasks in Web-based assessment of Japanese as a foreign language

In recent L2 reading and reading assessment literatures the use of extended-response reading comprehension tasks has been proposed for assessment purposes. The main reasons supporting this argument include the rich information available for locating readers' comprehension problems, the relative "pureness" of these measures unaffected by reading comprehension questions, as well as the potential effectiveness of these task types for assessing global comprehension (Bernhardt, 1991; Johnston, 1984; Riley & Lee, 1996). However, practical applications of these task types, which are relatively uncommon and beset by complex scoring processes, require in-depth research that explores their underlying nature. The present study compared two such extended-response reading assessment tasks---written summarization and free recall---in terms of the constructs these task types tap into as well as the dependability of ratings awarded to learner-produced protocols.; The participants were 160 learners of Japanese as a foreign language enrolled in Japanese language courses at five universities in the United States. Four major findings of a series of qualitative and quantitative analyses conducted on multiple data sources are the following.{09}First, the heightened attention to the global text structures for gist formation observed in a small subsample of the learners' concurrent verbal protocols collected during task completion indicated relative suitability of summarization for assessment of global comprehension. Second, the completed written summary and recall protocols were found to be similar in the extensive use of omission of unnecessary information as well as copying and paraphrasing without much integration of information beyond multiple sentence levels. Third, a confirmatory factor analysis (CFA) of the ratings awarded to the summary and recall protocols suggested that the recall tasks tapped into two highly correlated and yet distinct constructs, comprehension and integration of main ideas vs. comprehension of details, whereas the summary tasks assessed comprehension and integration of main ideas only. Finally, the partial interdependence among the summary and recall rating scales and the susceptibility of these two task types to method effects found in the rater's verbal protocols recorded during rating sessions as well as in the CFA and a multivariate generalizabiIity study suggested the need for further refinement of the rating criteria.