| Writing assessment has a long and contentious history in the United States. The debate has centered on issues of reliability and validity of direct and indirect measures of writing achievement. A historical review of direct writing assessment from its origin at Harvard shows the absence of protocols that identify changes in the writing achievement of writers over time. The need for such a protocol was seen as a result of a program evaluation of a professional development program focused on the teaching of writing in upper elementary and middle schools. A corpus of 400 pre- and post-papers from 200 writers was evaluated using two methods. The first used traditional scaled-scoring procedures. Every paper was scored both holistically and on six analytic traits by trained scorers. The results of these scaled-scorings were used to calculate changes in the writing achievement. The second protocol, called Forced-Choice Scoring and introduced in this report, requires readers to compare the pre- and post-writing samples to each other and judge the differences between them both holistically and with six analytical traits. The differences are ranked as "slightly," "moderately," or "significantly" better. Comparison of the data generated by these two scoring protocols reveals equivalent numbers of writers demonstrating improvement between the pre- and post-writing samples. However, a detailed analysis of these numbers shows a more complicated picture. The protocols identified the changes in writing achievement the same way for writers only approximately 40% of the time. These differences point out the need to develop an array of writing assessment protocols and the importance of matching the assessment to the goals of the evaluation. Further, this study indicates a striking need too develop a map of writing development across the lifespan. |
