| This study responded to a call for an evidential approach to test development and validation. By adopting an effect-driven testing approach, the study first conceptualized desirable effects, while paying attention to existing problems or needs. Next, it considered the way in which those effects could be achieved by using Evidence-Centered Design as an assessment framework. The context of the study was the ESL program of the University of Illinois at Urbana-Champaign. An ESL diagnostic reading-to-write test was developed in collaboration with the ESL writing instructors and administered to ESL students enrolled in four ESL writing classes. The study then explored validity evidence for the intended effects. Validity arguments on four intended effects were formulated for the inferential links between test development and validation.;The four intended effects were: (a) with a process-oriented test design, which is characterized by a sequential presentation of mediating tasks, the writing test provides test-takers with cognitive scaffolding to test-taking process (Effect Hypothesis 1) and also assists ESL instructors' and researchers' understanding of skill dependency in reading-to-write process (Effect Hypothesis 2), and (b) with a formative diagnostic test design, which features ongoing assessment and detailed diagnostic feedback, the diagnostic test helps students' understanding of their own skill profiles (Effect Hypothesis 3) and also assists students in monitoring and guiding their own learning (Effect Hypothesis 4). The study adopted a mixed-method approach for data collection and analysis to gather sufficient and meaningful evidence for the four hypotheses. Data was obtained from the target test users, which were ESL students and instructors. Major types of data included test scores, worksheets, and test users' responses to questionnaires and interviews.;To find the evidence for the intended effects of the process-oriented design, the study investigated test users' evaluations of the design, students' test-taking processes, and the relationship among the scores of multiple tasks. The first and second claims were supported by several findings, such as the students' and instructors' satisfaction with the test design, the beneficial roles of mediating tasks in the reading-to-write process, and the observed gap between reading and source-use skills that many ESL students experience. The study also investigated test-users' evaluations of the diagnostic profiles and of the ongoing assessment system to find the evidence for the intended effects of the formative diagnostic test design. Students positively evaluated meaningfulness and transparency of the profile, indicating several effect-driven features as their reasons for their evaluations. Students were interested in taking the post-instructional diagnostic test, and the students who took the test indicated that there were several benefits of taking the test. These findings supported the third and fourth effect claims.;Multiple perspectives and the evidence observed during the validation process not only strengthened validity arguments, but also enhanced understanding of the complexity of test effects. In addition to the supporting evidence, however, some factors that might mediate the achievement of the intended effects were observed. The observation of the mediating factors, such as diverse needs for a diagnostic test and various meanings of test effects among different users, suggested the importance of a systematic research design and careful interpretations of effects in a positive washback study. |
