| This dissertation presents an innovative approach to the development and empirical evaluation of Automated Writing Evaluation (AWE) technology used for teaching and learning. It introduces IADE (Intelligent Academic Discourse Evaluator), a new web-based AWE program that analyzes research article Introduction sections and generates immediate, individualized, discipline-specific feedback. The major purpose of the dissertation was to implement IADE as a formative assessment tool complementing L2 graduate-level academic writing instruction and to investigate the effectiveness and appropriateness of its automated evaluation and feedback. To achieve this goal, the study sought evidence of IADE’s Language Learning Potential, Meaning Focus, Learner Fit, and Impact qualities outlined in Chapelle’s (2001) CALL evaluation conceptual framework.;A mixed-methods approach with a concurrent transformative strategy was employed. Quantitative data consisted of Likert-scale, yes/no, and open-ended survey responses; automated and human scores for first and last drafts; pre-/post test scores; and frequency counts for draft submission and for access to IADE’s Help Options. Qualitative data contained students’ first and last drafts as well as transcripts of think-aloud protocols and Camtasia computer screen recordings, observations, and semi-structured interviews.;The findings indicate that IADE can be considered an effective formative assessment tool suitable for implementation in the targeted instructional context. Its effectiveness was a result of combined strengths of its Language Learning Potential, Meaning Focus, Learner Fit, and Impact qualities, which were all enhanced by the program’s automated feedback. The strength of Language Learning Potential was supported by evidence of noticing of and focus on discourse form, improved rhetorical quality of writing, increased learning gains, and relative helpfulness of practice and modified interaction. Learners’ focus on the functional meaning of discourse and construction of such meaning served as evidence of strong Meaning Focus. IADE’s automated feedback characteristics and Help Options were appropriate for targeted learners, which speaks of adequate Learner Fit. Finally, despite some negative effects caused by IADE’s numerical feedback, overall Impact, exerted at affective, intrinsic, pragmatic, and cognitive levels, was found to be positive due to the color-coded type of feedback.;The results of this study provide valuable empirical knowledge to the areas of L2 academic writing, formative assessment, AWE, and I/CALL. They have important practical and theoretical implications and are informative for future research as well as for the design and application of new learning technologies. |
