On The Relationship Between Evaluation Of Texts By Software And Human Raters

Human ratings have played the most important role in the process of evaluating the writing quality of essays for a long time. With the advent of big data and corpus, a lot of automatic software of analyzing texts is being developed and improved continually, which have made it possible to quantitatively investigate multiple textual features of writing essays and to rate essays automatically. It seems that the combination of evaluation of texts by software and human raters is an irreversible tendency, which can assess students’writing quality more reasonably. And to some extent, it reflects the integration of traditional teaching mode and advanced modern network technology.Coh-Metrix emerges as an automated text analysis tool, including abundant indices that can reliably examine textual features in a more comprehensive way. Recently, some studies with Coh-Metrix have been conducted to examine such textual features as word frequency, lexical diversity, referential cohesion, connectives, syntactic complexity, etc. However, the examination of relationships between textual features and writing quality using Coh-Metrix is still in infancy, thus it deserves further exploration in this area. Juku Correcting Network (Juku), as an automatic online scoring system of English writing, has been recently highly valued by many English learners and teachers. It has brought opportunities and challenges to traditional English writing teaching and learning.Writing is one of the most important skills in language teaching and learning. English writing quality, which can be represented by textual features, is an important element of learners’language proficiency. Many previous studies have paid much more attention to a singular textual feature or a certain type of textual features. Nevertheless, few studies have been designed to simultaneously investigate multiple textual features in writings as well as the relationship between these textual characteristics and writing quality, and no studies have put Juku and Coh-Metrix together to investigate English argumentative writings.The present paper aims to figure out the relationship between three textual features (i.e. lexical diversity, syntactic complexity and formality of text) and writing quality by conducting a multidimensional analysis of 550 English argumentative writings collected from Written English Corpus of Chinese Learners (WECCL 2.0) (Wen et al.,2009) under the perspective of Juku and Coh-Metrix. First,550 argumentative writings were randomly selected and put into Juku for ratings, and then the 550 compositions were respectively given to two college writing professional teachers to rate according to the scoring criteria (Jacobs et al.,1981). The average score of two writing raters for each essay is calculated as the final score of human ratings. After that, these compositions were divided into four groups based on the final score from Juku and human raters. Then, to be fair,193 compositions from each group were randomly selected respectively for the high score group of human ratings, the high score group of Juku (80≤x<90), the low-score group of human ratings and the low score group of Juku (70<x<80) (x=score). After that, all these four groups of English argumentative writings were put into Coh-Metrix for obtaining the figures of the 16 variables of textual features observed in this thesis. Finally, the Statistical Package for Social Science (SPSS) 19.0 was employed to carry out the statistical analysis of all the data through One-way ANOVA, Pearson Correlation Analysis, and Linear Regression Analysis, etc.Through the data analysis, the major findings are as in the following. As far as these three textual features are concerned, similarities between human ratings and Juku lie in:(1) The number of modifiers per noun phrase has a significantly positive correlation with writing quality, narrativity is negatively correlated with writing quality, and word concreteness shows a strongly positive relationship with writing scores; (2) The minimal edit distance of all words and minimal edit distance of lemmas have no correlation with writing quality; (3) Referential cohesion and deep cohesion are not found having significant relationship with writing quality. Differences are reflected in:(1) There is no relationship between lexical diversity and writing scores by human raters, while a significantly positive correlation is found between lexical diversity and writing scores by Juku; (2) The number of words before the main verb, the minimal edit distance of part of speech, the syntax similarity between adjacent sentence pairs and the syntax similarity between all combinations of sentence pairs across paragraphs have been found having significant correlation with writing scores by Juku, but no relationship with writing quality by human raters; (3) Formality of text is found having a negative relationship with writing quality by human raters but no correlation is shown between formality of text and writing quality by Juku. In addition, no significant relationship exists between syntactic simplicity and writing quality by human raters, whereas syntactic simplicity is negatively correlated with writing scores by Juku.Based on the outcomes mentioned above, the current research suggests that English learners should be aware of what textual features may contribute to writing quality, pay attention to modifiers per noun phrase and have a good mastery and appropriate application of word concreteness while practicing argumentative writings. Additionally, when proceeding English writing quality assessment, it is suggested that teachers should construct a duplex evaluation mechanism that consists of teacher evaluation and online automatic assessment by which students’ writing quality can be evaluated more objectively. It is hoped that the present study could be served as a reference for the English argumentative writing teaching and learning in the future.