Assessment Construct In Foreign Language Teaching

This thesis views human essay rating as a problem solving process that iscomposed of three interrelated states: the initial state, the intermediate state and thegoal state (Anderson1990:257). When an essay is assigned to a rater, s/he is in theinitial state where s/he identifies the problem and is aware of the goal of assigningscores to it. Immediately, the rater begins to evaluate the quality of the essay and s/heis in the intermediate state. In this state, the rater has to undertake a series of cognitiveoperations, such as collecting information, interpreting meaning, processing theinformation collected, recognizing the situational and contextual influences,considering ethnic issues, making initial decisions and etc. When scores are assignedto the essay and the rater makes his or her final decision, s/he is in the goal state, andthe problem is solved. A rater’s problem solving behaviors are viewed as guided ordirected by his or her special cognitive structure (Sternburg1985; Ausubel1968;Zhang2007). This structure is constructed through the dynamic mechanismequilibration, which is fulfilled through assimilation, when the rater’s priorknowledge structure is confirmed by the outside stimuli, and adaptation, when therater’s prior knowledge is contradictory to it.(Piaget1977; Gao2007).Based on literature reviewing and pilot study, this paper hypothesizes that theprocess of human essay rating is the interplay of three cognitive structures raters haveconstructed: the writing structure, the rating structure and the contextual structure.The writing structure is the mental representation of raters’ standard on essays ofdifferent proficiency levels. The rating structure is the mental representation of thecognitive operations raters undertake in solving the problem. The contextual structureis the mental representation of raters’ knowledge concerning the social aspects of thewriting test. Upon the hypothetic framework, this research seeks to answer thefollowing three research questions: How are the three cognitive structures like? Whatare the factors that influence the construction of each of the structures and how dothey contribute to the construction? What is the nature of human essay rating process? This research takes as an example the essay rating of Test for English Majorsband-8(TEM8)2012. Four data collection methods were employed in the presentresearch:1) ratings of17raters each rating25essays were collected in order to obtaingeneral information concerning essay qualities and raters’ rating qualities so as toprovide background information for further analysis.2) A questionnaire focusing onelements affecting essay rating process was issued to75raters in the attempt to elicitfactors contributing to raters’ construction of cognitive structures.3)6raters’ thinkaloud verbal protocols (TAPs) when each rating10essays were recorded, transcribedand coded to gather information from which an inference regarding the ratingstructure could be made.4) Follow up interviews to the6TAPs raters were conductedto investigate the writing and the contextual structure and their construction.The data were analyzed in the following four sections:1) Multi-facets RaschModel was used in analyzing17raters’ rating quality;2) factor analysis (bothexploratory and confirmative) was used in analyzing the questionnaire data;3)transcription of6raters’ TAPs report was coded and qualitatively analyzed usingNvivo8.0;4) the follow up interview recording was transcribed and coded andqualitatively analyzed using Nvivo8.0.This research draws the following main conclusions:1) There are six principalfactors influencing essay rating process: rating scale, rater training, ratercharacteristics, rating context, text feature and rating supervision. The aboveidentified factors can be further brought into three broader interplaying factors: theinstitutional constraints, the rater characteristics and the text features.2) The researchfindings confirm the hypothesis that the nature of human essay rating process is theinterplay of three cognitive structures: the structure of writing, the structure of ratingand the contextual structure.3) The constructions of the three structures arecontributed by a number of personal and social factors, each of which interferes, withdifferent degrees, into different components of the three cognitive structures.The research may bring implications to the conceptualization of the construct ofessay assessment, relationship between rating reliability and validity, theunderstanding of the role of rater characteristics in human essay rating, the role of the rating scale and its development, the purpose and key points of rater training, the wayof supervision, TAPs approach in investigating essay rating process.