The Effectiveness Of Individualized Feedback Based On MFRM Statistics Analysis On Limiting Rater Variability

Though the effectiveness of individualized feedback on improving rater behaviorhas been investigated in several previous studies, different methodologies adopted inthese studies have brought different results. Until now there is no agreed conclusionas to how the individualized feedback will influence rater’s rating behavior. Therefore,this thesis aims to find an answer by improving the designing of the experiment.Sixteen raters with approved qualification participated in the present study. Theywere averagely divided into the experimental group and the control group. Firstly, atraining session was conducted. Then raters were involved in four rounds of ratingactivities, each time rating45different pieces of compositions. The scores of eachtime were analyzed through Many-facet Rash Model, from which raters’ performanceon severity, bias and consistency could be calculated and then transferred into writtenwords in the feedback. The raters in the experiment group received feedback two daysafter each rating administration. When four rating rounds were completed, successfuladjustments of the three aspects were compared between the experimental group andthe control group and raters’ perceptions about the feedback were collected throughquestionnaires and interviews.As for the quantitative analysis about raters’ incorporation and application of thefeedback, the Chi square test was conducted and it showed that raters in theexperimental group showed significantly more successful adjustments than raters inthe control group, which indicated that the feedback was successful in improving theirrating performance. Because the collected data of successful adjustments of severityand consistency was too small; therefore, conclusions couldn’t be drawn about whichof the three aspects of the rating behavior is more amendable to the suggestions in thefeedback. With respect to raters’ perceptions about the feedback, the quantitativeanalysis through the questionnaire showed that the majority of raters held a positiveattitude in spite that they proposed some suggestions for improvement of the feedback’ format in the interview. Though the raw statistics displayed that there wasan obvious relationship between raters’ perceptions and their uptake of the feedback,it is dangerous to draw such a conclusion as the sample is small and the differencecould be put down to random variation.Individualized feedback for improving rating behavior, therefore, was proved tobe effective and was supposed to be introduced in the rating administration, especiallywhen there were many non-experienced raters who were fossilized in their ratingbehavior and were susceptible to feedback. Besides，improvements should be madeabout the content and format of feedback in order to result in larger uptake.