The Adjustment Of Scores Rated By Outliers In Subjective Ratings With LONGFORD Method

A score does not always presents the ability of examinees who take the test, namely the validity, is the most stubborn but the essential problem in both subjective and objective testing. To guarantee a high validity, a high reliability is required. Researchers had done a lot in the past to gain a high reliability, such as to increase raters and average scores they gave, to recruit raters who are experts in the tested subject area, to train them in the task of rating, and to instruct them about the details of the rating scales. However, none of these methods resolve the problem of subjective assessment of testing. Longford(1993) introduced four schemes, which are sadj scheme, uadj scheme, tadj scheme and sadj scheme, to solve this problem.This study has two goals. One is to introduce a new method to assess the reliability in subjective rating. This method which can evaluate the variances of true score, severity of rater and inconsistency (rater-by-examinee interaction) is based on the generalizability theory and variance component analysis. It can assess further the ratings of every rater and each pair raters who scored the same examinees. So it can provide an overall assessment of the rating process. The other is to introduce the three of the four schemes, whose pool information about the true score by utilizing the variance components and by applying shrinkage estimators, and then to use the sadj scheme to adjust scores given by outliers in HSK (advanced) compositions. The benefit of the adjustment is obvious. Meanwhile, some deficiencies are found. How to perfect these schemes would be an arduous task in subjective rating in the future.