A Comparative Analysis. Certain Standards Set Method Psychometrics

Standard setting remains an important, pervasive, controversial and difficult issue in contemporary psychological and educational testing. The consequences associated with standards set on tests are typically tied to the benefits of nearly all human beings including content experts, psychometricians, policy makers and the publics. For this reason, research and practice in standard setting is burgeoning, especially focusing on the comparison of standard setting methods.This study first gives a general introduction of the topics on standard setting, which cover the definition of standard setting, the history of standard setting, the models and methods, the literature review on the topic of the comparison of standard setting methods etc. It follows with item analysis using Item Response Theory on the CET-4 (September 1999). The purpose of this study is twofold. The primary purpose is to compare three standard setting methods: Nedelsky's method, Angoff's method and Cluster-analytic method using borderline group. The comparison is in three different psychometric aspects: the rate of congruence with the external criterion, the value of test information function in IRT, and the interjudge reliability in GT. The secondary one is to use discriminant analysis to facilitate standard setting on test battery. Accordingly, the scores and item responses of 1649 examinees are obtained. The sample also consists of 5 judges who are familiar with both the outline of CET-4 and the level of examinees' knowledge and skill. The main results in this study are the followings.1) CET-4 (September 1999) consists of high-quality items with high discrimination and medium difficulty. So the results of standard setting on it are reliable and valid.2) There is a difference in the cutoff scores among the three methods. Two-way ANOVA further indicates that there is a statistically significant difference between Nedelsky's method and Angoff's method.3) The cutoff score derived from Cluster-analytic method leads to examinee classifications that are most congruent with the external criterion. The cutoff score established by Nedelsky's method produces the highest value of test information function. A similar result is observed by investigating the interjudge reliability of Nedelsky's method and Angoff's method.4) 14 or more is the optimal number of judges to employ for establishing stable cutoff scores on CET-4 with Nedelsky's method and Angoff's method.5) The weights obtained by discriminant analysis sorting from largest to smallest are that of Listening, Reading Comprehension, Vocabulary and Structure.6) There are considerable changes from unweighted context to weighted one among Nedelsky's method and Angoff's method.