The Application Of Grade Response Multilevel Facets Model Under Three Types Of Rating Designs

Based on the requirements of curriculum reform,The tests increasingly emphasize the exploration of student analysis and problem solving ability.Subjective items m eet this purpose,because these items can reflect subjects' problem solving strategy adopted in answer and then can infer the potential knowledge structure or cognitive skills acquired by subjects.The openness of the answers of subjective questions makes the current scoring still depend on professional raters.Therefore,it is becoming one of the concerns in the current psychology and education evaluation of how to improve the scoring reliability in a timely and effective way in order to ensure the objectivity and fairness of a test.Combined with the actual situation,we use statistics and measurement methods to separate the sources of variation and detect the influence factors of variation in a reasonable and concise rating design context,which will give targeted training to the raters who has rater effects.It is very important to improve the accuracy of the score.The grade response multilevel facets model(GR-MLFM),which is proposed by Kang,Sun and Zeng(2016),is used to detect the rater effect and the related influence factors.GR-MLFM integrates the advantages of the many facets Rasch model,the multilevel random coefficient model,and the grade response model.And the researchers have also verified by simulation and empirical studies and prove the model has good applicability and stability in the fully crossed rating design,but how it performs in nested and mixed designs of how performance needs further validation.Therefore,this study will validate the applicability and stability of GR-MLFM under the fully crossed,nested and mixed rater designs and compare the results of the three rating designs with each other.Study 1 and study 2 are simulation studies.The applicability and stability of the GR-MLFM simple model and the complete model under the fully crossed,nested and mixed rating designs are discussed respectively in study 1 and study 2.In study 1,the impact of specific predictors is not considered when designing the model.In study 2,subjects' gender and raters' grade of self-confidence and sense of responsibility are respectively used as predictive variables of subjects and raters to make the full model,and then we compare estimated values of all parameters with their true value.The results of study 1 and study 2 show that GR-MLFM can accurately estimate each parameter under all three rating designs;and the results under fully crossed design are consistent with mixed design,both are better than nested design.Study 3 is an empirical research.The purpose is to apply the completed GR-MLFM model to the empirical data under three types of rating designs on the basis of simulation studies to explore the performance of the model in the actual situation.The results show that GR-MLFM performs well in practice,can accurately estimate parameters and effectively detect the rater effect;based on the fully crossed design,the parameter estimation results in the mixed rating design are more consistent.