Formulation And Application Of Grade Response Multilevel Facets Model In Scoring Subjective Item

With the development of education, our country pays more and more attentions to quality-oriented education. Meanwhile, because students’ abilities, such as analytical ability, comprehensive ability, conclusion ability and problem solving, can be reflected by subjective items, so subjective items are more commonly used in exams. However, when scoring these subjective items, rater effects would be occurred because subjective items do not have standard answers, and different raters have different cognition to the score rules. And these rater effects will affect the scoring results extremely, and finally hinder the development of tests. Thus, some methods that used to detect the rater effects were developed by researchers, for example, generalized analysis, many facets Rasch analysis, multilevel analysis and so on. Nevertheless, still some limitations existed in these methods; they cannot handle all of the problems that occurred in subject scoring circumstance. Such as it will lead to biased estimates for these methods used to detect rater effects when the task is successive processing.This paper attempt to combine with item response model, multilevel model, and rater model to form a new rater model, namely grade response multilevel facets model (GR-MLFM), which not only can it cope with successive processing task, but also can detect the factors that affect raters, furthermore, it still can detect a variety of rater effects precisely and effectively. This model contains three components:the random component, the link function, and the nonlinear component; these components which make GR-MLFM as a nonlinear mixed effect model. To examine the reasonable of this model, three studies, which include two simulation studies and an empirical study, are conducted in the paper by using MCMC algorithm.The purpose of simulation 1 is to examine the parameter recovery of GR-MLFM. In this simulation, the model does not contain any predictors, which namely the null model. To reduce the sample errors,50 replications are used, and three indices, bias, percentage bias (PB), and root mean square error (RMSE), are employed to evaluate the recovery. The results show that there are small enough differences between the estimates and true values, all values of these four indices are small enough that can be ignored. It illustrates that GR-MLFM can recover these parameters fairly well. At the meantime, the study also compares GR-MLFM with generalized multilevel facets model (G-MLFM), which conducted by Wang and Liu (2007). And the results indicate that most of the estimates of G-MLFM far away from the true values, only four estimates close to the true values; moreover, values for the four indices are large when use G-MLFM, it can be seem that G-MLFM has a poor recovery. These results demonstrate that it is inappropriate for G-MLFM to detect rater effects when the task belongs to successive processing task, while GR-MLFM is appropriate in the successive processing task. These findings in accordance with the studies of Tutz (1990) and Andrich (1995).The second simulation aims to evaluate the parameter recovery of GR-MLFM under the condition that the model contains predictors that belong to both person and raters, which namely the full model.30 replications are employed to reduce sample errors. The results indicate that almost estimates of parameters are close to the true values, the values between the estimates and true values are less than.1, except the fixed effect of rater 3 (γ30), the value is large than.1 between the estimate and true value. Meanwhile, only γ30 leads to slightly large value at these four indices and the PB is 10.101%, which indicates that the estimate has substantial bias with the true value (PB≥10%); and the other estimates yield small values at these three indices. Through these results, we can see that the model can fit the data precisely and stably, it is promising to apply the model to detect rater effect.The third study is empirical study; it is used to evaluate the empirical validity of GR-MLFM.4 subjective items, which can be used to detect students’ problem solving ability of mathematic, are employed in the model. Also, the gender of student and 4 predictors of raters, responsibility, stability of emotion, confidence, and experience, are added into the model to investigate the rater effects. Results show that with all of 20 raters, almost raters do not display substantial rater effects (severity/leniency), only rater 9 displays substantial rater effect (severity effect). Furthermore, gender has no effect on students’ problem solving ability; T test shows that students with different gender have the same ability in problem solving. Furthermore,2 predictors of raters have significant effect on raters, these factors can predict the scoring patterns of raters, of these, responsibility have positive effect on severity, and confidence have positive effect on leniency; while experience and stability of emotion of raters cannot predict the rater effects.