Development And Research Of Q-Matrix Estimation And Validation Methods In Cognitive Diagnosis

In the field of education measurement,cognitive diagnostic theory（CDT）,which emphasizes both ability level and cognitive level,emerges.It can provide refined,multidimensional evaluation results,improving the former "only score" evaluation of students.Q-matrix,as the basis and premise of cognitive diagnosis,builds a bridge between items and attributes,and its accuracy affects the application of cognitive diagnosis in the classroom（DeCarlo,2012）.In practice,Q-matrix is usually calibrated by domain experts and may be wrong.In order to reduce the subjectivity in Q-matrix calibrated by domain experts and the burden of experts,a lot of studies have been carried out from different perspectives.However,some methods need time-consuming and tedious calculation.The performance of some methods depends on certain conditions such as large sample size.In addition,the existing methods are mainly applicable to the two-level scoring items,while the research on the Q-matrix estimation（validation）methods of multi-level scoring is relatively weak.In view of this,this article,based on the perspective of model-data fit,proposed two Q-matrix estimation（validation）methods: the optimization of response distribution purity（ORDP）and D statistic.Then,the following four studies had been carried out.Study 1: Comparison of two new methods for Q-matrix validation of two-level scores.Simulation and empirical data analysis were conducted to investigate the performance of ORDP and D statistic compared with R and RMSEA,and to investigate the effect of five influencing factors（test length,number of participants,item quality,error rate and attribute hierarchy）on the validation rate.The results showed that :（1）D statistic performed best in general,followed by ORDP;（2）The validated Q-matrix of ORDP and D statistic methods fits the data better,followed by RMSEA method.Study 2: Comparison of two new methods for Q-matrix validation of multi-level scores.The performance of ORDP,D statistic and existing-2LL,AIC and BIC methods were investigated,and the influence of three influencing factors（number of participants,item quality and error rate）on the validation rate was investigated.The results show that :（1）D statistic method is the best overall performance,followed by BIC and ORDP methods;（2）The Q-matrix modified by D statistic method is closer to the real Q-matrix,and the modified Q-matrix fits the data better.Study 3: Comparison of two new methods for Q-matrix estimation of two-level scores.The performance of ORDP and D statistic compared with R and RMSEA was investigated,and five influencing factors（test length,number of participants,item quality,number or type of basic items and attribute hierarchy）were investigated on the estimation rate.The results showed that :（1）D statistic and RMSEA performed best on the whole,and their performance varied under different experimental conditions;（2）The Q-matrix estimated by ORDP and D statistic methods fits the data most.Study 4: Comparison of two new methods for Q-matrix estimation of multi-level scores.The performance of ORDP and D statistic compared with the existing-2LL,AIC and BIC methods was investigated,and the influence of three factors（number of participants,item quality and number of basic items）on the estimation rate was investigated.The results showed that:（1）D statistic method was superior to the other two methods in general.The results of ORDP and BIC method were basically the same.（2）The Q-matrix estimated by D statistic method fits the data most.In general,two new Q-matrix estimation and validation methods are proposed from the perspective of model-data fit.The results show that the proposed methods can estimate the Q-matrix well in both the two-level scores and the multi-level scores,which provide methodological support and experimental evidence for Q-matrix estimation（validation）of cognitive diagnosis.In the future,the characteristics of the new methods can be explored based on complex models such as more general cognitive diagnosis model and reaction time model,and the joint estimation of Q-matrix and item parameters under online calibration can be further studied.