Research On The Robustness Of Item-level Model Comparison Statistics In Cognitive Diagnosis Models

For the past decade,cognitive diagnostic models?CDMs?have received considerable attention as a psychometric model.Through this multi-dimensional model of potential categories,we can provide students with more detailed feedback on their progress and shortcomings in learning.CDMs make up for the limitation that the traditional measurement theory only focuses on the test results,and emphasizes the relationship between the explicit response and the internal psychological process of the subjects from a micro point of view.At present,many specific and general CDM have been developed in the literature.The correctly designated CDM can provide a higher accurate attribute mastery model than the general model,so it is very important to select the most appropriate CDM at the item level.Under the conditions that the Q-matrix or attribute hierarchy are correctly specified and the saturated model provides the best model-data fit,many methods are available for selecting the most appropriate CDM from the saturated CDM at the item level,such as the Wald and likelihood ratio?LR?tests.However,CDM is a simplification of reality,under the most circumstance if not all,CDMs seldom perfectly represent real world phenomena.It is reasonable to explore the robustness of item level model comparison statistics under model-data misfit condition.This paper contains two simulation studies and an empirical illustration:Study 1 was to investigate the impact of the Q-matrix misspecification on the empirical performance of the Wald statistic based on the observed information matrix(W_Obs),the sandwich-type matrix?W_{S w}?or the empirical cross-product information matrix(W_XPD),and the LR statistic for item-level model selection with respect to the Type I error and power.The simulation results showed that:when the Q-matrix is misspecified,the W_Sw and the W_XPDare robustness.Study 2 was to investigate the impact of the attribute hierarchy misspecification on the empirical performance of the four statistics.This simulation results showed that:the attribute hierarchy misspecification has a strong impact on the robustness performance of W_{Ob s},W_Sw,W_XPDand the LR statistic.Study 3 assessed the project of the examination for the certificate of proficiency in English?ECPE?that involves 2,922 candidates based on G-DINA model.The results show that:attribute hierarchies do exist in ECPE,but some of items may have the wrong Q-matrix Settings.