The Interval Estimation Method Of Attribute Classification Consistency In Cognitive Diagnosis And Applicationin Model Misspecification

Reliability is indicator about evaluating consistency and stability of psychology and education tests.It reflects the magnitude of the random error in test.It is a key indicator to measure quality of the measurement tool.In the field of cognitive diagnosis,the research about reliability is a relatively new area.At present,researches about attribute reliability of cognitive diagnosis tests mainly include: Improved cronbach'? coefficient(Gierl,Cui,& Zhou,2009),attribute retest consistency(Templin & Bradshaw,2013),attribute classification consistency(Wang,Song,Chen,Meng,& Ding,2015)and so on.The above methods about calculating the attribute reliability of cognitive diagnosis are only point estimation,and they can't provide the accuracy of the estimation.Interval estimation not only gives a range of estimation,but also gives the accuracy.Therefore,it is important to report reliability interval about the attribute reliability of cognitive diagnosis test.In the field of psychology measurement,model-data fit test of cognitive diagnosis test is an important theme.Model-data fit test of cognitive diagnosis test is mainly to study the validity and sensitivity of the statistics.In practice,practitioners don't know the model that really fits the data.When choosing model,it is necessary to determine which model more suitable data by using model-data fit test.There are three studies in this research.Study 1 used three interval estimation methods(bootstrap method,parallel test method,parallel test pairing method)to estimate the reliability interval of cognitive diagnosis attribute classification consistency(Wang et al.,2015)by simulating study,and then compared the results of three interval estimation method estimated.Study 2 investigated whether the application about the three interval estimation methods in actual research data was similar to that of the simulation study.Study 3 explored the performance about estimating average of reliability and standard error by bootstrap method in model misspecification.The results show:(1)Whether tests contained 5 or 7 attributes in independent attribute hierarchical relationship,standard error and reliability interval what three interval estimation methods estimated were all affected by the quality of items,sample size or length of test.Standard error and reliability interval decreased with the increase about the number of items and the number of subjects.Standard error and confidence interval increased with the decline about the quality of the items.(2)Whether tests contained 5 or 7 attributes in independent attribute hierarchical relationship,average of reliability by three interval estimation methods estimating were all affected by the quality of test,sample size and length of test.Average of reliability increased with the increase about the number of items.Average of reliability had small change with the increase of the number of subjects.With the quality of items declined,average of the reliability had a large downward trend.(3)When the number of items and the number of subjects were less,standard error and reliability interval were close by parallel test method and bootstrap method estimating.However,the estimation accuracy of bootstrap method was improved rapidly with the increase of the number of the subjects.When the number of items and the number of subjects were large,the results by bootstrap method estimating were basically close to that by parallel test pairing method estimating.Parallel test pairing method took the longest time to estimate reliability interval than other methods.the calculation process of parallel test pairing method was very complicated.But compared to other methods,bootstrap method required the least time,and its calculation process was simple.Therefore,we recommended that researchers can use bootstrap method to estimate reliability interval of cognitive diagnosis test.(4)In the case of empirical data for the Tatsuoka score subtraction data(5 attributes,15 items,536 subjects),the results by bootstrap method and parallel test method estimating were approximated.Compared to the parallel test method,the results by bootstrap method estimating were closer to that of the parallel test pairing method.This empirical study supports the results of the simulation study(Study 1).(5)The attribute classification consistency and the standard error performed well in the whole study by bootstrap method estimating.The correct selection rate of bootstrap method was equal to that of AIC and BIC in which DINA generating model was.When generating model was GDINA,correct selection rate of bootstrap method was the same as that of BIC,but it was better than that of AIC.When generating model was RRUM,correct selection rate was better than that of BIC,but it was not as good as that of AIC.The correct selection rate of bootstrap methed increased with the increase of the number of subjects.