A Nonparametric Approach To Cognitive Diagnosis:the Manhattan Distance Discriminating Method

Cognitive diagnosis is a recently developing measurement theory.Different from the traditional theory,cognitive diagnosis is not a simple test score,but pays more attention to the mental process of cognition.As the development of our country's educational revolution,there is a trend in educational testing that goes beyond unidimensional scoring and to provide a more complete profile of skills that have been mastered and those that have not.And then the diagnostic information about students can be useful for a further learning,teaching and policy making.As cognitive diagnosis is especially suitable to meet the demand,it becomes more and more popular.Usually,to achieve a cognitive diagnosis,a valid test and a fitting model are necessary.According to incomplete statistics,more than 100 cognitive diagnostic models have been developed till now.However,some reasons have limited the widely use of these models.Current methods for fitting cognitive diagnostic models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization(EM)and Markov chain Monte Carlo(MCMC)that frequently encounter difficulties in practical applications.In response to these difficulties,non-parametric classification techniques have been proposed as heuristic alternatives to parametric procedures.The non-parametric classification techniques are characterized by easily understanding and implementing,high efficiency and independent of sample size which are very beneficial for the application and popularization of cognitive diagnosis.Among these non-parametric classification techniques,the distance discriminating method is especially simple and easy understandable.It classifies examinees according to minimizing a distance measure between observed responses(i.e.,examinee's test item scores),and the ideal response for a given attribute profile that would be implied by the item-by-attribute association matrix.However,researches on the non-parametric classification techniques begin late and there are just a few researches about the distance discriminating method which are also setting the item scores as either 1(i.e.,correct)or 0(i.e.,incorrect).In the wake of educational test reform and the diversity of test forms,the scores of test items are not only 0 or 1.And the methods which are developed in that kind of scoring are are no longer efficient and will lose much information of examinees' responses which can lead to lower classification accuracy.Although a generalized distance discriminating method has been extended to test with polytomous response,the computation of the generalized distance is weighted by the examinees' item response probability based on the item response theory which makes the method complicated and difficult to understand.Futher,the method is not justified theoretically.By contrast,the hamming distance discriminating method(HDD)assigning examinees to proficiency classes by proximity to ideal response patterns which only needs to count the number of times disagree between examinee's observed response pattern and the ideal response patterns for given attribute profiles.The hamming distance is very natural and widely used for binary data,and the current researches of the HDD either develops discriminating methods(e.g.,Random method,Bayes method and Weighted Hamming distance method)or compares with other existing methods(e.g.,the generalized distance discriminating method).The common of these studies ignored the factors that affecting these different discriminating methods and have not been explored them systematically.This paper demonstrates that hamming distance is a special case of a more general distance—Manhattan Distance which measures the difference between the observed response pattern and the ideal response patterns in cognitive diagnosis and proposes a more generalized manhattan distance discriminating method(MDD)which can be used for test with both binary and polytomous scoring settings or just one of them.Classification consistency of the MDD in cognitive diagnosis is also given.Monte Carlo simulation is used to investigate the factors affecting the proposed method and the fault tolerance of the proposed method.A real data example is provided with an analysis of agreement between the nonparametric method and facts of students' own situation.Results demonstrate:(1)When the two assumptions and three conditions of study 1 are satisfied,it proves that the MDD has its consistency,which means as longs as test length goes infant,each examinee clustered into the proficiency classe by the MDD is the real class membership of the examinee.(2)The HDD is a special case of MDD when the item score is either 1 or 0.(3)MDD has a high classification accuracy and an irreplaceable advantage in operability compared with the existing methods,such as GDD-P(a generalized distance discriminating method for test with polytomous response)and GRCDM(Grade Response Cluster Diagnostic Method).(4)When the Q-matrix is right,three discriminating methods of MDD have almost the same rate of correct classification and can be chose according the actual demand.(5)Consistent with the existing nonparametric method,the sample size has little effect on the MDD,and more specifically,even though only one examinee can be classified.(6)The attribute hierarchy has an effect on the correct classification rate of MDD,the closer of the attributes' connection and the less ideal profiles of skills,the higher of correct classification rate.And when the number of ideal profiles is approxiamte,the looser of the attributes' connection,the easier to distinct different ideal response patterns,and the higher of correct classification rate.(7)There is no necessary of knowledge state to form a normal distribution for MDD and different distributions of knowledge state have almost the same correct classification rates.(8)The number of attributes has a little effect on the MDD classification accuracy.As the attributes become more,the maximum drop doesn't exceed 15%.(9)The misspecification of the attribute hierarchy or Q-matrix has a negative impact on the MDD classification accuracy:the more mistakes in the attribute hierarchy or Q-matrix,the bigger fall of correct classification rates.And the falls from big to small are the Weighted MDD,Bayes MDD and Random MDD.However,compared with two other methods,the impact will be less for MDD.(10)The MDD has a good empirical validity,the classification result of MDD has a high degree of agreement with the actual situation.