| Cognitive diagnostic assessments (CDA) is an approach where the psychology of learning is combined with methods and models in statistics for the purpose of making inferences about students' specific knowledge structures and processing skills. This study used a four-step principled approach to test design characterized by: (1) the development of cognitive models, (2) the construction of test items according to the knowledge and skills specified in the cognitive model, (3) the use of a diagnostic psychometric analysis to assess the plausibility of the underlying cognitive model and to providestudents' attribute probability estimates, and (4) the creation of detailed score reports that map examinees' mastery levels to provide more detailed information about students' problemsolving strengths and weaknesses.Being among the first applications of the AHM to non-retrofit data from an operational testing program, the findings of this study add substantially to our understanding of the necessity of a principled approach to assessment design, and also contribute to a growing body of literature on CDA. Results of this study revealed that cognitive models adequately fit the data for the total sample of students; however, the fit for the observed and expected response data differed for high and low ability students. The average attribute probability estimates were ordered, as expected, from least to most difficult. In addition, the ordering of the attributes did not differ as function of the performance level of the students and the correlational pattern of the probability estimates indicated both convergent and discriminant evidence supporting the hierarchical structure of attributes. Concerning the reliability of the models, all six attributes in Subtracting 2-digit numerals produced consistent interpretations about the mastery of attributes, whereas Comparing and ordering numbers, only the decisions made for Attribute 1 were found to be consistent. Limitations of the study and recommendations for future research were also discussed. |
