Researches On Methods For Aiding Item Attributes Identifying In Cognitive Diagnostic Assessment

Cognitive diagnostic assessment (CDA) combining psychometrics and cognitivescience has received increased attention recently, but it is still in its infancy (Leightonand Gierl,2007). The CDA based on the incidence Q-matrix (e.g., Embretson,1984;Tatsuoka,1995; Tatsuoka,1990) is quite different from the traditional item responsetheory. The entries in each column of the incidence Q-matrix indicate which skills andknowledge are involved in the solution of each item. So the Q-matrix plays animportant role in establishing the relation between the latent knowledge states and theideal response patterns so as to provide information about students’ cognitivestrengths and weaknesses.Correct attribute specification is a fundamental step for cognitive diagnosticassessment (Im,2007). The Q matrix is often constructed by subject matter expertsand is an iterative process by analyzing think-aloud verbal protocols and performingstatistical validations and item analyses (Buck et.al.,1998; Jang,2009). The processescannot guarantee that the attributes associated with test items be identified completelycorrect. In practice, situations occur where the Q-matrix is difficult to specify. Jang(2009) think that adequate specifications of cognitive skills become a quitechallenging task when the CDA is applied to existing non-diagnostic tests. Even forthe cognitive diagnostic tests, DeCarlo (2011) mentioned that the fraction subtractiondata presented a case in point about difficulties with respect to specifying a Q-matrix.Furthermore, some researchers found misspecification Q matrix systematicallyinfluencing the consequences (Im,2007, see Tatsuoka,2009; Rupp&Templin,2008).Considering the Q matrix of importance for CDA and difficulty to indentifying,an attempt in this dissertation is to propose two schemas for aiding attributesidentifying. This dissertation is comprised of three major components. First, aproposed method for aiding attributes identifying based formal concept analysis (FCA)is presented, including the framework of FCA and rationale of FCA for attributesidentifying being described. For the attributes undefined, the proposed method is theunsupervised learning schema for aiding attributes identifying. Because of examinee’sguessing and slipping, the resulting structure of items is usually extremely complex,hence uneasy to be comprehended. Stability index is described and used as a noise filter criterion to build a concise representation respecting the original taxonomy inthe proposed method. The results of simulated data analysis show that theperformance of the FCA method is promising.The second part focuses on on-line item attribute identification in CDAwhich is the supervised learning schema for aiding attributes identifying.Although the study about the on-line calibration in item response theory has beenlong history, few, if any, on-line item attribute identification in CDA has beenfound in the literature. On-line item attributes identification is a new field andstudy of the impact of item bank hasn’t been found in the literature. So this studydiscussed how to implement the on-line item attribute identification in cognitivediagnostic computerized adaptive testing (CD-CAT), and introduced threemethods of on-line item attribute identification, Maximum Likelihood Estimation(MLE), Marginal Maximum Likelihood Estimation (MMLE) and a novel methodnamed as Inter&Diff based on intersection and difference, where intersectionand difference are set operations in Set Theory. The new method isnoncompensatory cognitive diagnostic model-free (CDM-free). In other words,when model-data fit is not so good, the Inter-Diff method could be employed toidentify attributes in the raw items on-line.To explore the performance of those methods when the structures of itembank vary, item banks (i.e., being the necessary and sufficient item bank or not)are constructed and analyzed using the deterministic inputs，noisy and gate model(DINA).Firstly, when the item bank is necessary and sufficient, the simulationresults showed that MMLE worked better than both MLE and Inter&Diff, butMMLE was slightly sensitive to the fixed item parameters. Adaptively seedingraw items worked better than randomly seeding raw items when the correctclassification rate of the entire pattern was relatively high. Especially when theattribute hierarchies are linear type, convergent type and syllogistic reasoninghierarchy, the result of Inter&Diff also could be comparable to MMLE or MLEas the number of response and the accuracy of knowledge states classificationincreased. However, Inter&Diff could work without assuming item parameters.Secondly, when the item bank is not necessary and sufficient, beforeemploying simulation studies, we describes the impact of knowledge states’ equivalent classes on the item attributes vectors’ equivalent classes. And twodefinitions are the following: Definition of a discriminate item attributes vector:the item attributes vectors’ equivalent classes only include one item attributesvector. The definition of an indiscriminate item attributes vector: the itemattributes vectors’ equivalent classes include more than one item attributes vector.Two simulation experiments are conducted, considering six attributes underthe independent condition. The simulation results show that log odds ratios arealmost all above zero. It indicates that the correct classification rates of thediscriminate item attributes vector is significantly better than indiscriminate itemattributes vector. The more number of the items in reduced Q matrix except thewhole reachability matrix could compensate the insufficient item bank to someextent. It also demonstrates that the reachability matrix is important for item bankdesigned for cognitive diagnostic computerized adaptive testing.The third part generalizes the application of the argument algorithm in themodification Q matrix theory, and provides real data applications to examineviability of the FCA method and on-line identification methods. Using thosemethods to analyze the fraction subtraction data of K. K. Tatsuoka, all patternmatch rates of item attributes vector are only44.44%(unsupervised method) or72.72%(supervised method) fewer than that of simulation study (about75.69%of unsupervised method). Through the further analysis of fraction subtractiondata, we found the reason is that the representive of samples of examines(knowledge states) is not enough for identification.Through simulation study and real data analysis, thus, we conclude that theFCA method has been used successfully as an exploratory analysismethod in application of attributes identification, and on-line attributesidentification methods are essential for item replenishing of item bank or Qmatrix validation for existing test. However, the DINA model is only in ourconsideration. These methods can be directly used with common conjunctivenoncompensatory diagnostic models.