Validity and reliability of selected knowledge structure assessment methods

The research was to investigate the reliability and validity of three knowledge structure assessment methods: Pathfinder Network (PFNet), Concept Mapping without giving students relationships (CM), and Concept Mapping when students are given relationships to use (CMR). The network model was assumed by the three methods. Two computer programs, KNOT 4.2 and Inspiration 4.0, were used for displaying concept maps.; Twenty-seven eighth grade students served as subjects. The same thirteen mathematics concepts on the topic of numbers and operations were used for all three methods. A repeated measures design was employed and data were collected in five, 30-45 minute, sessions over four weeks.; Students' responses to the knowledge structure assessment tasks were scored using a proposition inventory and a criterion map. Three types of map scores were computed: (1) total proposition validity score (TS): the sum of the individual proposition scores; (2) congruence score (CS): proportion of propositions with a score equal to or greater than 1 in the student's map out of the total number of propositions in the criterion map; (3) salience score (SS): proportion of propositions with a score greater than or equal to 1 out of the total number of propositions in the student's map.; The study found that: (1) test-retest reliability of the PFNet relatedness ratings was low; (2) high interrater reliability was achieved when the scoring procedure of Ruiz-Primo et al. (1996) was applied; (3) knowledge structure representations from the PFNet were not equivalent to those from the CM for all the students, but they were quite similar for the students who had highly reliable PFNet relatedness ratings; (4) representations from the CM and the CMR methods had a substantial proportion of common links and equivalent relations, but there were also substantial differences between the relations in the two types of maps; (5) the correlation between the map scores and scores on a mathematics performance assessment (QCAI) was positive and moderate; (6) reliability of the relatedness ratings was not associated with students' math achievement measures. Implications of these results and the directions for further research were also discussed.