Graph and property set analysis: A methodology for comparing mental model representations

The purpose of this dissertation study was to conduct the next stage of research in the development of a new methodology (Smith, 2005) based on an analysis of Graphs and Property Sets (GAPS). The objective of the methodology is to measure the degree of similarity in structure and content of mental model representations. Such measures are useful in determining if and to what extent instructional interventions promote understanding and the acquisition of expertise with regard to complex phenomena and problem solving situations. This methodology builds on earlier research (e.g., Spector & Koszlka, 2004) and was tested in a prototype study (Smith, 2006). The research was developmental in nature and consisted of a formative evaluation of the methodology aimed at answering the following questions: (1) Does the methodology provide useful comparisons of student-constructed models based on relevant attributes of structure and content that are embedded in the model elicitation methodology? (2) What improvements in the methodology are needed prior to further research and development and eventual implementation in the form of a mental model assessment tool? (a) What improvements are needed regarding the mental model elicitation methodology? (b) What improvements are needed in the mental model representation analysis methodology?;Participants included three professors who are experienced instructional designers and 19 graduate students in an introductory design course in the Instructional Systems Program. Participants were trained to use the methodology to represent their mental models in responding to an instructional design problem. Mental model representations of students were compared with that of the professor teaching the introductory instructional design course. The comparisons addressed: (a) the degree of similarity in content and structure; and (b) specific areas in student models which might indicate misconceptions or knowledge gaps. The mental model representations of the other two experienced instructional designers were compared to that of the professor teaching the course. This analysis determined that the methodology has utility in comparing the models of persons with similar expertise (experienced designers) as well as those with different levels of expertise (professor/student). It also confirmed that the methodology identifies more similarities between persons with similar expertise than between persons of different levels of expertise.;Answer to research question 1. A comparison of student-constructed models based on relevant attributes of structure and content is considered useful if it reveals misconceptions or gaps in knowledge that, if present, will affect the design and/or delivery of instruction for the purpose of improving the potential for learners to achieve the targeted learning goals.;The comparison analysis results were shared first with the professor teaching the class of student participants to determine the usefulness of the methodology in identifying misconceptions or knowledge gaps that can affect instructor decisions concerning the design and/or delivery of instruction. Next, the results were shared with the other two professors, one of whom also was an instructor for beginning students in instructional design. The third professor, who did not teach instructional design students, did not comment on the specific application of results; however, the two professors teaching in the instructional design program responded that the information would aid them in making course design and delivery decisions. They indicated surprise regarding (a) the amount of information that the methodology could produce and (b) the extent of the gaps in knowledge that were revealed between entry level students and their professor.;Answer to research question 2. Required improvements in the methodology were addressed using qualitative data obtained from analysis of mental model representations and participant responses to questionnaires and interviews.;Questionnaires and interviews were used to obtain participant feedback on the representation process. Comparative analysis data and data from the questionnaires and interviews were examined to determine what improvements are needed prior to further study and implementation of the methodology. The initial analysis results and recommended list of changes were shared with the experienced practitioner group (i.e., the professors) to obtain their reactions to the proposed improvements. The list of recommendations include: (a) an improved training plan with more examples and additional practice; (b) assessment of understanding of both the representation process and the problem statement prior to model elicitation; (c) better design of the model to be used for comparison with student models; and (d) a set of guidelines for constructing the database and performing qualitative analyses. (Abstract shortened by UMI.)...