Evaluating the implementation and impacts of middle grades inquiry-based engineering design modules

This dissertation uses a format where I present and discuss three articles that were written in conjunction with my work on a research and evaluation team. The articles are based on a multi-year project researching and evaluating the design, development, and implementation of a middle grades integrated Science, Technology, Engineering, and Mathematics (STEM) program. Each of the articles demonstrates the complexity in researching and evaluating curricular development in multifaceted, rapidly changing environments. While the focus of each article differs, they all examine research and evaluation in the context of the design and development of middle grades engineering-design modules. The selected articles address challenges associated with assessing program objectives and evaluating program quality in complex education programs.The first article, presented in Chapter 2, provides an overview of the nature of the EYE program and examines the extent to which participation in the EYE program resulted in the achievement of program objectives. There is evidence that EYE Module participation has a positive impact on participating students as well as teachers. This study also revealed challenges associated with determining the impact of program participation simultaneously with program revisions and assessment development. The second article, presented in Chapter 3, examined the evaluation of fidelity of implementation of inquiry-oriented educational programs. This article was intended to examine one way of triangulating information to determine fidelity while considering variation in implementation consistent with program theories of learning. When applying this model to implementation of the EYE program, we found many teachers were implementing the modules with low to moderate fidelity, especially math teachers. The third article, presented in Chapter 4, examined the factor structure of an occupational values scale intended to measure student interest in STEM careers. When reviewing the assessment's evidence base, we found no published literature regarding the assessment's psychometric properties. We found that the AWE Work Values scale assesses two separate sets of occupational values: 1) using analytical and problem solving skills and 2) personal satisfaction. In Chapter 5, I discuss the implications of the research presented in these articles to the field of integrated STEM. This includes a need to further develop and test tools and methods for measuring program impact. While the articles address several issues related to research and evaluation within the context of K-12 integrated STEM education, the concepts and implications also apply to the general field of instructional design. The primary lesson is the need to systematically evaluate designed instruction, including measuring fidelity of implementation and designing evaluations in advance to ensure appropriate data are gathered. Recommendations for future research include identification of common outcomes for integrated STEM, psychometric testing of integrated STEM attitude measures, impacts of other factors such as interest in STEM, and development and testing of instructional design models for integrated STEM.