Institutional effectiveness in an open system: A cross-case study of graduation rates in the Montana University System

This is a mixed-methods, non-experimental, cross-case study designed to examine the environmental conditions that influence institutional effectiveness within one state-governed university system. The purpose of the study was to examine nationally defined graduation rates as a performance indicator of institutional effectiveness. Eleven delimited campuses in the Montana University System constituted the sample population. The quantitative analysis compared federally mandated graduation rates with graduation rates resulting from a modified graduation rate methodology. The qualitative analysis evaluated environmental conditions at each of the institutions. The cross-case analysis synthesized the quantitative and qualitative data using an open-systems framework to identify emergent patterns. Three questions guided the research: (a) How can open-system theory inform practice in evaluating effectiveness of postsecondary institutions?, (b) How do graduation rates for all students vary within one state-governed postsecondary education system, and (c) How do environmental conditions of a specific institution explain graduation rates within one state-governed postsecondary system? This study is significant because it expands the definition of who gets included in graduation rate calculations, it utilizes state-level student-unit-records data to measure disaggregated at-risk student group performance, it evaluates how graduation rates are influenced by unique environmental conditions at individual institutions, and it contributes to a national discussion regarding postsecondary productivity. Findings of the study include: (a) graduation rates were variable among institutions, (b) institutions with higher percentages of at-risk students had lower graduation rates, (c) no single identified at-risk student group consistently performed better or worse than other student groups, (d) no two institutions had the same characteristics, and (e) graduation rates by themselves did not account for differences between institutions. In conclusion, policymakers are reminded that evaluation of institutional effectiveness requires disciplined examination of interrelations and patterns within a larger open system. Graduation rates do not account for variable environmental conditions and they do not provide an adequate measure of institutional productivity, performance, or effectiveness.